This is certainly true if you encrypt a JPG image, as a cracking program can easily check whether a brute-force result yields a valid JPG file.
It may be much more difficult with just the text of the key, because that looks like gibberish anyway. Even with a relatively short password, many brute-force attempts may yield some similar gibberish, so the cracking program cannot discern the valid key from other wrong results with wrong passwords.
This is an important argument. It should be noted, however, that it is relevant if private key is in the raw, 32-byte form. If it is, the attacker has no idea whether he got the right passphrase until he performs ECDSA to create the public key, then ripemd160(sha256()) to create the address, and then search the blockchain for the address. That's a lot of work for each try, and only when you get lucky can you start hoping that there are unspent outputs.
In case of the usual WIF-encoded private keys, it seems easier because of the checksum. First of all, the brute-force attacker checks if candidate begins with a "5". Next, she converts it to hex, and checks if last four bytes are sha256(sha256()) of the remainder. If not, move on. If yes, only then search the blockchain. This eliminates most of slow steps related to blockchain search.
Does this make sense? How does computational burden compare between all the hashing and checking the blockchain in the above examples?