You're basically correct in your summary of the entropy argument. Assuming you're using a published method for generating your private key from your passphrase (sha256 is particularly popular) then the question boils down to "how hard is it to guess your passphrase."
Here's the basic math: A Bitcoin private key is 256 bits, but realistically we're not interested in finding your exact private key, merely finding some private key for which the RIPEMD160 hash of the public key matches your address - yes, it is possible for more than one private key to map to the same address... This means that in order for us to not lose any entropy, any passphrase or key we select should contain at least 160 bits of entropy itself. If we're talking about natural English text such as you'd find from a book or poem, each word only represents about 1.5 bits of entropy. log2(1.5^274) results in 160.280... bits of entropy so we'd need a whopping 274 words of natural English text to make a suitable passphrase.
Now on the other hand, if we choose the
xkcd method and build our passphrase using completely random words from a set of the 4,096 most common words then each word represents a whopping 12 bits and our privkey can be constructed with 14 words since log2(4096^14)=168.
Still, if you have a memory like I do, 14 random words is still a bit much, so I've come up with another method. I had two identical sets of dog tags printed up. Each dog tag can hold something like 15 characters per line, but I only used about 10. Each line is numbered 0-9 (I used two tags) and each line contains 8 characters of random Base32 text. Base32 was chosen because it is case-insensitive and most dog tag embossers can only do uppercase letters. Each line of text represents log2(32^8)=40 bits of entropy, so any 4 lines combined creates exactly the 160 bits needed for a private key. I memorize a PIN and my dog tags do the rest. My memorized PIN is useless without the physical artifact that is always on my person (a backup copy of which is in a safe place) and the artifact is useless without the memorized PIN. Better still, multiple PINs means multiple accounts, so I don't have to re-memorize a huge amount of data. Sure, if someone stole my tags and knew how to use them it wouldn't be hard to generate all 10,000 possible private keys and sweep away all funds in every account, but I'm still safeguarded against everything but a physical theft perpetrated by someone who knows EXACTLY what they're after. If I need more security than that, I'll generate a one-time true-random paper wallet and store it in a fire safe or something.
So in conclusion, if you don't have an absolutely stellar memory you're going to need a paper backup - if you can memorize it, chances are it doesn't contain enough entropy.
BTW, shameless plug I know, but I did write a
Bitcoin Address Tool for Android. It's 100% compatible with
Casascius' Bitcoin Address Utility, generates QR codes for both the privkey and address, stores no data, is open source and when combined with airplane mode makes brain wallets on-the-go somewhat easier.