How difficult is really difficult? It depends on how much in the way of computer resources someone is willing to expend on the attack, which depends on how much money you have stored in your brain wallet.
You need to test how many passphrases your computer can create brain wallets from per second. If your computer is old then multiply this up to a decent high end computer available now. How long will it take them to crack your password? (there are 3 x 10^7 seconds in a year).
Now work out the number of possible passphrases according to the scheme you are using. If you go with printable ascii characters then there are 95^x =~ 10 ^ (x * 2) possiblities, where x is the length of your password. If with a randomly chosen word from the most common 10000 words in english then 10 ^ (x * 4) possiblities where x is the number of words. eg.
8 characters =~ 4 words = 10 ^ 16 possibilities
12 characters =~ 6 words = 10 ^ 24 possibilities
I've just generated a sample 10 ^ 24 possiblity password for each of these so you can decide which type is easier to remember.
Lj0Z4c|==i5CJ
heroin swallowed goddamn hustle serge imitating
If you assume the crackers top end hardware will depreciate to near nothing over 5 years, it is only worth them trying to crack your password if:
value_of_bitcoins_in_brain_wallets > cost_of_hardware * password_possibilities / (passphrases_per_second * 5 * 3 * 10^7)
Note that if many brain wallets use the same generation code then a hacker can test all of them at the same time, so the important value is that stored in brain wallets using the same generation code as you, not that stored in your wallet.
The best defence is probably to make brain wallets slow to generate. If the cracker can crack brain wallets at 350billion per second then you need a long passphrase. If the first step of the brain wallet is to sha hash the passphrase 80 billion times, it might take you 80000 seconds to create your brain wallet (just under a day) on your 1MH/s CPU, but even a cracker with a 80GH/s ASIC is only going to be able to try crack passphrases at one per second. A 3 word (or 6 random character) passphrase now takes them 100000 years to crack...
Ivanol