You are correct. They only have to attack your passphrase. However, Bitcoin-Qt and Armory both use "key-stretching" which is guaranteed to slow down key guessing by a couple orders of magnitude.
Bitcoin-Qt uses the simplest version -- the password is hashed 25,000+ times to get the encryption key (the exact number is chosen based on your system's speed). This means that you either try to brute force the encryption key directly (impossible), or you try guessing the passphrase but requires hashing each guess 25,000+ times. This dramatically slows down an attacker. But, GPU acceleration is feasible on this, because hashing is so cheap in terms of RAM and simplicity.
Armory uses something more advanced, related to scrypt. The password isn't just hashed N times, each hash is saved into a lookup table which is then used for another N lookup operations. This means that they not only have to do the sequential hashing, but each thread that is guessing passwords needs to have N*64 bytes of dedicated RAM to store the lookup table, and that lookup table is specific to the passphrase being guessed (so threads can't share it). This makes it pretty much useless to apply GPUs, since much of their speed advantage is due to having only a couple kB fast memory per thread, not megabytes.
Armory also tests your system speed to set the parameters to about 0.25sec. If you use a single CPU on a completely randomized 6-character password, it would take about 500 years for 50% chance to find the password. Obviously, attackers may have multiple systems/cores to parallelize this, but they have to use CPUs, because of what's mentioned above. Therefore, if you use approximately 8 characters of real entropy, it's pretty much useless for someone to try, even if they have extraordinary resources.
Of course, people use weak passphrases. But there's only so much you can do to protect people from themselves...
Thanks for the great explanation! I should have realized the clients would do something like this; I use 1Password which does the same with its master password. I still don't think the very occasional security transfer of wallets would hurt, just to easy my paranoia.