It is not theoretically impossible, although such a scenario is not going to happen anytime soon. Right now, around a few tens of thousands are using Bitcoins actively. If the usage grow to a few million, then we can expect the exchange rates climbing to five-digits, or even six-digits. It may take five or ten years, to achieve that much adoption and usage.
It all depends upon
what exactly these tens of thousands are actively using Bitcoin for, lol. Does just holding bitcoins pass as active usage?
Active users mean people who either send or receive Bitcoins at least once a week (miners, traders, gamblers, and even users here who are part of sig-campaigns). I will not classify those who are just holding their coins as active users. There is a tendency to inflate the BTC user statistics. However, I am sticking to conservative estimates.
Okay, it would be weird to call ordinary holders active users. But in respect to your initial point ("if the usage [should] grow to a few million"), they must be active users even as holders. Let's call them actively holding users, if you are not quite happy with calling them just active users (I'm not either). How come? Since you talk about expecting the exchange rates to climb to five digits (or even six digits, wow), the rank-and-file Bitcoin
holder would play a very important role in this process because by simple holding he diminishes the amount of coins in circulation (well, trading) which directly contributes to the exchange rates rising. As you can see, by doing nothing the regular Bitcoin stasher bug is actually doing very much (by helping others earn profits, lol)...
So we don't in fact need the usage to grow to a few millions (as you mean it), we just need a few million Bitcoin holders