Agreed. I got in early. Not quite $1 early, but after the first rise and fall. Now granted, I figure I'm doing alright with mining but yeah, if I had the capital back then to buy up a bunch of coins at $2, or $5, etc., I'd be more than happy. But then again that's the risk we all take to an extent and I could imagine it being quite painful to have fully sold all coins at a (then) high price. But who's to say the same isn't true today? I'm happy the price is this high but if you're confident it's going to go far far above this, it's always time to buy.
It's a chicken-and-egg problem, if the early adopters had mined a hell load of coins and hoarded and refused to sell, the economy around bitcoin would not have developed, the price would have been nowhere near what it is now, to get people to join, they must redistributed the coins by selling them, (they may have done it involuntarily, we are all pawns of the invisible hand after all) eventually the free market will make things fair, that's why the unfair advantage of early adopters theory is a myth.