Now that these risks have given us huge returns, I have already seen at least one, "Oh, you just got lucky," in reply to someone saying, "But.. But.. I made all this money!" This is where it starts, and I have already seen where it typically progresses:
* You didn't work for that money, you were just lucky!
* You just got lucky, so you don't deserve that money!
* It's unfair that you have that money and I don't, since you just got lucky (and I didn't bother taking the risk and called you stupid for it). That money should be spread fairly!
* Finally, the money is either taken, or we are resented for having it
To be honest, I see a lot more of this type of "early adopter envy" within Bitcoin communities than I do outside of them. It also extends to mining. There's a subset of people who regard it as "just not fair" that acquiring substantial amounts of BTC is no longer as easy as it was even a couple of years ago. They perceive the risks they would need to take in order to "get into Bitcoin" now as higher than the risks taken by early adopters.
Many people convince themselves that "given the chance" they would not only have been early adopters (whether of BTC, Google stock or some other new kid on the investment block) but they would have held onto their investments through the ups and down as well, when that's likely not the reality. It's not an exclusively Bitcoin thing.
"Unearned" income is becoming an important contributor to financial security for many people who will never "get ahead" with the money they earn from their jobs. People look to dabbling in real estate, playing the stock market, precious metals, and other types of investments to propel them across socio-economic barriers and give them a shot at the middle class and beyond. Investments are never risk free, even if they may appear so in hindsight. You can't "get lucky" unless you have skin in the game.
I see a certain desperation in some people in respect of Bitcoin. They view it as their only path to financial well-being. Counting on any one type of investment to secure your financial future can be very dangerous, but I don't see the problem as being Bitcoin itself but rather a particular attitude - if Bitcoin didn't exist these people would be regarding some other investment in the same manner. Some people seem to have a talent for losing money on everything they touch and they're often the people who can least afford to lose it. Those people will likely lose money on Bitcoin (I'm convinced they'd manage to lose money on Berkshire Hathaway stock).
Honestly, I think you need to stop worrying about whether people believe you "deserve" whatever you've made out of Bitcoin and just enjoy the benefits it's brought you. Money doesn't care where it came from.