If you asked me that in early 2012, I would've said something along the lines of: Getting over the hurdle of getting enough people to understand bitcoin that it develops a self-sustaining community.
That seems to have happened. We have hundreds of bitcoin-focused companies building tools and services; even if 90% fail (which is typical of startups), we still have a solid core. There's been over $600M in VC poured into Bitcoin, the fruits of which we're only beginning to see (this trails 1-3yrs, at least), bitcoin is accepted by some of the biggest companies in the world, and bitcoin is being seriously discussed at high levels of government, the corporate/finance world, and academia.
If in 2012 you told me that it would only take 3 years to get to where we are today, I would've written it off as very unlikely. The magic part has probably already happened... As Wences Casares of Xapo likes to say, it's probably far more unlikely to have gone from 0 to where we are today, than to go from where we are today to 1 billion users eventually.
So what worries me? With the perspective of having seen this ecosystem develop over the past 4 years, not that much. While it's still a coin-toss what bitcoin's ultimate fate is, there's far less to worry about than just a few years ago. If I had to give a concrete answer, perhaps some worldwide coordinated government crackdown on Bitcoin's legality....which seems highly unlikely.
And if you're looking at charts day-to-day as a potential source of things to worry about, then you're doing this Bitcoin thing all wrong. Think long term.
This is where I'm at right now too, very well summarized.
Having been around to see and feel actual fear that it could go away, none of this price movement over the last few days has affected me, nor the 15 months on the way back down from 11/13.
All that said, do not put any money into BTC that you're emotionally attached to - because even knowing all of that, the violent price movements
will beg emotion from you, and probably get it. Especially the first few major movements.