I see your point and I can't say that I strongly disagree with it
People that buy on the rise and sell on the fall can be called market conformists or even opportunists (actually, it is a good profit earning strategy), so they are definitely not the ones who start or change trends. But while they are not, they still add momentum, power and force to the trend, so, at the end of the day, it is exactly these people that make a trend reversal more difficult when the time has come. And from this point of view, they do affect market direction just by supporting it, thereby preventing its change
In fact, the answer to your question is pretty simple and straightforward
Bitcoin was and for the most part still is a vehicle for speculation, but when the jig is up and it no longer brings profits, speculators lose interest in it