Whether Bitcoin's price matters or not both depends on your psychological reference point for 'value' and on the use case.
If your psychological reference point for 'value' is the US-Dollar, you'll see Bitcoin as a more or less ordinary investment like stocks or bonds. A drop in price will hurt you, because you'll get less $$ if you sell your Bitcoin which equals to a negative return on investment.
If your psychological reference point for 'value' is Bitcoin, you'll see the US-Dollar gaining value and maybe you'll ask yourself if buying US$ is a good investment. But you'll know for sure that the US$ is manipulated by a central bank and that such currencies will be a thing of the past. So it doesn't hurt you at all.
That's for the investment / store of value use case. The merchant / trade use case isn't affected by the price at all, because either Bitcoin is directly converted into US$ (for those merchants with a US$ value reference point) or it is kept and never converted, making US$-price meaningless.
