Bottom line, PRICE DOES NOT INCREASE BECAUSE DIFFICULTY INCREASES.
Suppose difficulty increased by 100x. According to your logic, prices would still not increase because difficulty does not affect price.
Let's say Bob is a Bitcoin newcomer. He decides to mine to get them, but then difficulty increases by 100x. So instead, he decides to buy them because mining would be far less profitable. Other people have the same thinking as Bob, and decide to buy instead of mine. This drives up the price. Difficulty does therefore affect price.
People want money, and just because of that, price affects difficulty.
That's much more powerfull than your assertion on how difficulty affect price.Price to difficulty ratio absolutely affects where new "investors" choose to invest - mining or buying coins. If I want to get 50 coins, I have two options, buying them (or trading for them), or mining them. If I buy them, it will push the price up. If I mine them, it will not. So when people become interested in Bitcoin and want to make an investment, the ratio absolutely could decide how the price adjusts.
However, this factor may not be as large as people make it out to be. The price/difficulty ratio has been extremely favorable for miners up to this point, yet the price still has surged. So difficulty is still catching up. It's hard to put TON of systems online quickly to mine. Hardware takes time to assemble, ship. Rigs take time to set up. So if there are ever big price spikes, difficulty can keep rising for a long time after that, even without any more price surges.
Right now, the balance is still favored more toward miners than it has been in the past. So you'd expect to see more people still going for mining than buying. The higher the price gets, the harder it gets to bump the price up even higher, but raising the difficulty costs the the same. So the price jump will be much less than the difficulty jumps for a bit.