Difficulty is more than just a number, it's a part of the very mechanism that allows bitcoins any value at all. How can you suggest difficulty may not affect price when the existence of price or value at all is dependent upon the mining process which is governed, in part, by the difficulty algorithm?
I can give you a direct example of how difficulty does not cause price.
People mine with GPUs and price of Bitcoin is $30. Suddenly a magical being switches everyone GPU that was used for mining into an ASIC card, that has the same power usage, just 100 times more hashing power. This magical being also switches the difficulty to be 100 times bigger on the exact last block where difficulty usually adjusts, so no 2016 block adjustment period is observed.
Miners who were not paying attention for the next couple of months would not see any difference, their costs are still the same, their share, and thus the number of Bitcoins is still the same. Nothing changed for them. Exchanges would not see any difference, still the same number of coins is being produced each two weeks, so supply from miners would be the same. People who trade Bitcoins and did not pay attention (and did not look at the difficulty numbers) would also see nothing has changed.
So, we would see the difficulty increase 100 times, yet no 100 tomes increase or decrease in price. Tell me again why should difficulty changes cause changes in the price?
OK, some people may think, like you, that increase in difficulty should increase the price, and if enough people think that, the price will indeed rise with the difficulty. But this is all in the perceptions, and hard to model. The price will rise, even if enough people think that it will rise, no need to involve difficulty at all.
As for it being a part of Bitcoin that gives it value: every part of Bitcoin works together, you can't take out anything and still have it work. Difficulty is definitely not *the key* part that gives it value, I would say the key part is how it all works without central authority, all all parts together enable that property.
I have a feeling that you are actually thinking of hashing when you say that it is the key part. But, again, even hashing is just a way to get unchangeable history, and sure this is a key part of Bitcoin, but, again, all other parts work together, you can't just name one part and say that it is the part that gives bitcoin value.