There are two factors at play when considering the difficulty:
1) The miners that can mine at present prices not at a loss.
2) The miners that can't mine profitably at present prices.
Not quite sure where this is going, but I'll try to address it.
Do you deny that #2 exist? If they do exist, what are they doing with their coin?
What kind of affect is that having on the supply?
Well, I guess I do, in a sense. I mean, someone may mine bitcoins at a cost of $4/btc when the Mt. Gox market shows $3/btc, but if they keep doing it, is it really fair to say they aren't profiting? They're doing it for some reason, whether for the anticipated future profit, or because for them personally the cost of going through Mt. Gox works out to over $4/btc (or is just impossible), or just for the personal enjoyment as a hobby, which is worth the extra $$$. I don't see the relevance, because if they continue to mine, even "unprofitably", then they don't affect difficulty anyway.
If they
are not (present tense) mining, then they aren't miners... they are either buyers or someone sitting and watching from the benches, so they aren't relevant there either.
If they
were mining, but stop and walk away due the price dropping relative to the difficulty, then their
demand has been reduced, and they help lower difficulty, and lower price as well by allowing the coins they would have mined and kept to be available for others to buy.
If they
were mining but stop and start buying, due to the price dropping relative to the difficulty, then they help lower difficulty, but do not impact the price. If they were mining 50 coins a month, but stop and instead have the
same demand, and buy 50 coins a month at the current price, then the remaining supply of available bitcoins doesn't change, and the price stays the same.
At no point would the action of these "unprofitable" miners raise the price.
#2 exist because there are investors (and will be more) out there that see that Bitcoin is secure, fast, and global and they are willing to hold that kind of a currency, even if it costs them more in fiat government money today, because they know the fiat government money isn't necessarily always going to be around. It's essentially a risk vs. reward decision. And I'm saying there are large financiers right now in global politics that have a lot of this fiat money that might through $3 million into an FPGA array for no reason other than "risk".
So you're saying that there are new people who
want bitcoins. That would be increased demand then, wouldn't it?
Increased demand is what pushes up the price. ("I expect bitcoins to be $100 soon, or more if the dollar collapses! I want some!" so they buy more and take them off the market, reducing supply and driving the price upward.)
And this increased price--even anticipation of it--is what drives more people (sometimes those same people!) into mining, which drives up the difficulty.