Anyway, is it possible to create separate "specific-use-only" wallets you could store in, say, your phone or your Trezor, where funds could only be sent to specific whitelisted addresses?
I don't see a reason to create such a thing, there already exist gift cards (as a private payment system)
The idea is that the coins in the wallet could only be sent to specific addresses -- legitimate merchants. If a thief demanded your bitcoins, he'd have to steal the entire physical wallet device and could only spend the coins at legitimate merchants. He could not simply transfer coins to his own wallet. If the hardware wallet were stolen, the police can easily put together a database of blacklisted addresses which are pushed to merchants (this could be very effective if bitcoin change could be forced to go into old addresses instead of generating new ones). This DOES NOT affect fungibility.
I image a hardware wallet as a device is a device that can be purchased anonymously. It would be cheap, and designed to cheaply connect to a public network wirelessly to send transactions and monitor the blockchain. You would use it to transfer money to another private person's wallet or the pay terminal of a retailer. It's not designed to hold a lot of money, that is, more money than you are willing to lose. It would be analogous to filling your own wallet with cash for the cash purchases you intend to make for the day. The reason people may want anonymous wallets is the same reason today you would choose to buy something with cash rather than using a credit card. You may not want the receiver to be able to know who you are.
Now, if you lose your hardware wallet, you can go home and sweep the addresses that it owns with the optional wallet backup program. Same thing if a thief physically takes your wallet, you can sweep the wallet with the backup program before the thief does, you get your money back. If the thief sweeps the wallet before you do (after cracking or coercing the password on the device such as a 4 digit pin), then the police can track the money until it hits a public address that may unblind the thief (i.e. a retail pay terminal with surveillance.)
If you keep a large amount of money in such a wallet, then you may be asking for trouble. There's always a small possibility that manufacturer of the wallet may have inserted or allowed a hack that allows someone else to gain access to the wallet's private keys.
The idea of a hardware wallet is that it would become an acceptable risk to carry certain amounts of money, and the device or the money can be lost without causing a catastrophic loss of funds. There's no reason to implement a specialized limited fungibility system for bitcoin, the retailer can just sell you a gift card instead.
I could see a gift card being an okay solution if it were instead, say, redeemable private currencies I could redeem for a currency of my choice, but I was thinking more along the lines of something I could use at any legitimate merchant, but not a fencer, illegal gun salesman, or street-corner drug salesman. When I was thinking about this, I was envisioning a rubber hose kind of scenario, where bitcoins were being demanded of me, but it would be impossible for me to send them to the thief, so threats to make me hand money over to him would be ineffective since I couldn't send money to his address. I could give him the physical wallet hardware and password, but then he'd have to only use it at registered merchants, where the blacklisting and unblinding, as you mention, would come into play.
I'd guess this solution, if feasible, would permit conversion of the "specific use" keys/funds back into "true bitcoins" (thus not really impacting fungibility except while you're away from your cold storage device) while still allowing you to use the bitcoins at almost all merchants which accept BTC, which I definitely can't do with gift cards. It's also trustless insofar as BitPay (or whoever organizes the theoretical system) doesn't control funds -- they'd need my keyparts for each expenditure, which I'd hold. Idunno, though. It does all sound very complicated with fairly little reward. Hopefully, there're better ideas out there which I like from a security:convenience perspective.
