Just use a plastic card with a credit card style number on it, and a pin
Having a card with some number other than the private key on it is a possibility, and would work well with the existing credit card financial market, but the downside is the centralization. In order to get a "bitcoin credit card" in that case, I have to open an account with the BitcoinInv or whoever is offering the service. That central authority now has my wallet file, and I have to transfer funds to my account there. Then, the situation is like I described; I only need to bring my card with my BitcoinInv account number on it with me to the store, and use their POS system to make the purchase from my account. However, that centralization now means BitcoinInv becomes a honeypot for thieves. If someone can get in and crack whatever security is behind all the wallet files there, they can walk away with funds from (potentially) anyone who has an account there (depending on the security measures in place).
Having the private key on the card itself, and if the POS system were independently-verifiable to not keep those private keys after being used, there's no central location for a thief to hit to steal all the customer's money.
So, yes, I'd like to see something like the process you're working on come to be, BitcoinInv, but in order to make it work, whatever agency becomes that central point needs to be big enough to be the central processing point, have the servers be reliable enough to handle lots of load, and have ironclad security. That, I believe, is a bigger hurdle than reprogramming a POS card scanner to understand the account number coming in isn't a credit-card-style number of five groups of four numbers, but a string of hex or base-58 characters, no?
If/when bitcoin (or a successor) takes off big-time it will be enabled by a network of complementary services. The idea that the average retail customer will interact directly with the core bitcoin network, with the associated pain of wallet management, transaction absolute finality (ie no chargeback, aka consumer protections) and the rest, strikes me as ludicrous. In time, it will come to be seen as "just another currency", adopted because of its primary benefit of predictable money-supply growth. The bitcoin network will be the currency's (near)RTGS. The world's financial plumbing system (visa, CLS, SWIFT, whatever) will require little more than trivial tweaks to support it (primarily, links to the bitcoin core network for gross settlement transactions). If this prediction is true then the world's POS devices can already support bitcoin with next-to-no effort.
I agree! The issue, however, is the "If/when". Currently the credit-card infrastructure has assurances like the FDIC and whatnot that make me the consumer feel okay handing my credit card to a waiter to go do something a back room with (presumably charge me for my dinner and nothing more). Getting to that point is a great long-term goal, but to start with, but there's no one big enough yet to be the strong arm that cracks down on Bitcoin fraud and insures other central processors if they get robbed. So the power needs to stay with the consumers; I'd just like to see it make the step away from owning a smartphone to be the barrier to entry in the day-to-day transactions with Bitcoin.
Until the day that there's an agency that insures other central bitcoin processors against theft, I won't be keeping any large amounts of coins in online accounts such as the one you're describing, BitcoinInv. But still keeping pocket change there for day-to-day expenses would be quite convenient, and I would take advantage of that.