It is not feasible, with the speed of light as an upper limit, to do this faster than a second. As I understand it you actually use something like 10 minutes for one block, or even more for several block verifications. This means that verification of the absence of double spending is 10,000 or even tens of millions times too slow. This would get even worse in interplanetary trade of course.
It depends on your network layout. If you are using the entire public network, then yes you are depending on that time for the data to be processed by everyone else that is running a node. But, you don't have to do it that way. You can build a trusted node (like a server) and have your clients direct connect to it and let it do the fast verification for you.
It won't net you 20 confirmations in a few milliseconds, but it will help stop double-spending as long as you remain in control of that node because it knows what is being spent. The whole double-spending issue arises from trust. When you are comparing anonymous seller to anonymous buyer, the only "middle man" for trust is the entire swarm (hence confirmations that neither is cheating)
If you know you can trust your own node(s), then it reduces the likely-hood that double-spending will occur within your own trusted network. So if your example applied to an ATM like card for example, the bank (or whoever) would have all of those ATM machines networked to their own private node (or server) which would then further connect to the outside (Internet) for the rest of the transactions. So if you went to an ATM, withdrew 100 BTC (say the entire account only had that much), then moments later went to another ATM elsewhere and tried to withdraw that amount again, somewhere along that trusted chain would spot a double-spend attempt and basically report back no BTC left for that address.
The whole verification process is how you put the swarm to work, all those clients are examining if your transaction was valid or not, but you don't have to wait for them to transfer the coin around. You can spend the coin around in circles all day without a single transaction verification. It's only when double-spending occurs does the network correct for itself and fix the balances.
Banks already have this exact same problem. They all have their own networks that connect to other bank networks and it's just as easy to double-spend on their ATM as has been demonstrated at black-hat hacker conventions for probably 10+ years now. I remember reading about the issue back in 1997 because a lot of networks will still on dial-up then for banks and the latency issue was part of the attack because everyone had their own *database* that all had to inter-connect.
With bitcoin, everyone is the database and banker and guard at the same time, so while it's not perfect, it's actually a lot better than what banks are using. Have you ever noticed at an ATM they have about 20 "network" stickers for all the different banks and setups that the ATM has to be compatible with?