In my thinking there are two balances, - apologies if this is a bit long....
The Reserve Balance which is an amount of NXT you cannot withdraw and you can initiate instant transactions up to that level - this is a permanent reserve until cancelled.
The other balance is the Instant Balance which is updated as soon as an instant transaction is broadcast by a node i.e. 0 confirmations.
This reflects the liability the account has created with an instant transaction.
The node the transaction is broadcast through will have an realtime view of this because it will update the accounts instant balance before broadcasting the tx, all nodes seeing the TX will also update the instant balance for that account.
If the account tries to initiate more TX that would make Instant Balance > Reserve Balance this would create an error.
An attack vector such as you describe would rely on being able to send the TX through a node which had not yet updated its instant balance total for the account in question.
For instant transactions to work I would want to ensure that both accounts had to be connected to a node and both nodes had the same view of the instant transaction balance of the sending account. If the seller is logged into the buyer node then this is a possible edge case attack.
This means that the sellers account can confirm that there are sufficient reserved funds for the instant purchase because it also has a view of the buyers instant balance that it can verify with other nodes - this would be a possible client verification/check during the purchasing process, the seller NRS node is passive in this process other than providing data to the sellers software client.
Even if the buyer switches nodes, the seller doesn't and the sellers node reconciles the instant balance of the sellers account using normal time line rules.... So unless the buyer can get the seller onto a node that doesn't know the buyers balance or initiate trades with lots of sellers which it knows are connected to nodes which won't get the instant balance update then an attack will fail ( I think)
Once the instant TX is confirmed the liability reduces and the instant balance can be reduced.
Me, as a bad buyer could easily game the system by replaying all the transactions over and over again, in order let nodes delete all (except one). I can play this game over and over again.
I would rather have a combination of
http://qubic.boards.net/thread/9/fighting-scam (hope CfB does not mind) and CCT.
One part we missing so far is that
the seller can cheat as well.
We should have a refunding transaction instead of arbitrary limits.
