Since when we send the BTC payment it deduct the BTC amount from the balance. If so how it can happen?
It is a type of exploit. The person sends funds without a transaction fee, then sends again with a fee. The transaction with the fee will get confirmed and the other will eventually get rejected by the network. It can also happen if a mining pook has then 51% of the global hashrate. They would technically be able to chose which transaction were confirmed and which weren't.
But if the network rejects the transaction, then the funds should get returned to the sender. Eventually it will become a scam as the receiver will be waiting for confirmations which will never get done. Am I correct?
If you make a payment but include 0 fee and have a dust output then perhaps, it won't get confirmed either. I doubt merchants would accept these kind of transaction unless they can't wait for a few seconds or check for competing transactions.