Topic: How to get rid of doubled transactions (Read 2583 times)

OK, I tried again, this time deleting few malled transactions. End result? Some accounts were deleted/not linked with BTC address anymore. Just great.

So there is currently no solution whatsoever to get rid of these malled transactions...

I still haven't cleared them. But I read somewhere, that transactions "timeout". How long does it take for transaction to timeout?

Oh, forgot to mention this; I have several incoming malled transactions. These are still in the list.

Honestly I have no idea.  pywallet will delete the tx.  Will bitcoind "accounts" handle that incorrectly?  I don't know.  I don't think anyone knows for sure.  You will find the "accounts" are not a good feature in bitcoind.  It probably should be dropped from the mainline client.

Still if you made a backup RIGHT BEFORE "cleaning up" and it messed things up you just restore the backup you made immediately before doing the "cleanup".

Backups are not the issue, have few of them. I am more interested whether this method will work and work correctly. The total cleanup as suggested earlier completely messed all accounts. Suddenly for example, my fees account had a masive 1.5 BTC inside (from nowhere) and overall balance was like 2-3 BTC less than before.

Correct.  If one of the tx is already confirmed you should be able to just delete the other one using pywallet.  If neither tx is confirmed and you delete one it is possible that is the one which ends up confirmed in which case you would need to go back and delete the other transaction.  Also until confirmed it is possible another node may re-relay the transaction you deleted.

Always make a fresh backup before using pywallet, it is a low level tool and very powerful.  It does exactly what you tell it to even if you are doing something "stupid" like deleting funded private keys.  You can loose funds if you make a mistake in pywallet without a backup.

I might not have been clear but I agree.

Bitcoind = equivalent of online banking
Bitcoin accounting (i.e. custom backend) = equivalent of traditional accounting software

Most users will find out eventually they don't want to try and do accounting in the wallet just like you wouldn't want to try to do accounting using online banking.

I am visiting http://pywallet.tk/ and seeing this:


So, I can remove these transactions manually, right? Will this method work? I cannot afford to put my service to halt again for an hour.

Actually as a user of traditional accounting software I can assure you that "it is nothing like that" and why I (and many others) don't even *touch* the Bitcoin "accounts" functionality.

Although the Bitcoin devs are amazing in what they can do - they don't seem to understand "accounting" much at all.

The problem is that the client does this by default.  For example, look at these 2 transactions on block 285303:

http://blockexplorer.com/tx/359e7c0c45f8633e3b85e0c5de79c8c0e7600fc1b4e17ba67e7bf4541379aa2a#o0
Inputs:
txid: 290fd5edf00e...:1 amount: 0.0035285 from address:   1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
Outputs:
0 23885ee7630b... amount: 0.00290698 1LuckyY9fRzcJre7aou7ZhWVXktxjjBb9S
1 3393e29876b7... amount: 0.00052152 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ

Notice that the client has split the amount into 2 amounts, one to the payee (1LuckyY...) for the desired amount and the other which credits the remaining balance back to the same account (1FSRTDG...).  This is because each transaction must use the entire balance of the inputs in the outputs.
Then this transactions is logged on the same block that uses the output from the first transaction.

http://blockexplorer.com/tx/3393e29876b7090e439762a0c3bf359818821bfb2c44317e25c144e9b2cdcab9#i74103801
Inputs:
359e7c0c45f8...:1 amount: 0.00052152 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
00c8195a8fea...:1 amount: 0.00079582 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
2b473eb3853d...:1 amount: 0.0003604 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
Outputs:
0 444f9a598a6a... 0.00145349   1LuckyR1fFHEsXYyx5QK4UFzv3PEAepPMK
1 07c00c8a3ada... 0.00012425   1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ

If the first transaction gets mutated and the hash is changed from 359e7c0c45f8... into something else then the 2nd transaction will become orphaned, corrupting the wallet and preventing the coins from being spent until it is cleaned up.

Well that is just my opinion.  I know Gavin likes it.  For a personal wallet it might be fine but I think you will find that building a custom datalayer which connects to bitcoind to be a better long term solution.  If nothing else it provides better flexibility and scalability.

I guess it is kinda like quickbooks (or other accounting software) vs online banking.  It is far easier to use accounting software and then link to online banking to pull the incoming and outgoing txs then it is to somehow try to use online banking for accounting.

Yes, this is what I need. Didn't know BTC's internal accouting was garbage... So far it worked without any issues.

That alone won't help, the issue is that the doubled transaction affects the QT clients "account" recording.  The sender isn't spending unconfirmed change, the tx is being modified (by those trying to prove a point) between the sender and receiver.

One relatively simple option would be to patch the QT client to DELETE unconfirmed mutated transactions (but not traditional double spends) once the confirmed version has one or more confirmations.  If the developer didn't want to delete it, a flag could be added "duplicate" instead which by default would hide the double from transaction display and the "accounts" balance.

An even better patch would be "smart" enough to realize that in a doubled transaction (same inputs, same outputs, different hash) only one copy of the transaction can ever end up confirmed.  Since they both have the same value it would never even consider the second copy.

So
Copy1 receive - unconfirmed balance increased (or decreased for spend)
Copy2 receieved - no change (maybe a special symbol indicating a double and by default doubles aren't even shown

If copy 1 ends up confirmed there is no change other than the confirmed balance increases and unconfirmed balance decreases.
If copy 2 ends up confirmed, copy1 gets the "double" flag, copy2 "double" flag is removed.  The user sees no change in total balance, and the visible tx changes.  The only change is the hash so to the user they simply see the tx hash change.

Sorry about that.  Yeah the built in qt accounts system is well ... garbage for reasons beyond this.   Sorry I assumed when you said "account" you were talking about your own internal accounting.  That would be a problem.

So if you want to use QT built in "accounts" your only option would be to either wait or use a tool like pywallet. The amount of time before they fall from the memory pool is not deterministic, it depends on your nodes resources and the tx volume so it could be hours or weeks.  Obviously some patch for handling this better will be needed but in the interim those are your only options.  As a heads up if you delete the "copy" prior to one of them being confirmed, it probably will come right back, this is because other nodes will still be relaying it.  If you delete the unconfirmed copy AFTER the other one is confirmed the duplicate should not be relayed anymore (as all nodes will see the unconfirmed one as a double spend).

The other option (and in the long run I think you will find this works better) is to scrap the QT "accounts" system and just use the client/wallet as a transaction engine.  In your own database you would track user balances, payments, withdraws, etc, and just query (using RPC API) the client for low level data.  This provides more scalability and flexibility.

For example you can assign a user a deposit address by calling getnewaddress and recording the result in your database linked to the particular user.  When a payment is received at that address you can look it up and know which user to assign it to.  To make detecting transactions and blocks, bitcoind supports the ability to assign a callback function (could be an external program, web call, etc).  It will pass the tx or block id to your system when an inbound tx (or confirmation) or new block is detected.

But a simple patch of not spending unconfirmed change outputs would work around this problem?

I am not, this is what bitcoin-qt does by default (I use RPC API).

The problem is a fault in the Bitcoin client itself.  Despite everyone's instance that the problem is outside of that, there is still a fault in the bitcoin client.

If 2 transactions are made from the same wallet on the same block and the client chains them (i.e. uses the outputs of the first transaction for the second transaction) and the first transaction is mutated by the network, then the second transaction becomes invalided and rejected by the network.  The client does not clean up the second transaction and it's coins become un-spendable without a lot of technical knowledge of how to fix the corrupted wallet.

Small time users that only do a few transactions a day are not seeing the problem.  Large users like mtgox and coinedup that do many transactions from the same wallet in the same block on a regular basis are getting hammered by this fault.

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

The only issue is that amounts in accounts are reduced as long as these unconfirmed transactions are on. I only want to release this "locked" amount back to customers, so they can spend it.

I already use min 6 conf. (do not consider spending coins if they are not min 6 conf).

Basically what I need is a tool that would allow me to manually remove these long unconfirmed transactions - they do not get removed on their own - at least they didn't so far (except if bitcoin-qt restart prolongs removal of them).

And the issue are not incoming transactions, but outgoing.

An example is; customer A loads on 1 BTC. He then pays for rig B amount 0.5 BTC. Because of the attacks, there are 2 transactions created, each 0.5 BTC (total 1 BTC). One gets confirmed, another one is in state of never being confirmed. But side effect is that his account would show final balance 0 - he is unable to spend the remaining 0.5 BTC for anything else. That is the issue I would like to fix. I hope you understand me now.

Why do you need to remove them?
Why are you giving credit to users for 0 confirm txs?

They will drop off automatically when they are deleted from the memory pool to make room for newer transactions.

If you absolutely need to delete them you can use a tool like pywallet to manually remove the tx from your wallet.  Still before you do that you should question why do you need to remove them.  Think of 0-confirm tx as unpaid orders.  Do you need to delete unpaid orders? 

This is a patch that stops the qt wallet from spending change outputs with less than 2 confirms. It might help.

http://www.reddit.com/r/Bitcoin/comments/1xm49o/due_to_active_malleable_transaction_relayers_it/