Topic: Why importprivkey needs rescanning blocks instead of checking UTXO? (Read 308 times)

The wallet interface and design is setup to show *all* transactions, sent and received.  The only way to do that is by some form of rescanning.  There is no way for the wallet to indicate "stuff might have happened before X but I don't know about it", so without that failing to scan the history could get you into trouble (for example, you accidentally defraud business partners or customers by not crediting them for payments you received outside of your wallet's view, or you unintentionally file false tax reports, or you accidentally make a payment again which you already made, etc).  Beyond confusing the user, the wallet UI expects the inputs for all spends to exist, otherwise it has cosmetic problems like displaying "negative fees".

You can see more about this in this issue I opened in 2016: https://github.com/bitcoin/bitcoin/issues/9409

AFAIK that is the only real reason that it is the way it is. Changing the interface in such a way that a partial wallet could be displayed isn't easy, and doing so in a way that won't break the interface for existing users may not be possible.

Re, achow's comment about technical debt-- I don't really agree. Yes the wallet contains transactions, but if that were the only issue, less than two hours of actual development(*) would be required to make a wallet centric rescan that used the existing scantxoutset functionality to quickly find a list of heights to scan. The whole process could be done in minutes rather than the 8 hours a rescan currently takes. Even if the blocks had to be fetched from the network it (due to pruning) it wouldn't be too big a deal.  But with that done, you'd still be left with the wallet interface being wrong and confusing and sorting that out would be a massive change because it would have to change how the user interface works, presumably break API compatibility, etc.

The original reason the wallet contains transactions is almost certainly because of how Satoshi expected SPV to work: Rather than scanning blocks like multibit/bitcoinj work which has terrible performance (which has mostly caused the death of these clients) or expecting bloated centralized servers to have huge indexes (resulting in most working electrum servers probably being run by spy companies) satoshi expected they payees would get handed evidence of the confirmed payment from payer.  The copy of the txn in the wallet in Bitcoin used to have a copy of the merkle proof in it, presumably for this purpose.  The proof was removed because it was bloating up wallet files from large users and wasn't used, I don't even think we really understood what purpose it would be useful for when we removed it.  Payment protocols took a different route than one which would have made it useful in any case.

But even ignoring the unused SPV use case, possession of the complete transactions is needed: E.g.  For non-segwit utxo, the whole transactions are needed by untrusting parties in order to validate the amount of fee paid. This comes up with offline signers and hardware wallets. (In fact, because segwit's sighasher is still somewhat flawed, there are still some vulnerabilities to fee overpayment if you don't look at the inputs, even with segwit in use.)

Given that, you can't simply just go and eliminate the wallet storing transactions as some refactoring job against the codebase. To drop transactions entirely you'd have to either drop support for non-segwit outputs or at least the ability to export them for non-trusting signers-- real functionality the actually gets used. It isn't technical debt.  But the propensity to call any behaviour that developers don't understand or like technical debt is as strong as ever.

(*) Actual development meaning the time it takes for a knowledgeable person to actually write a correctly working feature. Getting new functionality merged into Bitcoin Core is another matter entirely, and would probably take months of effort now.  As a result very little new user visible functionality is added to the software.

There are many reasons, but it mostly comes down to technical debt in Bitcoin Core's codebase.

Originally, Bitcoin Core did not build a UTXO set. Instead it indexed every transaction in the blockchain and would look up transactions directly. So when a new transaction arrived, it would lookup the transactions specified in their inputs and check whether the specified outputs were marked as spent. There was no separate UTXO set as we do today. So to get the UTXOs for an import, you had to rescan the entire blockchain. Even after the change to construct a separate UTXO set, this hasn't really been changed.

The second issue is the wallet itself. The wallet does not work with UTXOs. It uses entire transactions, which you cannot get from the UTXO set. The wallet computes the balance by iterating through every transaction stored and checks whether each output belongs to it before adding its amount. So there is no explicit UTXO set in the wallet either, it has to check every transaction it stores. Because of this, we cannot use just the UTXO set as we require storing the entire transaction, even if only one UTXO as actually ours.

The last reason is that users expect to be able to see their transaction history and be able to look at transaction details. Both of these are not possible without a rescan. Just the UTXOs do not provide the transaction history, and the UTXOs also do not give any other details about the incoming transaction.

---

Reworking the wallet so that you can just scan the UTXO set (and so that does things with just UTXOs too) is on my list of things to do. But there are other priorities in the wallet right now as what it does now is still mostly sane.

You're correct. Bitcoin Core still doesn't allow the client to import the private key and spend funds based on the UTXO set. I believe they don't intend for Bitcoin Core to be used if the client doesn't have a history of the transaction relating to the addresses. It would definitely be useful but I doubt this feature is at the top of their priorities right now.

If the client can mark someone else's transaction as valid and it is based on checking UTXO set, it should also allow spending my own funds. Why it is not possible without rescanning blocks?