Topic: Organic UTXO consolidation (in the course of regular payments) (Read 276 times)

Most of the time, that's going to be pretty awful behavior. Say fees are high, so your algorithm is going to try avoid consolidation for as long as possible. Then finally say there's a big transaction that can only be created via spending the whole wallet at a high-fee time. With very high fee rates, that mistake can easily end up thousands of dollars in fees :/

I suspect most people would rather avoid that degenerate case, and instead just keep a bit extra money in their wallet. Or even erring with a "hot wallet doesn't have enough balance" and topping it up is probably preferable. If you look at the coinsayer API, you'll see it recommends setting maxInputsToSelect to 3. which is the sort of behavior most people want (although you're free to set it to what ever you like, if you don't mind sweeping the wallet like that)

fee_per_full_coin should be adjusted if the median fee, averaged over a prolonged period, becomes significantly higher (or lower). And even if you lag a bit with this adjustment, your utxo set will not degrade instantly. You can monitor the recent median fee and tell if your fee_per_full_coin setting is too low or too high - you basically want it so that with median fee, you will be mostly choosing averagely small txs (assuming your utxo set is already in good shape). 2 or 3 inputs seem like a good choice, but as the size of inputs varies depending on the properties of addresses that the wallet contains, the actual setting will be specific to the concrete wallet setup.


70 out of 100, more than 2x of usable amount, is quite a big extra, IMO.

An interesting concept. One of the imperatives that was set when we created our system was that the risks should be eliminated or mitigated to the maximum extent possible, so this was not considered. But this is quite easy to add, as an option for those who can tolerate the risk, so we might think about implementing it.

Another important requirement was that if there is a spendable balance on wallet, and a payment order comes to spend this balance, it should be sent - the developer using an API should not bother about if there enough big utxo available, or if the transaction limitation might be hit, or whatever - if there's enough funds for the order amount (and network fee, if the fee is taken from the wallet, and not from recipient)  - send it, use every utxo available, even if it means sending several txs.

CPFP means you just paying more fee in the child tx - if you wanted to use the result of the sweep soon, you could just have added more fee at the tx creation


If you do not conflate utxo from different wallets (or sub-wallets), there's no problem - user should have used different wallets/sub-wallets for different purposes, if this is a concern

Same. (see: https://simplexum.com), but it is not focused solely on coin selection, so probably you've done more work in this direction.


What would prevent this from scaling ?


Yes, but if you have too much small utxo, and you need to send large payment at inconvenient time (when fee is high) - you will pay a premium, as delaying payment is often not an option. Also you might hit transaction size limitations, and will need to send several transactions to satisfy one payment.

Holding both fair amount of big utxo and a lot of small utxo in hot wallet might mean your hot wallet balance is higher than necessary, and your risks are also higher.


But this means you are freezing this funds for a week, and need to get funds for operation from somewhere else, like your cold wallet.

Also, big sweeping transaction is more visible.

You can do special sweeping tx without this formula. The idea here is that if you do payments regularly, the fee in the network varies, and you set your fee_per_full_coin reasonably high - you will get gradual UTXO consolidation, without the need for separate sweeping transaction.

It may not be very useful for a single user wallet that spends occasionally, but for busy service that receives and sends a lot of payments - might mean that when they suddenly need to send a big payment, and the network fee is high - they won't need to send a huge-sized tx and pay a lot in fees, because their UTXO set will be in good shape.

For UTXO-based cryptocurrencies, it is important to have the set of managed Unspent Transaction Outputs in optimal shape, because if you have too much small UTXO, you will have to create bigger transactions to send the same amount of funds, and hence pay bigger network fee, or even need to send several transactions to transfer the needed amount.

Typical approach to handle this is to do periodical sweep of UTXO, bundling them all together, at the time when network fee is low.

We implemented another approach, that does not require manual UTXO consolidation, and at the same time, conserves network fee - when fee in the network is high, the algorithm tries to use less UTXO per transaction, and when network fee is low, it will create bigger transaction, effectively performing UTXO consolidation along with regular payments. This organic UTXO consolidation allows to maintain healthy UTXO set automatically, without manual consolidation runs.

We implemented it this way:

Define a rate that you are OK to pay in terms of monetary percentage: we call it "fee per full coin". Basically, you say, "to send 1 btc, I am willing to pay 0.001 btc"

When building a transaction, we calculate optimal tx size like this:

    optimal_tx_vsize = utxo_sum * fee_per_full_coin / fee_per_kb

The closer candidate transaction to the optimal size, the better score it will get.

Of course we cannot always find the optimal transaction, but we will try to choose the one with best score, depending on available UTXO set.

The resulting effect is that when fee_per_kb is high, optimal_tx_vsize will be low (may be even zero), and we will try to find the smallest tx possible with current UTXO set. When fee_per_kb is low, optimal_tx_vsize will be high, and we will choose transactions with more UTXO, performing organic UTXO consolidation.

---

If more wallets implement this type of optimizations, it will result in more healthy global UTXO set.