Topic: Q: The difference between txn memory pool and UTXO (Read 981 times)

Yes, I understand now.
My issue was that not only was I overall incorrect in my understanding, but also that my
termonology and vocabulary was preventing me from articulating my intention to you. For
example, which you state, "coins" and "addresses" are abstractions and do not exist within
the code for functioning, but are representations for convenience and use. The problem is
that I am aware that they do not exist, but since I am a laymen with all of this in general,
when I use "address" and "coin" it is due to me not knowing the proper terms, so I refer
to the abstraction so you can determine what I am intending to say. But ultimately, that
leads to increased confusion and misunderstandings. Majority of the time, the proper term
I should have been using was "input", and at other times, "output".



This visual representation is great.
If you didn't create that, I would still be reading all your words very slowly over and
over attempting to understanding the procedure properly.

I can see all the different parts, their individual importance, and how it reacts to
certain events or non-events.



I understand now. My original understanding was very wrong.



I always knew that certain terms the community used didn't actually exist within the code,
but I did not understand that it essentially breaks down to the input/output specifically. I always
viewed the input/output only as the ledger's accounting for "this is where the coins originated"
and "this is where the coins are going". Its very wrong, on different levels.


I fully understand now.
Thank you for taking the time to correct and explain this.

I still need to get a good book about the basics of Bitcoin and really learn about everything else properly.
My misunderstanding centered from assuming input/output was just the ledger entries' status.
Which is either due to my lack of computer programming/science or just my incorrect assumption.

It is probably time for me to start reading books by experts who write for novices.

I truly appreciate the reply's and overall discussion. As mentioned, I'm new to Bitcoin (only been researching for about 2 months) but what I've come to realize every day is there's more and more misconceptions of how things actually work.

You can come across a concept at a high level and it seems fine, but then when you get down to the nuts and bolts the details are harder to track down. But that is why it is important to have these kinds of discussion to break this flow of misconstrued understanding that has probably been built upon something we read somewhere and presumed to be true.

This is not true.

The sum of the values associated with the outputs is not allowed to be greater than the sum of the value provided by the inputs.

However, the sum of the values associated with the outputs IS allowed to be less than the sum of the value provided by the inputs.  When this is so, the extra value is a "transaction fee" which the miner can include in the output of their coinbase transaction when they include the transaction in their block.

At the block level, the sum of all the outputs of all the transactions in the block is not allowed to be greater than the current block subsidy plus the value provided by all the inputs of all the transactions in the block. (It is allowed to be less than, but why would a miner pay themselves less than they are allowed to?).

There are no transactions in the UTXO.  Only outputs.

Even before a transation receives a confirmation, the outputs that are referenced by the inputs are REMOVED from the UTXO, and the new outputs are ADDED to the UTXO.


Every one of those transactions also spends outputs that used to be in the UTXO.  Those spent outputs are removed.


They are valid unspent outputs.  They can be used in another transaction.


Transactions don't get spent.  Outputs do.  The spent output remains spent.  The new confirmed transaction has its own outputs. Those outputs were unspent and in the UTXO before the transaction got confirmed. They remain unspent and in the UTXO when the transaction is confirmed.  Confirming the transaction doesn't change that.


The outputs of those unconfirmed transactions are in the UTXO.  The outputs that are spent by the inputs are removed from the UTXO.

The outputs that are added to the UTXO are removed from the UTXO if the transaction is invalidated OR if the transaction is dropped from the mempool OR if the outputs are spent by another transaction.


Transactions are validated before being added to the UTXO.  If the transaction is invalid, then it won't be added to the mempool or the UTXO.  However, a transaction that was previously valid can become invalid. In that case the now invalid transaction is removed from the mempool and its outputs are removed from the UTXO (and depending on why it became invalid, some of the outputs spent by its inputs may be added back into the UTXO.


Well, to start with, there are no addresses.  Those are an abstraction that our wallets and block explorers create for us humans to make it easier to talk about the transfer of control over value. They don't exist in transactions or the blockchain.

Next, inputs never "move".  They are listed in a transaction and they remain there.

Finally, coinbase transactions have outputs, but never have inputs.

Here's a good analogy:

==========
Transaction A (Coinbase)
Inputs:
        none
Outputs:
        Value:1250000000   Encumbrance: Signature required from private key associated with public key Z
==========

("Transaction A, first output", added to UTXO)

==========
Transaction B
Inputs:
        Transaction A, first output, signature from valid private key
Outputs:
        Value 1000000000  Encumbrance: Signature required from private key associated with public key Y
        Value 0240000000  Encumbrance: Signature required from private key associated with public key X
==========

("Transaction A, first output", removed from UTXO
 "Transaction B, first output", added to UTXO
 "Transaction B, second output", added to UTXO)

==========
Transaction C
Inputs:
        Transaction B, first output, signature from valid private key.
Outputs:
        Value 0500000000  Encumbrance: Signature required from private key associated with public key W
        Value 0400000000  Encumbrance: Signature required from private key associated with public key V
==========

("Transaction B, first output", removed from UTXO
 "Transaction C, first output", added to UTXO
 "Transaction C, second output", added to UTXO)

==========
Invalid transaction D
Inputs:
        Transaction A, first output, signature from valid private key
Outputs:
        Value 0500000000  Encumbrance: Signature required from private key associated with public key U
        Value 0740000000  Encumbrance: Signature required from private key associated with public key T
==========

(Transaction D rejected because Transaction A, first output is not in the UTXO, it was removed by Transaction B)

==========
Transaction E
Inputs:
        Transaction B, second output, valid signature
        Transaction C, first output, valid signature
Outputs:
        Value 0730000000 Encumbrance: Signature required from private key associated with public key S
==========

("Transaction B, second output", removed from UTXO
 "Transaction C, first output", removed to UTXO
 "Transaction E, first output", added to UTXO)


So at this point the UTXO lists:

• "Transaction C, second output"
• "Transaction E, first output"

If transaction E takes too long to confirm and is removed from the mempool then "Transaction E, first output" will be removed from the UTXO list.  "Transaction B, second output" and "Transaction C, first output" will both be added back into the UTXO list.

In that case, the UTXO list would revert to:

• "Transaction B, second output"
• "Transaction C, first output"
• "Transaction C, second output"

Note that the UTXO is just a list.  Things are added to that list, and removed from that list, but that doesn't have any effect on what is in the transaction.  They transaction remains static exactly the way it was created with its own lists of inputs and outputs.


The term "input" is not just a place holder. An input is an actual list item in a transaction.  A transaction consists of two lists.  The list of inputs is a list of references to transaction outputs.  It is part of the transaction and it doesn't go away. It is permanently stored in the blockchain when the transaction is confirmed.


No.  Inputs are not listed in the UTXO.  Additionally, outputs are removed from the UTXO when they are referenced by an input.


Inputs are never spent or unspent.  Their existence is what causes an output to be removed from the UTXO.


Like I said earlier,
"Bitcoin doesn't work the way a lot of people expect it to, and when they are first exposed to the whole input and output concept it can be confusing."


There are no "coins".  There are ONLY inputs and outputs in a ledger.


I'm happy to discuss it with you.  The first step is to unlearn what you think you know and then to understand that from a technical standpoint:

"coins" are an abstraction for us humans. They do not actually exist at transaction and blockchain level.

"addresses" are an abstraction for us humans. They do not actually exist at transaction and blockchain level.

A transaction is really just 2 lists. There isn't much more to it. The first list is called "inputs", and is a list of outputs of other transactions along with signatures that satisfy the encumbrance of those outputs.  The sum of the values of those outputs is the total value available for the transaction to use in its own outputs. The second list is called "outputs" and is a list of new unspent outputs each encumbered with a requirement (typically a signature requirement). The sum of the values of those outputs is not allowed to exceed the total value supplied by the inputs.

I guess I just don't understand then.
So you are saying that even before a tx receives a confirmation, it is "added" to the UTXO set.
So if the mempool is flooded with millions of spammy valid txs, but they pay an appropriate fee, in
theory, there would be an increase of millions of new UTXOs within the UTXO set for at least 72 hours?
So the data in the UTXO set are not always valid unspent transactions?


Don't you mean the prior tx remains spent, and it child is unspent, but is already in the UTXO set?

So, all broadcasted but unconfirmed txs are all already in the UTXO set and thus are only removed
from the UTXO set if the spending tx was invalidated or dropped from the mempool?
The UTXO set assumes all broadcast txs are valid until told otherwise.


What I thought was that: The inputs were the spent outputs from it's parent tx. When an
address has valid inputs that are unmoved, the address is an "unspent output". Thus the UTXO term.
So metaphorically, the input only exists to validate spent outputs that are being transformed into
unspent outputs within a new address for example. Coinbase's would be the only true input in my
thinking, since they "originate" outside of the blockchain. This was my reasoning.

I thought outputs were moving from a state of unspent to spent continuously.
Inputs were just another representation for of an unspent output.



My point was that the term "input" was just a place holder to describe an unspent output.
So when I gather multiple inputs to make a new tx with 1 output, I am really gathering
multiple unspent outputs and making a new tx to spend them to another address where the
process could begin again if I controlled that new address.

Do inputs exist within a UTXO?
Are you saying UTXOs are unspent inputs then?

I guess I really do not understand.
This is not my field, so I do not know the basics of computer science that some would use
as a standard foundation. I should probably read about this aspect specifically.


Edit: After a superficial search, I understand why I am incorrect. When I am using the
term "input" I am only referring to the "coins" associated with the prior tx, when in reality an
"input" contains an "outpoint", a "signature script", and a "sequence number". I assumed they
were separate and part of the full tx and not part of the input/output aspects of the tx.
Ultimately, I need to read a well written book on Bitcoin for Novices and really learn everything
from the beginning and properly.

I understand.  Bitcoin doesn't work the way a lot of people expect it to, and when they are first exposed to the whole input and output concept it can be confusing.



Actually, the outputs of an unconfirmed transaction are added to the UTXO as soon as the transaction is seen.  It doesn't wait for the transaction to be "included in the blockchain".  Furthermore, the only way a "spent transaction output" becomes an unspent output is if the transaction that spent it either becomes invalid or is dropped from the mempool to free up space for newer transactions. It doesn't become an unspent output when it gets included in the blockchain. It remains spent.


This is incorrect.  An input doesn't become an output.  It remains an input.  Outputs that haven't been spent yet can be referenced in a future input, but inputs don't get referenced, or removed, or change state.


This is also incorrect.  Inputs exist in transactions.  Transactions are recorded in the blockchain.  The inputs don't disappear when the transaction is added to the blockchain. They remain a part of the transaction itself.

I intended to mean what you have stated. My answer was the layman version.
I didn't want to go into the output is an input is an output, depending on its status
at a certain point in time.

I guess when I said "spent transaction", I should have said "spent transaction output"
(which is the input to the "unconfirmed transaction"), and when that gets included into
the blockchain, becomes an unspent output itself. What I mean is, that the input
aspect of the tx is always transient since its existence is only to become an output (or
be rejected). There are spent and unspent outputs, but not spent or unspent inputs, since
inputs only exists within the mempool, which is temporary. At least that is how I see it.

But if there is something in my statement that is contradictory or wrong or I am wrong in
general, I will correct it.

I'm not sure if AgentofCoin is misunderstanding the process, or if I am just misunderstanding what AgentofCoin is saying however...


Transactions are made up of inputs and outputs.

Inputs are references to outputs of earlier transactions. They meet the requirements that the output was encumbered with, and they supply value to the transaction.

Outputs encumber that value with requirements that must be met for that output to be used as an input to another transaction.

The UTXO is a list of the outputs which a node has not yet seen used as an input.  When an output in the UTXO is seen as an input to a transaction, the node removes that otuput from its UTXO so that any other transaction trying to use it as an input will be identified as invalid.

The mempool is a list of the transactions that a node has heard about from a peer, but which is not in the node's blockchain.  When the transaction shows up in the node's blockchain, it is removed from the node's mempool.  It also may be removed from the mempool to free up space for newer transactions if too much time has passed without being confirmed.

Good reply.  Thanks!

A UTXO is an "unspent transaction output", and is what "bitcoins' are after being confirmed and not moving.
A mempool tx is a "spent transaction, and are "bitcoins" that are still pending for confirmation.

 - UTXOs in theory never leave the UTXO set, but are divided and consolidated over time.

 - Mempool Txs are either dropped when they are confirmed or dropped after 72 hours of no confirmation.
(the 72 hour limit is based on the standard node code, but nodes can choose their own preference).



In theory, UTXOs are never "removed", only mempool txs can be "removed".

The spent output is not removed from the UTXO until it's associated mempool tx has been confirmed.
When the blockchain is updated, the old UTXO is overwritten into either consolidation (added to an
already existing UTXO), division (split into multiple new UTXOs) or a single new UTXO.


EDIT: I was mostly incorrect, see Danny's answer below.

Hi there, new to Bitcoin and just had a clarification question.

When does a transaction leave the UTXO and when does a transaction leave the memory pool?

From my understanding, a validating node receives a transaction and is added to node's mem. pool. The node looks up that the inputs are valid in the UTXO. If input exists in the UTXO, the txn is removed from the UTXO. Once a block arrives at the node (or if the node is a miner and solves a block) and the block is validated, the transaction is removed from the transaction memory pool.

Is this true? I'm just kinda unsure about at what point txns are removed from UTXO vs memory pool.

Thanks!!