Topic: Bitcoin Mempool Mismatch Between Nodes (Read 674 times)

I'm now noting that my nodes running core V 20.2 now shows over 1 gigabyte in each of their local mempools while mempool dot space shows 1.74 Gigs.  That should mean I'm seeing the majority of the mempool dot spaces mempool however this should not be possible as also according to mempool dot space more than 50% of the current mempool is taproot transactions, none of which can be seen by my nodes.  Something seems amiss here. It should also be noted that among my nodes the discrepancy in memory usage and TX count continues amongst them unabated since Taproot's introduction.

No, it will eliminate them from YOUR mempool. It's going to do absolutely nothing for the mempool in general.
And since mining pools (with one exception) and miners are trying to make as much BTC as possible they are going to want those TXs and will be running software that will allow them to get them.

All you are doing is running old outdated software with other potential vulnerabilities that does not support newer features.

-Dave

We discussed this back on the first page of this thread. Taproot support was first added in 0.21.0, so if you are running 0.20.x then you will ignore all taproot transactions.

If you wish to filter ordinal transactions, there are other ways to do this without running software which is several years out of date.

Update:

My node is not seeing any taproot transactions whatsoever and as per mempool dot space, all the inscriptions transactions that it's seeing is in fact taproot.  That said, it's clear
that running v. 20.2 is eliminating the ordinals spam + any taproot traffic from the mempool.

Ok, so after continuing though and checking dozens of transactions labeled as inscriptions on mempool dot space and cross checking them with my node and blockstream dot com, I can now safely say that my version of Bitcoin core does not seem to be seeing these inscription transactions while they're in the mempool but only after they are confirmed by the miners.  That being the case, if you're like me and you don't want to perpetuate these inscriptions, you might just want to downgrade to core v. 20.2 as it certainly seems to be filtering out these inscriptions from nodes mempools.  Also, if I'm not mistaking, this likely explains the reason for the differences between my nodes mempool sizes and transactions as from the mempool dot space and blockstream dot info as they are likely running the very latest versions of bitcoin core and therefore propagating these transactions.  Merry Christmas everyone and may everyone have a happy (no ordinals or inscriptions) New Year!   

Hello fellow Bitcoiners,

I hope you're all having a good holiday season so far!  I'm replying back to try to further look into this issue with the memory mismatch between nodes and to try to dig deeper into my understanding of how bitcoin transactions propagate on the network and how they should be able to be interpreted by our nodes assuming everything is operating normally while in the mempool.  As you may recall I've been quite skeptical of how things have been operating lately since the ordinals and inscriptions were released as this is when I've noticed these memory discrepancies first arise and I've been running several nodes for years.

Either way I wanted to compare what my node sees (Windows 11 based node running version 20.2 currently with a French IP address) to what both mempool dot space and blockstream dot info sees.  With the new goggle feature recently added to mempool dot space allowing folks to quickly identify the types of transactions that are currently pending in the mempool (I'm hopeful they'll also allow this feature to explore previous blocks in the future), I've been trying to see where the discrepancies are.  With that said, I've been looking into the various types of TXNS in it's mempool, then cross checking them with with both my node's mempool vs. blockstream dot info's mempool.  Oddly enough (or what seems that way to me) is that while I've been continuing to see a huge discrepancy in my mempool memory usage on my node and pending TXNS (mines currently 580.75 MB with ~ 88457 pending TXNS vs. mempool dot spaces 316,391 TXNS and 1.51 GB as of ~ 1735 UTC today) it seems as though there are specific types of TXNS that I'm not seeing as noted below:


That said, the few below transactions were both in mempool dot space as well as in blockstream dot info's mempools had the following results in my mempool:

17:22:01
 
getrawtransaction 7d5db5b07609af7432fec8c395c3330e730f6fb9f36d8cf61cb56addeec0240f   - This was flagged as an incription TX)


17:22:01
 
No such mempool or blockchain transaction. Use gettransaction for wallet transactions. (code -5)


17:23:00
 
gettxout d71afc2f7e688798b7af5860a9730325ddec2c35db5a526f91e5fc4590d50a92 ,0 (this TXN showed up in my mempool as well as can be seen below)


17:23:00
 
{
  "bestblock": "00000000000000000002088865449c4fafb0e05d498254429b7b71cd13ffb79b",
  "confirmations": 0,
  "value": 0.37011741,
  "scriptPubKey": {
    "asm": "OP_HASH160 3354fa2425aa222086c0ac4df2e3cf38f69e27cb OP_EQUAL",
    "hex": "a9143354fa2425aa222086c0ac4df2e3cf38f69e27cb87",
    "reqSigs": 1,
    "type": "scripthash",
    "addresses": [
      "36NSBe6tr6rAYnKrj5tHnTGNLUdJ7wFd7q"
    ]
  },
  "coinbase": false
}


17:23:42
 
gettxout 01b75016c9a58675b333a38e8eaa495b5d1643bae2aefe2453dc36010dad2e40 ,0 (this was flagged in the mempool dot space as a coinjoin and only showed up after it was confirmed by the network in my node)


17:23:42
 
null

I should also likely point out my current node config file for this node is set as follows:

prune=0
txindex=1
dbcache=5000
datadir=E:\Bitcoin
assumevalid=0
peerbloomfilters=1
port=33985
maxmempool=1500
permitbaremultisig=0

That all being the case, I'm unsure why the mempool size continues on as my computer has more than adequate resources to see the full mempool.  I should also note that another of my nodes running on Windows 10 (also version 20.2) sees a similar amount of memory and TXNS that my French IP nodes sees).  That all being said, any feedback from you related to why the discrepancy in size and why there are missing transactions would be much appreciated.

I really hate to think this is some kind of attack but considering the legal and technical turmoil of late nothing would surprise me.  Thanks in advance!

Didn't (and don't) know who your provider is or what your IP is, but it's a common issue with a lot of VPNs. Public IP4 IPs are expensive. With the IP4 address space exhausted $35+ per IP is common.

So it's not like they can just keep throwing new IPs into the mix. So blocks of IPs get blacklisted and then issues start to come up because even if they are doing a 1 < -> 1 IP for your VPN which most places don't, the last person on that IP could have been an ass and killed it's reputation. And as I said if it's a shared public IP like 90%+ of the providers then it's even worse. Since you don't know what other people on that IP are doing.

-Dave

Good morning everyone,

@ o_e_l_e_o,

I appreciate your explanation and the breakdown of the quirks of the mempool / mining process.  Also I didn't realize that the inbound and outbound connections were functionally identical but it makes sense when I think about it so thank you for that comment. 

@ DaveF,

Yes my computer has more RAM than that but I'm running the blockchain on an external HD that's connected via USB 3.1.  Also I believe that I am properly sending data to the bitcoin network behind the VPN as certain gateways for our VPN provider do allow for port forwarding which I've poked a hole in my router to open.  In fact over the last 20 hours or so it appears that my node has sent over 5GB of data.  At one point I had 37 inbound connections behind the VPN (now it's down to about 1/3 of that however).

Also just an FYI I did notice that the same issue with the block propagation delay to my node (behind the VPN) happened again as I waited for another transaction I sent shortly after I posted.  This issue however seems to happen fairly regularly as this has happened several times over at least the last year or two I've been spot checking with both with my node in the clear and the one running behind the VPN connection.  Regardless though everything seems to be synched up at this point. Either way I appreciate your feedback. Curious though, how did you know what my VPN provider was?  I'm assuming you seen the IP and looked it up but how did you get the IP?  Thanks.

Could be the VPN. Remember, most VPN endpoints are abused and their IPs do get blocked even if it's not your fault.
I use the same VPN provider you do and have a malicious node, my node does not get blocked my IP does, which just so happens to be the same as yours.
Your node shows a connection but it's really not sending anything.
Have seen it now and then.

Are you running on at least an SSD with 8gb ram and a processor that is less then 10 years old?
Could also be hardware that can not keep up churning away on something.

-Dave

Either you wait for the mempool to empty, or as DaveF says, you increase the fee and your transaction will evict some other lower paying transaction from these nodes' mempools.

10 outbound connections by default, but up to 115 inbound connections by default. Note that apart from the 2 block-relay connections, outbound and inbound connections are functionally identical - it's only the way the connections are established which are different.

It isn't. Anyone can increase their transaction's fees and gain access to a mempool which is already at its limit by evicting other transactions. Since mempools are run locally, it is always in miners' best interests to do this, since it allows them to maximize their profits.

Looking at the time this happened, you simply got unlucky with when blocks are mined. Keep in mind blocks are mined on average every 10 minutes, but this can vary from a few seconds to over an hour in reality.

Block 812,957 included transactions with a fee of 3 sats/vbyte. At this point, mempool.space was probably recommending a fee at around 5 sats/vbyte, and appropriately so. However, it then took 42 minutes to find the next block, by which point the minimum fee had increased to 24 sats/vbyte. Because of this backlog which was created during this 42 minute time period, it then took the network to block 812,970 to mine transactions at 5 sats/vbyte and include your transaction in a block. This had nothing to do with mempools being full - indeed, if nodes were rejecting your transaction then it would never have been mined at all. This is simply a quirk of how bitcoin works and it happens all the time.

Wow this is getting even stranger.  The good news is that my transaction did just confirm.  But the weird part about it was that my electrum wallet (it's connected to it's own default nodes not mine) showed that my transaction was confirmed about 10 minutes before my node seen it.  I wonder what would cause such a propagation delay to my node?  It is behind a VPN but it does have inbound connections as well.  Anyway I'm glad to see everything has settled.

Thanks again to everyone who replied to this message thread and helped me understand this process that much better!

All,

I did transmit a transaction from my node (version 20.2) and I can see that there's been 6 blocks added to the blockchain since my transmission was sent.  The odd thing is that there has been also at least 2 retransmissions of my transaction according to my node's log.  I'm starting to think this can be a bigger deal.

Note that this fee was ~5.39 sats per byte and I believe that it was within the 20 minute expected range according to mempool dot space.  Yet it's been almost 2 hours now and nothing but attempted retransmissions from my node. Any thoughts on this would be much appreciated.  Thanks.

Yep, the maximum of outgoing connections is 10 for now. We discussed this today in this thread.
8 of them are full-relay and two additional block-relay connections.

To ur concern: When u are sending/broadcasting a new transaction from ur own node this TXID will propgated to the nodes that already connected to ur node. From there it will move forward to other nodes in the network.
If this transaction will not be forwarded to the miners mempool due to cap-limit of the mempool it needs to rebroadcasted again once it cleared or you have to adjust your fee.

I wouldn't necessarily be so paranoid about that because the Bitcoin network is designed to be decentralized. That's the strength of Bitcoin and that's what makes it special. Most bitcoiners resist as best they can when it comes to centralization.

If your fee is higher then the fees that in the TXs being dropped then you will just push a lower fee TX out of their mempool.

It's not something you should worry about. There are also a lot of well connected nodes and services that will broadcast your TX

https://live.blockcypher.com/btc/pushtx/

You don't even need to be running core, you can build your TX manually using python (and other code too) and push it out through a variety of ways.

-Dave

Thanks o_e_l_e_o.  I appreciate your feedback.  What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?   Maybe my transaction never gets to a miner's mempool since it's already full how then would it get confirmed?  I think nowadays the default amount of nodes that your node connects to is only 10, right? What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?  Anyway I really hope I'm just being paranoid and ignorant of the technical operations here but this seems quite odd to me.

300 MB will likely cause a mismatch for some time, yes. The average memory usage of nodes' mempools has been above 300 MB pretty consistently for the last 6 months. Given this, two nodes both with the default limits will be dropping different transactions at different times and therefore will always have a mismatch between their mempools.

There is only a possible attack vector if you start accepting unconfirmed transactions, which has never been safe. If you wait for any incoming transactions to receive at least a couple of confirmations, then whether or not those transactions were in the mempool of one, both, or neither of your nodes prior to being confirmed in a block is irrelevant.

Hi all,

Yes my issue is resolved in respect to the hang up I experienced after Clicking on Settings> Options > Reset Options .  As I noted in my other thread I added the prune=0 option in the config file.  After doing so however I see that two different options I set in my configuration file there was a message that said:

Options set in the dialog are overridden by the command line or in the configuration file:
-prune=0 -dBcache=5000


The funny thing is that prior to the other thread I never had added the prune option to the option file at all. 

To summarize what I've noted so far with this particular issue however is that that there are apparently 2 different config files that are created with Bitcoin Core v.20.2 (maybe others too?) since when clicking Settings> Options > Open Configuration File the GUI button launches one, but that one isn't the one the system reads on start up.  The startup one is the one that needs to be modified and is stored in the %APPDATA%\Bitcoin directory.

Going forward, I'm curious to know if the 300 MB default limit is going to be adequate for the nodes going forward if there's constant mismatches between nodes when the default mempools being set to 300 MB.  Do anyone here believe this is a serious issue?  Perhaps this can somehow been an attack vector of sorts?  If so hopefully the core team is already looking into this.  Regardless I really do appreciate everyone's feedback here.

Fixing that problem will allow him to launch his node since your cannot use tx-index=1 with a pruned node.

That won't make any difference to his mempools, though. Whether or not the node is pruned or not, or whether the node is using tx-index=0 or =1, should have no bearing at all on the size of its mempool.

I think that his problem is already fixed. Correct me if I am wrong but he decided to reset his options via GUI.
He used the wrong data directory for his config file since he stored it on a external hard drive.
Refering to this post in another thread: https://bitcointalksearch.org/topic/m.63005361

He was using the pruning mode and txindex at the same time which caused that conflict. Since he disabled pruning he should be fine?!

You'll probably need to wait longer than that.

As I explained earlier in this thread, nodes do not automatically rebroadcast transactions, nor do they actively try to fetch transactions they don't know about from other nodes. They simply relay transactions (if valid) the first time they hear about them. So if there is a discrepancy of say 5,000 transactions which one of your nodes knows about and the other doesn't, that discrepancy will only decrease if:

A) These transactions are mined
B) Your node which knows about these transactions drops them, either for exceeding the size limit or exceeding the time limit
C) Someone (most likely the creator of these transactions) rebroadcasts them and there is a path from their node to your node of nodes which don't already know about the transactions (if a node receives a transaction already in its mempool, it will not relay it again)

The best way to check for ongoing issues would be to either wipe your both your mempools or to sync them to each other, so they are both starting from the same point.