Topic: Bitcoin Mempool Mismatch Between Nodes (Read 671 times)

I'm now noting that my nodes running core V 20.2 now shows over 1 gigabyte in each of their local mempools while mempool dot space shows 1.74 Gigs.  That should mean I'm seeing the majority of the mempool dot spaces mempool however this should not be possible as also according to mempool dot space more than 50% of the current mempool is taproot transactions, none of which can be seen by my nodes.  Something seems amiss here. It should also be noted that among my nodes the discrepancy in memory usage and TX count continues amongst them unabated since Taproot's introduction.

No, it will eliminate them from YOUR mempool. It's going to do absolutely nothing for the mempool in general.
And since mining pools (with one exception) and miners are trying to make as much BTC as possible they are going to want those TXs and will be running software that will allow them to get them.

All you are doing is running old outdated software with other potential vulnerabilities that does not support newer features.

-Dave

We discussed this back on the first page of this thread. Taproot support was first added in 0.21.0, so if you are running 0.20.x then you will ignore all taproot transactions.

If you wish to filter ordinal transactions, there are other ways to do this without running software which is several years out of date.

Update:

My node is not seeing any taproot transactions whatsoever and as per mempool dot space, all the inscriptions transactions that it's seeing is in fact taproot.  That said, it's clear
that running v. 20.2 is eliminating the ordinals spam + any taproot traffic from the mempool.

Ok, so after continuing though and checking dozens of transactions labeled as inscriptions on mempool dot space and cross checking them with my node and blockstream dot com, I can now safely say that my version of Bitcoin core does not seem to be seeing these inscription transactions while they're in the mempool but only after they are confirmed by the miners.  That being the case, if you're like me and you don't want to perpetuate these inscriptions, you might just want to downgrade to core v. 20.2 as it certainly seems to be filtering out these inscriptions from nodes mempools.  Also, if I'm not mistaking, this likely explains the reason for the differences between my nodes mempool sizes and transactions as from the mempool dot space and blockstream dot info as they are likely running the very latest versions of bitcoin core and therefore propagating these transactions.  Merry Christmas everyone and may everyone have a happy (no ordinals or inscriptions) New Year!   

Hello fellow Bitcoiners,

I hope you're all having a good holiday season so far!  I'm replying back to try to further look into this issue with the memory mismatch between nodes and to try to dig deeper into my understanding of how bitcoin transactions propagate on the network and how they should be able to be interpreted by our nodes assuming everything is operating normally while in the mempool.  As you may recall I've been quite skeptical of how things have been operating lately since the ordinals and inscriptions were released as this is when I've noticed these memory discrepancies first arise and I've been running several nodes for years.

Either way I wanted to compare what my node sees (Windows 11 based node running version 20.2 currently with a French IP address) to what both mempool dot space and blockstream dot info sees.  With the new goggle feature recently added to mempool dot space allowing folks to quickly identify the types of transactions that are currently pending in the mempool (I'm hopeful they'll also allow this feature to explore previous blocks in the future), I've been trying to see where the discrepancies are.  With that said, I've been looking into the various types of TXNS in it's mempool, then cross checking them with with both my node's mempool vs. blockstream dot info's mempool.  Oddly enough (or what seems that way to me) is that while I've been continuing to see a huge discrepancy in my mempool memory usage on my node and pending TXNS (mines currently 580.75 MB with ~ 88457 pending TXNS vs. mempool dot spaces 316,391 TXNS and 1.51 GB as of ~ 1735 UTC today) it seems as though there are specific types of TXNS that I'm not seeing as noted below:


That said, the few below transactions were both in mempool dot space as well as in blockstream dot info's mempools had the following results in my mempool:

17:22:01
 
getrawtransaction 7d5db5b07609af7432fec8c395c3330e730f6fb9f36d8cf61cb56addeec0240f   - This was flagged as an incription TX)


17:22:01
 
No such mempool or blockchain transaction. Use gettransaction for wallet transactions. (code -5)


17:23:00
 
gettxout d71afc2f7e688798b7af5860a9730325ddec2c35db5a526f91e5fc4590d50a92 ,0 (this TXN showed up in my mempool as well as can be seen below)


17:23:00
 
{
  "bestblock": "00000000000000000002088865449c4fafb0e05d498254429b7b71cd13ffb79b",
  "confirmations": 0,
  "value": 0.37011741,
  "scriptPubKey": {
    "asm": "OP_HASH160 3354fa2425aa222086c0ac4df2e3cf38f69e27cb OP_EQUAL",
    "hex": "a9143354fa2425aa222086c0ac4df2e3cf38f69e27cb87",
    "reqSigs": 1,
    "type": "scripthash",
    "addresses": [
      "36NSBe6tr6rAYnKrj5tHnTGNLUdJ7wFd7q"
    ]
  },
  "coinbase": false
}


17:23:42
 
gettxout 01b75016c9a58675b333a38e8eaa495b5d1643bae2aefe2453dc36010dad2e40 ,0 (this was flagged in the mempool dot space as a coinjoin and only showed up after it was confirmed by the network in my node)


17:23:42
 
null

I should also likely point out my current node config file for this node is set as follows:

prune=0
txindex=1
dbcache=5000
datadir=E:\Bitcoin
assumevalid=0
peerbloomfilters=1
port=33985
maxmempool=1500
permitbaremultisig=0

That all being the case, I'm unsure why the mempool size continues on as my computer has more than adequate resources to see the full mempool.  I should also note that another of my nodes running on Windows 10 (also version 20.2) sees a similar amount of memory and TXNS that my French IP nodes sees).  That all being said, any feedback from you related to why the discrepancy in size and why there are missing transactions would be much appreciated.

I really hate to think this is some kind of attack but considering the legal and technical turmoil of late nothing would surprise me.  Thanks in advance!

Didn't (and don't) know who your provider is or what your IP is, but it's a common issue with a lot of VPNs. Public IP4 IPs are expensive. With the IP4 address space exhausted $35+ per IP is common.

So it's not like they can just keep throwing new IPs into the mix. So blocks of IPs get blacklisted and then issues start to come up because even if they are doing a 1 < -> 1 IP for your VPN which most places don't, the last person on that IP could have been an ass and killed it's reputation. And as I said if it's a shared public IP like 90%+ of the providers then it's even worse. Since you don't know what other people on that IP are doing.

-Dave

Good morning everyone,

@ o_e_l_e_o,

I appreciate your explanation and the breakdown of the quirks of the mempool / mining process.  Also I didn't realize that the inbound and outbound connections were functionally identical but it makes sense when I think about it so thank you for that comment. 

@ DaveF,

Yes my computer has more RAM than that but I'm running the blockchain on an external HD that's connected via USB 3.1.  Also I believe that I am properly sending data to the bitcoin network behind the VPN as certain gateways for our VPN provider do allow for port forwarding which I've poked a hole in my router to open.  In fact over the last 20 hours or so it appears that my node has sent over 5GB of data.  At one point I had 37 inbound connections behind the VPN (now it's down to about 1/3 of that however).

Also just an FYI I did notice that the same issue with the block propagation delay to my node (behind the VPN) happened again as I waited for another transaction I sent shortly after I posted.  This issue however seems to happen fairly regularly as this has happened several times over at least the last year or two I've been spot checking with both with my node in the clear and the one running behind the VPN connection.  Regardless though everything seems to be synched up at this point. Either way I appreciate your feedback. Curious though, how did you know what my VPN provider was?  I'm assuming you seen the IP and looked it up but how did you get the IP?  Thanks.

Could be the VPN. Remember, most VPN endpoints are abused and their IPs do get blocked even if it's not your fault.
I use the same VPN provider you do and have a malicious node, my node does not get blocked my IP does, which just so happens to be the same as yours.
Your node shows a connection but it's really not sending anything.
Have seen it now and then.

Are you running on at least an SSD with 8gb ram and a processor that is less then 10 years old?
Could also be hardware that can not keep up churning away on something.

-Dave

Either you wait for the mempool to empty, or as DaveF says, you increase the fee and your transaction will evict some other lower paying transaction from these nodes' mempools.

10 outbound connections by default, but up to 115 inbound connections by default. Note that apart from the 2 block-relay connections, outbound and inbound connections are functionally identical - it's only the way the connections are established which are different.

It isn't. Anyone can increase their transaction's fees and gain access to a mempool which is already at its limit by evicting other transactions. Since mempools are run locally, it is always in miners' best interests to do this, since it allows them to maximize their profits.

Looking at the time this happened, you simply got unlucky with when blocks are mined. Keep in mind blocks are mined on average every 10 minutes, but this can vary from a few seconds to over an hour in reality.

Block 812,957 included transactions with a fee of 3 sats/vbyte. At this point, mempool.space was probably recommending a fee at around 5 sats/vbyte, and appropriately so. However, it then took 42 minutes to find the next block, by which point the minimum fee had increased to 24 sats/vbyte. Because of this backlog which was created during this 42 minute time period, it then took the network to block 812,970 to mine transactions at 5 sats/vbyte and include your transaction in a block. This had nothing to do with mempools being full - indeed, if nodes were rejecting your transaction then it would never have been mined at all. This is simply a quirk of how bitcoin works and it happens all the time.

Wow this is getting even stranger.  The good news is that my transaction did just confirm.  But the weird part about it was that my electrum wallet (it's connected to it's own default nodes not mine) showed that my transaction was confirmed about 10 minutes before my node seen it.  I wonder what would cause such a propagation delay to my node?  It is behind a VPN but it does have inbound connections as well.  Anyway I'm glad to see everything has settled.

Thanks again to everyone who replied to this message thread and helped me understand this process that much better!

All,

I did transmit a transaction from my node (version 20.2) and I can see that there's been 6 blocks added to the blockchain since my transmission was sent.  The odd thing is that there has been also at least 2 retransmissions of my transaction according to my node's log.  I'm starting to think this can be a bigger deal.

Note that this fee was ~5.39 sats per byte and I believe that it was within the 20 minute expected range according to mempool dot space.  Yet it's been almost 2 hours now and nothing but attempted retransmissions from my node. Any thoughts on this would be much appreciated.  Thanks.

Yep, the maximum of outgoing connections is 10 for now. We discussed this today in this thread.
8 of them are full-relay and two additional block-relay connections.

To ur concern: When u are sending/broadcasting a new transaction from ur own node this TXID will propgated to the nodes that already connected to ur node. From there it will move forward to other nodes in the network.
If this transaction will not be forwarded to the miners mempool due to cap-limit of the mempool it needs to rebroadcasted again once it cleared or you have to adjust your fee.

I wouldn't necessarily be so paranoid about that because the Bitcoin network is designed to be decentralized. That's the strength of Bitcoin and that's what makes it special. Most bitcoiners resist as best they can when it comes to centralization.

If your fee is higher then the fees that in the TXs being dropped then you will just push a lower fee TX out of their mempool.

It's not something you should worry about. There are also a lot of well connected nodes and services that will broadcast your TX

https://live.blockcypher.com/btc/pushtx/

You don't even need to be running core, you can build your TX manually using python (and other code too) and push it out through a variety of ways.

-Dave

Thanks o_e_l_e_o.  I appreciate your feedback.  What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?   Maybe my transaction never gets to a miner's mempool since it's already full how then would it get confirmed?  I think nowadays the default amount of nodes that your node connects to is only 10, right? What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?  Anyway I really hope I'm just being paranoid and ignorant of the technical operations here but this seems quite odd to me.

300 MB will likely cause a mismatch for some time, yes. The average memory usage of nodes' mempools has been above 300 MB pretty consistently for the last 6 months. Given this, two nodes both with the default limits will be dropping different transactions at different times and therefore will always have a mismatch between their mempools.

There is only a possible attack vector if you start accepting unconfirmed transactions, which has never been safe. If you wait for any incoming transactions to receive at least a couple of confirmations, then whether or not those transactions were in the mempool of one, both, or neither of your nodes prior to being confirmed in a block is irrelevant.

Hi all,

Yes my issue is resolved in respect to the hang up I experienced after Clicking on Settings> Options > Reset Options .  As I noted in my other thread I added the prune=0 option in the config file.  After doing so however I see that two different options I set in my configuration file there was a message that said:

Options set in the dialog are overridden by the command line or in the configuration file:
-prune=0 -dBcache=5000


The funny thing is that prior to the other thread I never had added the prune option to the option file at all. 

To summarize what I've noted so far with this particular issue however is that that there are apparently 2 different config files that are created with Bitcoin Core v.20.2 (maybe others too?) since when clicking Settings> Options > Open Configuration File the GUI button launches one, but that one isn't the one the system reads on start up.  The startup one is the one that needs to be modified and is stored in the %APPDATA%\Bitcoin directory.

Going forward, I'm curious to know if the 300 MB default limit is going to be adequate for the nodes going forward if there's constant mismatches between nodes when the default mempools being set to 300 MB.  Do anyone here believe this is a serious issue?  Perhaps this can somehow been an attack vector of sorts?  If so hopefully the core team is already looking into this.  Regardless I really do appreciate everyone's feedback here.

Fixing that problem will allow him to launch his node since your cannot use tx-index=1 with a pruned node.

That won't make any difference to his mempools, though. Whether or not the node is pruned or not, or whether the node is using tx-index=0 or =1, should have no bearing at all on the size of its mempool.

I think that his problem is already fixed. Correct me if I am wrong but he decided to reset his options via GUI.
He used the wrong data directory for his config file since he stored it on a external hard drive.
Refering to this post in another thread: https://bitcointalksearch.org/topic/m.63005361

He was using the pruning mode and txindex at the same time which caused that conflict. Since he disabled pruning he should be fine?!

You'll probably need to wait longer than that.

As I explained earlier in this thread, nodes do not automatically rebroadcast transactions, nor do they actively try to fetch transactions they don't know about from other nodes. They simply relay transactions (if valid) the first time they hear about them. So if there is a discrepancy of say 5,000 transactions which one of your nodes knows about and the other doesn't, that discrepancy will only decrease if:

A) These transactions are mined
B) Your node which knows about these transactions drops them, either for exceeding the size limit or exceeding the time limit
C) Someone (most likely the creator of these transactions) rebroadcasts them and there is a path from their node to your node of nodes which don't already know about the transactions (if a node receives a transaction already in its mempool, it will not relay it again)

The best way to check for ongoing issues would be to either wipe your both your mempools or to sync them to each other, so they are both starting from the same point.

There shouldn't be any further discrepancies because when you use such a giant mempool limit, it should capture all of the transactions without dropping any of them - especially the ones with ultra-low fees.

At one point a week or two ago, I looked at mempool statistics and saw that the mempool with all transactions in it was a whole gigabyte large. So as everyone else's mempool reaches breaking point, you might start noticing this issue temporarily.

Update:  After resolving the issue in my earlier post by editing the options file located here:  %APPDATA%\Bitcoin I realized that this was not in fact the same file that the GUI is pulling up when I clicked on the Settings>Options>Open Configuration File.  Perhaps this is due to the fact that I'm running my node with a non default path for the blocks directory on an external hard drive?  Either way I've been able to modify the correction options file (not the one the GUI pulls ups but the one here: %APPDATA%\Bitcoin and as soon as I did it correctly updated my maxmempool=600 value.  I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are.  Thanks again to all who gave their feedback here!

Hi o_e_l_e_o,

Unfortunately I just hit another stumbling point. I tried clicking the "reset options" button which when I did it restarted my node and then tried to relaunch it. I've then started another thread to try to recover from that issue which I also posted in this message board.  Hopefully I can figure out how to get this node back online after the issue I'm having now so I can resume troubleshooting this maxmempool issue.

Odd. You could try copying your .conf file from the node which does have the 600 MB limit and pasting it over the .conf file for the node with the limit of 300 MB to see if that works. I don't use Windows, but often files having the wrong permissions can mess things up in Windows. Could this be the case here?

Also check you aren't launching with any command line options. These would override anything in your .conf file.

Unfortunately, it's common misconception among Bitcoiner. Ordinals is fully possible without Taproot since the arbitrary data is stored on witness data. Actual Taproot mess up is no longer have 10000 script size limit[1]. And even without Taproot and SegWit, what Ordinals do could be replicated using OP_RETURN, address/pubkeyhash, multisig address and others with few limitation.

[1] https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#user-content-Resource_limits.

Hello again everyone. Thanks again for all the replies.


 o_e_l_e_o,
 
Yes I believe that the file is in the correct directory as I've opened it via the method you suggested and edited that file every time. I've also checked for spelling or other issues and haven't seen any.

mvdheuvel1983,

The first thing I'd do is bring the issue here to see if someone smarter than me might have a clue as to properly troubleshoot the exploit and then perhaps have a public dialog to brainstorm the issue.

All,

I'm honestly not interested in running any of the recent versions as I don't support taproot. Reason being is that it appears there's a consensus that this has actually made the ordinals and inscriptions possible which seems to directly correlate to the mismatches I've been seeing between my nodes.

Thanks.

I read through the thread and although I am not able to give my opinion to your question, this last paragraph got me. I want to ask that assuming this were some kind of attack as you have mentioned what are some of the measures you would have implemented to counteract it. What proactive steps would you have taken to not only halt the ongoing attack but also to establish safeguards that prevent its recurrence in the future?

Good question, because 20.x does not support Taproot at all. Which means, Taproot transactions will not be visible in your mempool.

Are you sure you've not mistyped anything? It's not commented out, or set only for testnet? And are you sure your bitcoin.conf file is in the right location? Easiest way to check is to click on "Settings" -> "Open Configuration File" within the Core GUI.

Why not update to 25.0 though?

Have you considered the easiest possible solution to simple upgrade to a newer version of Bitcoin Core which might fix the issue by itself? The latest release is 25.0, so there is a lot of ground to catch. Is something keeping you from upgrading to a newer version of the software?   

Hi all,

@ o_e_l_e_o,

I’ve just run the command on both and while there is still a discrepancy, I see that for some reason even though the maxmempool of 600 is in both option files for both the v.20.1 and the v.20.2 version it seems as though the .2 version is not applying this 600 MB value and is sticking to the default 300.  I then checked what else I had in the two to compare apples to apples and then ensured to have only the same options enabled between the 20.1 and the 20.2 as to mimic the 20.1 that was seemingly applying the 600 MB value correctly.  Unfortunately this did not work as after rerunning the command I see that 20.2 is still stuck on the 300 MB default cap. 

@DaveF
I should also add that the computer running the .2 version is much more capable in terms of hardware and in fact has more ram than the other PC so there are no hardware limitations that should be coming into play here.

I next restarted the computer with the 20.2 version and again it’s still stuck with the 300 MB cap even though the following options are enabled in the options file:

If anyone knows why this could be happening in this 20.2 version I’d love to hear your feedback.  Perhaps this is a bug on this version?

You don't need to use any tricks in address bar. Just click on that last level of 0-1 range on that web page, and you will be redirected there. In exactly the same way you can click on other ranges, and you will be redirected to fees on that level or above. For example, if you click on 10-12 range, you will be redirected here: https://mempool.jhoenicke.de/#BTC,all,weight,10

Did not know you could append a number like that to have the minimum fee displayed set to that level. That's a neat trick!

Of course. We always had them, in some older versions, transactions were free, and some people still try to use them in that way. There are many transactions in range from zero to one satoshi per virtual byte. Some of them are just a result of mining pools, sending their own internal transactions for free, and including them in their own blocks (while also broadcasting them publicly), some of them are used in some off-chain protocols, like decentralized sidechain experiments, mining in LN experiments, or mempool-level-messaging experiments. Because nothing stops you from creating a group of nodes, that will use full-RBF, and increment fees one satoshi at a time, until reaching the final replacement of one satoshi per virtual byte, when all other nodes also try to process that batched result.

https://mempool.jhoenicke.de/#BTC,all,weight,0

Look at the bottom of the chart. That grey block can show you, what happens in 0-1 satoshis per virtual byte range. And if you have your own full node, even if it is pruned, you can directly receive that kind of traffic, if you change your configuration to allow that.

If you switch Johoe's to "BTC" (rather than "BTC (default mempool)") and sort by "count" rather than "weight", you should get a number which is roughly similar to mempool.space.

"BTC (default mempool)" shows the mempool with the 300 MB memory limit enforced. "BTC" shows the mempool with a much higher limit (although I'm not sure exactly how high he sets the limit). Although at the moment on the BTC/count graph, it says ~26k transactions at 1 sat/vbyte or more, but ~30k total transactions. I'm not sure what explains that deficit - transactions paying under 1 sat/vbyte?

Speaking of which, I have wondered something similar regarding the different mempools. As I am writing this, the one on Mempool.space shows 24k unconfirmed transactions. At the same time, Johoe's mempool has over 52k of unconfirmed transactions. It's not a small difference, it's more than the double. Could this really be about not being connected to the same peers, purging transactions of different sat values, or something else?

Scratch that.
One shows the weight of transactions, the other the total number of them. My bad.

Is there any reason you are on the 20.x instead of 24 or 25?

Since they are both on the same LAN you should be able to add the nodes to each others peer list.
Most VPNs will only deal with connections going to outside your LAN, otherwise you would not be able to print to a network printer or see network shares and so on when running a VPN.

How is the hardware that they are running on? About the same in terms of CPU / RAM / Drive?

-Dave

There have still been times over the last week where your nodes' mempools will have exceeded 600 MB, and as I explained above the exact point this happens will be different for both your nodes.

Use getmempoolinfo again and compare the number of transactions in each mempool ("size"). Has this gap closed at all?

The next thing I would try would be to delete the mempool for both nodes, start them up simultaneously, let them both run for a couple of days, and then check if the discrepancy is still there. Also, if it is the node behind the VPN which is constantly behind, have you checked it isn't intermittently losing connection, the VPN intermittently disconnecting, or the VPN blocking any traffic?

Okay this is seemingly a persistent issue even after I increased the default mempool to 600 MB for both over a week ago. As of this message I am still getting substantial mismatched between nodes whereas my node running version 20.1 is seeing a much larger memory usage then my 20.2 version. The even stranger thing is that the 20.1 which is the node that has inbound connections and was started several hours after the 20.2 behind the VPN yet it has a much larger size than the other (303 MB vs. 280 MB currently).

Something really seems off here and I'm hoping that someone here can help me understand why this is happening seemingly since around the time or the ordinals and inscriptions. Prior to this I have never seen a discrepancy before and I have been running the nose for years. Thanks in advance by any support!

Inbound/outbound connections won't make a difference here, provided both nodes have a good number of connections. Your nodes will receive unconfirmed transactions over both types of connection just the same.

Note that nodes won't try to fetch any transactions missing from their mempool. Once they drop a transaction, they won't learn about it again unless it is rebroadcast. If they receive a transaction which they do not add to their mempool due to it exceeding the size limit, then again, they won't learn about it unless it is rebroadcast. Given that the mempool has only recently come back down below the 300 MB default limit, if there was a delta between your nodes then this delta will persist until the backlog of transactions is cleared. Transactions which one node knows about but the other doesn't won't propagate between your nodes unless they are manually rebroadcast.

If one of your nodes has lost connection, had poor connections, been temporarily offline, hit the default limits before the other one, and so on, then any transactions not added to its mempool during this time will simply not be known about and the deficit will persist.

Thanks for all the replies everyone.  


@n0nce ,

IMO opinion the mempool difference in size isn't due to the two different nodes getting the transactions at slightly different times as they have been having a very similar discrepancy for the last several months and their sizes have been quite different for most (if not all) of that timeframe.  That in contrast to the few years I've been running them this way beforehand with no notable discrepancy in size other than the last two months and I don't recall changing any settings in either.



@o_e_l_e_o ,

Please note that my I did just upgrade y v 20.0 to 20.2 (the one behind the VPN) two days ago yet I'm still seeing the same discrepancy after over 48 hours of running them. Presently the 20.2 node running behind the VPN not accepting inbound connections has the following getmempoolinfo output:


{
  "loaded": true,
  "size": 5649,
  "bytes": 19164914,
  "usage": 95340400,
  "maxmempool": 300000000,
  "mempoolminfee": 0.00001000,
  "minrelaytxfee": 0.00001000
}

Vs. my 20.1 which is accepting inbounds:


{
  "loaded": true,
  "size": 13601,
  "bytes": 21971612,
  "usage": 107492160,
  "maxmempool": 300000000,
  "mempoolminfee": 0.00001000,
  "minrelaytxfee": 0.00001000
}


@DaveF ,

Technically they are both on the same LAN but one is behind a VPN connection not accepting inbound connections (the recently updated v20.2) and the other is open with inbound connections.

All,

I'm starting to wonder if perhaps this may be related to the transactions in the mempool being increased to higher than my nodes default 300 MB size and somehow my nodes picked different transactions to store locally when the levels dropped below that level however wouldn't of imagined this is possible as shouldn't they be prioritizing them the same way?  Either way from what I gather the mempool is below 300 MB now and the deltas between the two nodes continue to persist which would seem to invalidate my theory.

Also while I certainly appreciate all of the suggestions to try and correct the issue, the real reason for me bringing up this matter isn't to get my two nodes synched up so that they have the same mempool but to rather it's to understand why there is a discrepancy in the first place as this may somehow be some kind of attack that we aren't aware of presently that needs to be looked at (call me paranoid but we are competing with the most powerful and best funded organizations in the history of the planet, right?).  

Thanks all.

Are both these nodes on the same private internal network?
If so can you do an addnode command between the 2 of them?

They should at that point talk to each other and although the mempool will never be the same everything else being equal they really should be a lot closer.

-Dave

This could be more related to your devices rather than the mempools themselves.

Use getmempoolinfo on your two nodes and compare the outputs. "Size" is the number of transactions in your mempool, and "bytes" is the raw size of these transactions. I would expect these numbers to be fairly similar if your nodes are both running with similar mempool settings and similar uptimes.

"Usage" on the other hand, which is the 158 MB and 195 MB you are referring to, is the amount of RAM those unconfirmed transactions are using after they have been deserialized. This is also what the default 300 MB limit refers to, and not to the raw size of the transactions. The deserialized RAM usage will vary due to a number of factors external to the size of the mempool, such as the version of Core, the hardware of the system, the OS, and so on.

Your 2 nodes might just have very different sets of peers and thus get certain transactions earlier or later.
You could try logging the mempool size over the course of a few days and see if one is consistently smaller than the other or just at a certain point in time.

Another thing to try would be manually making the 2 machines peers, so they should exchange information between each other fairly frequently.

https://bitcoincore.org/en/doc/25.0.0/rpc/network/addnode/

Thanks @Findingnemo,

I just checked the config file on both nodes and oddly enough the only option that might be related was the dbcache value which was set to 10000 for the one with less transactions in it's mempool vs 5000 for the one with more.  Other than that there didn't appear to be any values in there that should affect the amount of memory for the mempool and I don't' even know if dbcache is even a value that does as it seems the one with the larger amount has less transactions that the other. I did however also note that the one with more transactions (v 20.0) was started ~ 5 hours earlier 11 days ago but I wouldn't imagine that would be an issue this much time into the future as I've never noticed it before the ordinals / incriptions spam and I've run nodes for years.


Either way I'm stumped here and hope there's nothing more sinister going on here. Thank you for the reply and hopefully others can chime in on this subject as well as it would be great to understand what's currently going on here. 

I don't know the exact reason for this but it may be due to the difference in the configuration setting, so check if there is any difference in mempool related setting using bitcoin.conf.

Possibly yes, because Stale blocks formed in the main chain but don't belong to the chain now so one of your nodes picked up the spam blocks and the other didn't pick those broadcasted from peers so the difference in numbers maybe the reason as well.

Still, let the experts of technical knowledge to find the actual reason.

Meanwhile, you can check this out that explains about stale blocks

I've noticed over the last few weeks that there's been a fairly notable discrepancy between my two nodes that are both running on windows with respect to the size of the mempool. Presently I'm running a version 20 and a 20.1 and note that there is about 195 MB of memory usage for the mempool on the older version versus 158 MB on the newer version. That really seemed odd to me considering that they both booted up on the same date (11 days ago). I've also noticed how fairly frequently the men pool has been exceeding 300 MB which is the default amount of memory that the nodes keep unconfirmed transactions in. Has anybody else out there noticed this discrepancy? Better yet, does anybody out there have an idea about why this might be happening or how?

I should probably add that version 20.1 is not operating behind a VPN and is allowing for inbound connections versus the one running version 20 that is behind a VPN and unable to receive inbound connection. That however was never an issue before the last several months. Perhaps it has something to do with the ordinals spamming?

Thanks in advance for any feedback.