Bitcoin Mempool Mismatch Between Nodes - page 2.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: BitcoinCanSaveUsAll on October 16, 2023, 01:05:52 PM

Update: After resolving the issue in my earlier post by editing the options file located here: %APPDATA%\Bitcoin I realized that this was not in fact the same file that the GUI is pulling up when I clicked on the Settings>Options>Open Configuration File. Perhaps this is due to the fact that I'm running my node with a non default path for the blocks directory on an external hard drive? Either way I've been able to modify the correction options file (not the one the GUI pulls ups but the one here: %APPDATA%\Bitcoin and as soon as I did it correctly updated my maxmempool=600 value. I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are. Thanks again to all who gave their feedback here!

There shouldn't be any further discrepancies because when you use such a giant mempool limit, it should capture all of the transactions without dropping any of them - especially the ones with ultra-low fees.

At one point a week or two ago, I looked at mempool statistics and saw that the mempool with all transactions in it was a whole gigabyte large. So as everyone else's mempool reaches breaking point, you might start noticing this issue temporarily.

BitcoinCanSaveUsAll

member

Activity: 104

Merit: 120

Update: After resolving the issue in my earlier post by editing the options file located here: %APPDATA%\Bitcoin I realized that this was not in fact the same file that the GUI is pulling up when I clicked on the Settings>Options>Open Configuration File. Perhaps this is due to the fact that I'm running my node with a non default path for the blocks directory on an external hard drive? Either way I've been able to modify the correction options file (not the one the GUI pulls ups but the one here: %APPDATA%\Bitcoin and as soon as I did it correctly updated my maxmempool=600 value. I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are. Thanks again to all who gave their feedback here!

BitcoinCanSaveUsAll

member

Activity: 104

Merit: 120

Hi o_e_l_e_o,

Unfortunately I just hit another stumbling point. I tried clicking the "reset options" button which when I did it restarted my node and then tried to relaunch it. I've then started another thread to try to recover from that issue which I also posted in this message board. Hopefully I can figure out how to get this node back online after the issue I'm having now so I can resume troubleshooting this maxmempool issue.

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Quote from: BitcoinCanSaveUsAll on October 15, 2023, 06:33:10 PM

o_e_l_e_o,

Yes I believe that the file is in the correct directory as I've opened it via the method you suggested and edited that file every time. I've also checked for spelling or other issues and haven't seen any.

Odd. You could try copying your .conf file from the node which does have the 600 MB limit and pasting it over the .conf file for the node with the limit of 300 MB to see if that works. I don't use Windows, but often files having the wrong permissions can mess things up in Windows. Could this be the case here?

Also check you aren't launching with any command line options. These would override anything in your .conf file.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: BitcoinCanSaveUsAll on October 15, 2023, 06:33:10 PM

All,

I'm honestly not interested in running any of the recent versions as I don't support taproot. Reason being is that it appears there's a consensus that this has actually made the ordinals and inscriptions possible which seems to directly correlate to the mismatches I've been seeing between my nodes.

Thanks.

Unfortunately, it's common misconception among Bitcoiner. Ordinals is fully possible without Taproot since the arbitrary data is stored on witness data. Actual Taproot mess up is no longer have 10000 script size limit[1]. And even without Taproot and SegWit, what Ordinals do could be replicated using OP_RETURN, address/pubkeyhash, multisig address and others with few limitation.

[1] https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#user-content-Resource_limits.

BitcoinCanSaveUsAll

member

Activity: 104

Merit: 120

Hello again everyone. Thanks again for all the replies.

o_e_l_e_o,

Yes I believe that the file is in the correct directory as I've opened it via the method you suggested and edited that file every time. I've also checked for spelling or other issues and haven't seen any.

mvdheuvel1983,

The first thing I'd do is bring the issue here to see if someone smarter than me might have a clue as to properly troubleshoot the exploit and then perhaps have a public dialog to brainstorm the issue.

All,

I'm honestly not interested in running any of the recent versions as I don't support taproot. Reason being is that it appears there's a consensus that this has actually made the ordinals and inscriptions possible which seems to directly correlate to the mismatches I've been seeing between my nodes.

Thanks.

mvdheuvel1983

sr. member

Activity: 1022

Merit: 368

Quote from: BitcoinCanSaveUsAll on October 03, 2023, 11:17:37 AM

Also while I certainly appreciate all of the suggestions to try and correct the issue, the real reason for me bringing up this matter isn't to get my two nodes synched up so that they have the same mempool but to rather it's to understand why there is a discrepancy in the first place as this may somehow be some kind of attack that we aren't aware of presently that needs to be looked at (call me paranoid but we are competing with the most powerful and best funded organizations in the history of the planet, right?).

Thanks all.

I read through the thread and although I am not able to give my opinion to your question, this last paragraph got me. I want to ask that assuming this were some kind of attack as you have mentioned what are some of the measures you would have implemented to counteract it. What proactive steps would you have taken to not only halt the ongoing attack but also to establish safeguards that prevent its recurrence in the future?

garlonicon

copper member

Activity: 821

Merit: 1992

Quote

Why not update to 25.0 though?

Good question, because 20.x does not support Taproot at all. Which means, Taproot transactions will not be visible in your mempool.

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Quote from: BitcoinCanSaveUsAll on October 14, 2023, 12:49:36 PM

If anyone knows why this could be happening in this 20.2 version I’d love to hear your feedback. Perhaps this is a bug on this version?

Are you sure you've not mistyped anything? It's not commented out, or set only for testnet? And are you sure your bitcoin.conf file is in the right location? Easiest way to check is to click on "Settings" -> "Open Configuration File" within the Core GUI.

Why not update to 25.0 though?

Pmalek

legendary

Activity: 2730

Merit: 7065

Quote from: BitcoinCanSaveUsAll on October 14, 2023, 12:49:36 PM

If anyone knows why this could be happening in this 20.2 version I’d love to hear your feedback. Perhaps this is a bug on this version?

Have you considered the easiest possible solution to simple upgrade to a newer version of Bitcoin Core which might fix the issue by itself? The latest release is 25.0, so there is a lot of ground to catch. Is something keeping you from upgrading to a newer version of the software?

BitcoinCanSaveUsAll

member

Activity: 104

Merit: 120

Hi all,

@ o_e_l_e_o,

I’ve just run the

Code:

getmempoolinfo

command on both and while there is still a discrepancy, I see that for some reason even though the maxmempool of 600 is in both option files for both the v.20.1 and the v.20.2 version it seems as though the .2 version is not applying this 600 MB value and is sticking to the default 300. I then checked what else I had in the two to compare apples to apples and then ensured to have only the same options enabled between the 20.1 and the 20.2 as to mimic the 20.1 that was seemingly applying the 600 MB value correctly. Unfortunately this did not work as after rerunning the command I see that 20.2 is still stuck on the 300 MB default cap.

@DaveF
I should also add that the computer running the .2 version is much more capable in terms of hardware and in fact has more ram than the other PC so there are no hardware limitations that should be coming into play here.

I next restarted the computer with the 20.2 version and again it’s still stuck with the 300 MB cap even though the following options are enabled in the options file:

Code:

maxmempool=600

If anyone knows why this could be happening in this 20.2 version I’d love to hear your feedback. Perhaps this is a bug on this version?

vjudeu

copper member

Activity: 906

Merit: 2258

You don't need to use any tricks in address bar. Just click on that last level of 0-1 range on that web page, and you will be redirected there. In exactly the same way you can click on other ranges, and you will be redirected to fees on that level or above. For example, if you click on 10-12 range, you will be redirected here: https://mempool.jhoenicke.de/#BTC,all,weight,10

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Quote from: vjudeu on October 14, 2023, 11:38:48 AM

https://mempool.jhoenicke.de/#BTC,all,weight,0

Did not know you could append a number like that to have the minimum fee displayed set to that level. That's a neat trick!

vjudeu

copper member

Activity: 906

Merit: 2258

Quote

transactions paying under 1 sat/vbyte?

Of course. We always had them, in some older versions, transactions were free, and some people still try to use them in that way. There are many transactions in range from zero to one satoshi per virtual byte. Some of them are just a result of mining pools, sending their own internal transactions for free, and including them in their own blocks (while also broadcasting them publicly), some of them are used in some off-chain protocols, like decentralized sidechain experiments, mining in LN experiments, or mempool-level-messaging experiments. Because nothing stops you from creating a group of nodes, that will use full-RBF, and increment fees one satoshi at a time, until reaching the final replacement of one satoshi per virtual byte, when all other nodes also try to process that batched result.

https://mempool.jhoenicke.de/#BTC,all,weight,0

Look at the bottom of the chart. That grey block can show you, what happens in 0-1 satoshis per virtual byte range. And if you have your own full node, even if it is pruned, you can directly receive that kind of traffic, if you change your configuration to allow that.

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Quote from: Pmalek on October 14, 2023, 07:22:41 AM

One shows the weight of transactions, the other the total number of them. My bad.

If you switch Johoe's to "BTC" (rather than "BTC (default mempool)") and sort by "count" rather than "weight", you should get a number which is roughly similar to mempool.space.

"BTC (default mempool)" shows the mempool with the 300 MB memory limit enforced. "BTC" shows the mempool with a much higher limit (although I'm not sure exactly how high he sets the limit). Although at the moment on the BTC/count graph, it says ~26k transactions at 1 sat/vbyte or more, but ~30k total transactions. I'm not sure what explains that deficit - transactions paying under 1 sat/vbyte?

Pmalek

legendary

Activity: 2730

Merit: 7065

Speaking of which, I have wondered something similar regarding the different mempools. As I am writing this, the one on Mempool.space shows 24k unconfirmed transactions. At the same time, Johoe's mempool has over 52k of unconfirmed transactions. It's not a small difference, it's more than the double. Could this really be about not being connected to the same peers, purging transactions of different sat values, or something else?

Scratch that.
One shows the weight of transactions, the other the total number of them. My bad.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Quote from: BitcoinCanSaveUsAll on October 03, 2023, 11:17:37 AM

....
@DaveF ,

Technically they are both on the same LAN but one is behind a VPN connection not accepting inbound connections (the recently updated v20.2) and the other is open with inbound connections.

All,

I'm starting to wonder if perhaps this may be related to the transactions in the mempool being increased to higher than my nodes default 300 MB size and somehow my nodes picked different transactions to store locally when the levels dropped below that level however wouldn't of imagined this is possible as shouldn't they be prioritizing them the same way? Either way from what I gather the mempool is below 300 MB now and the deltas between the two nodes continue to persist which would seem to invalidate my theory.

Also while I certainly appreciate all of the suggestions to try and correct the issue, the real reason for me bringing up this matter isn't to get my two nodes synched up so that they have the same mempool but to rather it's to understand why there is a discrepancy in the first place as this may somehow be some kind of attack that we aren't aware of presently that needs to be looked at (call me paranoid but we are competing with the most powerful and best funded organizations in the history of the planet, right?).

Thanks all.

Is there any reason you are on the 20.x instead of 24 or 25?

Since they are both on the same LAN you should be able to add the nodes to each others peer list.
Most VPNs will only deal with connections going to outside your LAN, otherwise you would not be able to print to a network printer or see network shares and so on when running a VPN.

How is the hardware that they are running on? About the same in terms of CPU / RAM / Drive?

-Dave

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Quote from: BitcoinCanSaveUsAll on October 13, 2023, 12:09:58 PM

Okay this is seemingly a persistent issue even after I increased the default mempool to 600 MB for both over a week ago.

There have still been times over the last week where your nodes' mempools will have exceeded 600 MB, and as I explained above the exact point this happens will be different for both your nodes.

Quote from: BitcoinCanSaveUsAll on October 13, 2023, 12:09:58 PM

The even stranger thing is that the 20.1 which is the node that has inbound connections and was started several hours after the 20.2 behind the VPN yet it has a much larger size than the other (303 MB vs. 280 MB currently).

Use getmempoolinfo again and compare the number of transactions in each mempool ("size"). Has this gap closed at all?

The next thing I would try would be to delete the mempool for both nodes, start them up simultaneously, let them both run for a couple of days, and then check if the discrepancy is still there. Also, if it is the node behind the VPN which is constantly behind, have you checked it isn't intermittently losing connection, the VPN intermittently disconnecting, or the VPN blocking any traffic?

BitcoinCanSaveUsAll

member

Activity: 104

Merit: 120

Okay this is seemingly a persistent issue even after I increased the default mempool to 600 MB for both over a week ago. As of this message I am still getting substantial mismatched between nodes whereas my node running version 20.1 is seeing a much larger memory usage then my 20.2 version. The even stranger thing is that the 20.1 which is the node that has inbound connections and was started several hours after the 20.2 behind the VPN yet it has a much larger size than the other (303 MB vs. 280 MB currently).

Something really seems off here and I'm hoping that someone here can help me understand why this is happening seemingly since around the time or the ordinals and inscriptions. Prior to this I have never seen a discrepancy before and I have been running the nose for years. Thanks in advance by any support!

o_e_l_e_o

legendary

Activity: 2268

Merit: 18748

Inbound/outbound connections won't make a difference here, provided both nodes have a good number of connections. Your nodes will receive unconfirmed transactions over both types of connection just the same.

Note that nodes won't try to fetch any transactions missing from their mempool. Once they drop a transaction, they won't learn about it again unless it is rebroadcast. If they receive a transaction which they do not add to their mempool due to it exceeding the size limit, then again, they won't learn about it unless it is rebroadcast. Given that the mempool has only recently come back down below the 300 MB default limit, if there was a delta between your nodes then this delta will persist until the backlog of transactions is cleared. Transactions which one node knows about but the other doesn't won't propagate between your nodes unless they are manually rebroadcast.

If one of your nodes has lost connection, had poor connections, been temporarily offline, hit the default limits before the other one, and so on, then any transactions not added to its mempool during this time will simply not be known about and the deficit will persist.

Topic: Bitcoin Mempool Mismatch Between Nodes - page 2. (Read 671 times)