Neither I nor any of the other developers are observing unbounded memory usage. It does go up a bit, but it always plateaus pretty quickly and doesn't seem to get into memory exhaustion.
Seeing some memory usage increase with this level of block verbosity is expected since the block has to be read into memory, decoded, and turned into JSON text with a ton of extra details that are also not explicitly in the block file itself, and text is not very efficient.
Topic: Running out of Memory RAM with RPC API REST /rest/block/ (Read 263 times)
What is extremely strange is that it happens when the full verbosity block is consulted. Because when the block is consulted without verbosity and I consult all the tx of the block separately (my python script), I do not have memory leaks, in fact my memory remains stable.
I want to note here since you use python that the Python interpreter does not garbage collect freed memory in many instances but instead holds onto it to reuse it for some other variable, even if you explicitly delete the variable or call the garbage collection functions in Python. It's not directly related to your problem, but if your script sees high memory usage that is not going down, this is why. The only way to fix that is to run your code in a subprocess that you create in your script.
But anyway, I have also previously experienced this problem in Bitcoin Core that you are referring to. The large memory usage happens when you try to fetch many transactions at once (as what happens when you fetch a block) because the node has to get more information for each of these transactions from other sources i.e. other blocks. The higher the verbosity, the worse the memory usage becomes. It is also quite independent of the number of RPC threads you are running, but the memory usage becomes noticeably higher if you batch many RPC calls together.
Hello,
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/.json the response is almost immediate but the memory of the daemon process for each block query increases a lot! and there is no release of this memory afterwards.
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/.json and process the json response and query each tx my memory remains always the same but is very slow.
Is there any solution to this problem?
Thanks!
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/
Is there any solution to this problem?
Thanks!
MEDIUM SOLUTION:
It turns out that I downloaded bitcoind and installed it, created symbolic links to my folders where my blocks are located, ran bitcoind -daemon and ran my scripts, the surprise I had was that my memory barely changed, I consulted more than a thousand blocks and my Memory barely increased 150MB, something quite insignificant considering the last 1000 blocks.
With Bitcoin Core Desktop version (Flathub) querying only one block and full verbosity increased my memory by approximately 100 MB.
Thank you for the actions taken, how do I follow up on these issues?
If you have a github account, you can subscribe to the issue to get notifications when comments are posted to the issue. If you have anything else to add, you can also comment on it yourself.How long does this whole process take?
It depends entirely on if anyone is interested in debugging this, and how hard it actually is to debug. Either way, someone will have to do the work to figure out what is wrong, then open a PR to fix it. Then the PR will need to be reviewed, and that can take anywhere from a few days to several years, depending on how complicated it is. You won't get the benefits of it until it hits a release, and the next major release is scheduled for October. There may be a backport release containing this fix before then, but these are not guaranteed nor are they scheduled, so don't count on it.Basically, it'll be at least several months before a release is made with this fixed.
Thank you for your time and your willingness to explain the workflow to me...
Thank you for the actions taken, how do I follow up on these issues?
If you have a github account, you can subscribe to the issue to get notifications when comments are posted to the issue. If you have anything else to add, you can also comment on it yourself.How long does this whole process take?
It depends entirely on if anyone is interested in debugging this, and how hard it actually is to debug. Either way, someone will have to do the work to figure out what is wrong, then open a PR to fix it. Then the PR will need to be reviewed, and that can take anywhere from a few days to several years, depending on how complicated it is. You won't get the benefits of it until it hits a release, and the next major release is scheduled for October. There may be a backport release containing this fix before then, but these are not guaranteed nor are they scheduled, so don't count on it.Basically, it'll be at least several months before a release is made with this fixed.
I seem to have reproduced this on my machine, at least calling getblock with verbosity 2 has increased the memory usage for most of the calls.
I've opened an issue on github so other developers can take a look and debug this: https://github.com/bitcoin/bitcoin/issues/30052
The behavior that I've observed is that it doesn't always increase the memory usage. This suggests to me that it's probably loading something into an unbounded cache, and the cache is growing too large causing the crash.
I've opened an issue on github so other developers can take a look and debug this: https://github.com/bitcoin/bitcoin/issues/30052
The behavior that I've observed is that it doesn't always increase the memory usage. This suggests to me that it's probably loading something into an unbounded cache, and the cache is growing too large causing the crash.
I think the error is reproduced with all current blocks, blocks with a lot of data.
Thank you for the actions taken, how do I follow up on these issues?
How long does this whole process take?
Thank you
I seem to have reproduced this on my machine, at least calling getblock with verbosity 2 has increased the memory usage for most of the calls.
I've opened an issue on github so other developers can take a look and debug this: https://github.com/bitcoin/bitcoin/issues/30052
The behavior that I've observed is that it doesn't always increase the memory usage. This suggests to me that it's probably loading something into an unbounded cache, and the cache is growing too large causing the crash.
I've opened an issue on github so other developers can take a look and debug this: https://github.com/bitcoin/bitcoin/issues/30052
The behavior that I've observed is that it doesn't always increase the memory usage. This suggests to me that it's probably loading something into an unbounded cache, and the cache is growing too large causing the crash.
~
You have a verry good processor actually.Let me ask....
Is it an additional SSD or the the one you installed your operating system on??
First of all, thanks to everyone who has taken the time to help me solve or detect the bug...
My Fedora Distro is installed in my laptop SSD and my org.bitcoincore.bitcoin-qt directory is in my external SSD. Im using a Bitcoin Core Flathub installation.
Im going to try the approach of install bitcoind and test to see if the problem remains.
Seems like there's a memory leak somewhere. Do you see the same problem if you use the getblock RPC with verbosity 2?
e.g.
Both that rest api and the rpc use almost the same code so this would help narrow down what's causing the problem.
e.g.
Code:
bitcoin-cli getblock2
Both that rest api and the rpc use almost the same code so this would help narrow down what's causing the problem.
First of all, thanks to everyone who has taken the time to help me solve or detect the bug...
I ran the command
Code:
bitcoin-cli getblock 00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054 2
and the memory leak problems continue, every query increases the memory allocated to bitcoin process by approximately 100MB.
What is extremely strange is that it happens when the full verbosity block is consulted. Because when the block is consulted without verbosity and I consult all the tx of the block separately (my python script), I do not have memory leaks, in fact my memory remains stable.
I'm curious, can you consult that block and see your memory? Is it just me with this problem?
Seems like there's a memory leak somewhere. Do you see the same problem if you use the getblock RPC with verbosity 2?
e.g.
Both that rest api and the rpc use almost the same code so this would help narrow down what's causing the problem.
e.g.
Code:
bitcoin-cli getblock2
Both that rest api and the rpc use almost the same code so this would help narrow down what's causing the problem.
~
You have a verry good processor actually.Let me ask....
Is it an additional SSD or the the one you installed your operating system on??
Using the API /rest/block/.json the response is almost immediate but the memory of the daemon process for each block query increases a lot! and there is no release of this memory afterwards.
Googling for Bitcoin Core memory leaks, I found this post:Quote
By default, since glibc 2.10, the C library will create up to two heap
arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
I don't know if it's going to solve your problem, but it's worth a try.arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
If this approach is with the bitcoin.conf file, I already try but the problem remain the same
Okay what's your pc processor if I may ask ??
For the hardware component try getting an additional SSD if you have an extra slot for hardisk on your pc . The exFAT format is for windows actually move your node_block directory to this new SSD drive and create a partition from your bitcoin.conf file. Note don't delete your old node_block directory...you can just remain it .
Now you can query from the new directory.... i don't know if I'm on point that align with yours
For the software component you could give the response give by @LoyceV a trial
My Bitcoin Core data is already in an SSD.
Please check again my previous responde from the one that you quote. Because I made some modifications
CPU:
Code:
CPU:
Info: 6-core model: Intel Core i7-9750H bits: 64 type: MT MCP
arch: Coffee Lake rev: A cache: L1: 384 KiB L2: 1.5 MiB L3: 12 MiB
Speed (MHz): avg: 800 min/max: 800/4500 cores: 1: 800 2: 800 3: 800 4: 800
5: 800 6: 800 7: 800 8: 800 9: 800 10: 800 11: 800 12: 800 bogomips: 62399
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx
It is not a scalable solution for me, the day will come when even if I have 500GB ram I will reach the RAM limit.
It also doesn't seem very smart to me to be closing and opening Bitcoin Core every 2 days.
It is interesting to note that using the API:
Code:
curl http://localhost:8332/rest/block/notxdetails/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
And then processing each tx query with my python script:
Code:
curl http://localhost:8332/rest/tx/{tx_id}.jsonThe memory always remains the same! but it takes between 30 to 45 seconds for my script to query each of the tx inside each block (blocks with more than 3K tx inside)
For the hardware component try getting an additional SSD if you have an extra slot for hardisk on your pc . The exFAT format is for windows actually move your node_block directory to this new SSD drive and create a partition from your bitcoin.conf file. Note don't delete your old node_block directory...you can just remain it .
Now you can query from the new directory.... i don't know if I'm on point that align with yours
For the software component you could give the response give by @LoyceV a trial
Googling for Bitcoin Core memory leaks, I found this post:
Quote
By default, since glibc 2.10, the C library will create up to two heap
arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
I don't know if it's going to solve your problem, but it's worth a try.arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
Bitcoin core uses the RAM alot and to reduce this your hardisk function usually gets involved and if it reads at a lower rate then you could end up getting a slow response because it will get used up faster while running your node ....
There are some couple of things you could do to speed it up
- Get a new SSD Drive with a larger storage since hardisk needs more space to peform faster
- If your PC has an additional slot for an extra ram then you could get an additional RAM for your PC else if you have just a single slot and your ram is low, then replace it with a higher Ram
- You could take down some background processing applications to free up usage too although it's less effective but it contributes.
It is not a scalable solution for me, the day will come when even if I have 500GB ram I will reach the RAM limit.
It also doesn't seem very smart to me to be closing and opening Bitcoin Core every 2 days.
It is interesting to note that using the API:
Code:
curl http://localhost:8332/rest/block/notxdetails/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
And then processing each tx query with my python script:
Code:
curl http://localhost:8332/rest/tx/{tx_id}.jsonThe memory always remains the same! but it takes between 30 to 45 seconds for my script to query each of the tx inside each block (blocks with more than 3K tx inside)
I made a Benchmark python script:
With API /rest/block/{blockhash}
Code:
myuser@server:$ python time_query_rest.py 800000 800000
Block Height: 800000
Number of lines in block data: 13666565
Total time to query all transactions in block 800000: 0.0 seconds
Number of transactions in block 800000: 3721
With API /rest/block/notxdetails/{blockhash}
Code:
myuser@server:$ python time_query_rest_notxdetails.py 800000 800000
Block Height: 800000
Number of lines in block data: 253731
Total time to query all transactions in block 800000: 38.34489599999994 seconds
Number of transactions in block 800000: 3721
Quote
How much memory do you have?
Can you add more ram if you are lite?
Also can you try a 5 minute query vs a 1 minute query?
Can you add more ram if you are lite?
Also can you try a 5 minute query vs a 1 minute query?
How much memory do you have?
12GB RAM availability
Also can you try a 5 minute query vs a 1 minute query?
Its the same, in each query begins to increase. If you want you can try it and see how it increases:
Code:
curl http://localhost:8332/rest/block/00000000000000000002a7c4c1e48d76c5a37902165a270156b7a8d72728a054.json
Thanks for your time.
Using the API /rest/block/.json the response is almost immediate but the memory of the daemon process for each block query increases a lot! and there is no release of this memory afterwards.
Googling for Bitcoin Core memory leaks, I found this post:Quote
By default, since glibc 2.10, the C library will create up to two heap
arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
I don't know if it's going to solve your problem, but it's worth a try. arenas per core. This is known to cause excessive memory usage in some
scenarios. To avoid this we have to set
MALLOC_ARENA_MAX=1
without this the usage grows beyond 36GB, with this it stays under 2GB
normally.
ref: https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific
~
Bitcoin core uses the RAM alot and to reduce this your hardisk function usually gets involved and if it reads at a lower rate then you could end up getting a slow response because it will get used up faster while running your node ....
There are some couple of things you could do to speed it up
- Get a new SSD Drive with a larger storage since hardisk needs more space to peform faster
- If your PC has an additional slot for an extra ram then you could get an additional RAM for your PC else if you have just a single slot and your ram is low, then replace it with a higher Ram
- You could take down some background processing applications to free up usage too although it's less effective but it contributes.
Also can you try a 5 minute query vs a 1 minute query?
Hello,
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/.json the response is almost immediate but the memory of the daemon process for each block query increases a lot! and there is no release of this memory afterwards.
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/.json and process the json response and query each tx my memory remains always the same but is very slow.
Is there any solution to this problem?
Thanks!
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/
Is there any solution to this problem?
Thanks!
How much memory do you have?
Can you add more ram if you are lite?
Also can you try a 5 minute query vs a 1 minute query?
Hello,
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core Desktop version (Flathub) on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/.json the response is almost immediate but the memory of the daemon process for each block query increases a lot! and there is no release of this memory afterwards.
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/.json and process the json response and query each tx my memory remains always the same but is very slow.
Is there any solution to this problem?
Thanks!
MEDIUM SOLUTION:
It turns out that I downloaded bitcoind and installed it, created symbolic links to my folders where my blocks are located, ran bitcoind -daemon and ran my scripts, the surprise I had was that my memory barely changed, I consulted more than a thousand blocks and my Memory barely increased 150MB, something quite insignificant considering the last 1000 blocks.
With Bitcoin Core Desktop version (Flathub) querying only one block and full verbosity increased my memory by approximately 100 MB.
I'm encountering a persistent issue with memory RAM exhaustion while running Bitcoin Core Desktop version (Flathub) on my system.
Here's a brief overview of the problem:
Problem Description:
I have a Bitcoin Core node set up on my system, and I query it every minute using an python script consulting for new blocks.
Using the API /rest/block/
To solve this problem I must close bitcoin core and reopen it from time to time
But if I use the API /rest/block/notxdetails/
Is there any solution to this problem?
Thanks!
MEDIUM SOLUTION:
It turns out that I downloaded bitcoind and installed it, created symbolic links to my folders where my blocks are located, ran bitcoind -daemon and ran my scripts, the surprise I had was that my memory barely changed, I consulted more than a thousand blocks and my Memory barely increased 150MB, something quite insignificant considering the last 1000 blocks.
With Bitcoin Core Desktop version (Flathub) querying only one block and full verbosity increased my memory by approximately 100 MB.
Jump to: