Okay thank you for your help.
Problem does not seem to be straightforward.
Will open a GitHub issue.
Topic: Full Node: Out of Memory Error (Read 723 times)
I am sorry but i don't see any relevant information here, maybe you should ask for moderators as some of them are part of the development team, at the same time. Or you can send these logs in a github issue and post the link here so we can follow what they are saying, i am interested to see what is the cause of this problem as i run a bitcoin full node, it may be usefull in the future.
In htop I see that the node is currently using 579M of memory.
Line written before bitcoind is killed in /var/lol/syslog:
And debug.log before and after crash (20:32):
Thank you for your help.
Line written before bitcoind is killed in /var/lol/syslog:
Code:
Feb 19 20:32:37 kernel: [258198.535578] dhclient invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Feb 19 20:32:37 kernel: [258198.535580] dhclient cpuset=/ mems_allowed=0
Feb 19 20:32:37 kernel: [258198.535584] CPU: 0 PID: 965 Comm: dhclient Not tainted 4.4.0-62-generic #83-Ubuntu
Feb 19 20:32:37 kernel: [258198.535586] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/09/2016
Feb 19 20:32:37 kernel: [258198.535587] 0000000000000286 000000006830faae ffff8800e850faf0 ffffffff813f7c63
Feb 19 20:32:37 kernel: [258198.535590] ffff8800e850fcc8 ffff880015006200 ffff8800e850fb60 ffffffff8120ad4e
Feb 19 20:32:37 kernel: [258198.535592] ffffffff81cd2d7f 0000000000000000 ffffffff81e67760 0000000000000206
Feb 19 20:32:37 kernel: [258198.535594] Call Trace:
Feb 19 20:32:37 kernel: [258198.535600] [] dump_stack+0x63/0x90
Feb 19 20:32:37 kernel: [258198.535603] [] dump_header+0x5a/0x1c5
Feb 19 20:32:37 kernel: [258198.535607] [] oom_kill_process+0x202/0x3c0
Feb 19 20:32:37 kernel: [258198.535608] [] out_of_memory+0x219/0x460
Feb 19 20:32:37 kernel: [258198.535611] [] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70
Feb 19 20:32:37 kernel: [258198.535614] [] __alloc_pages_nodemask+0x286/0x2a0
Feb 19 20:32:37 kernel: [258198.535616] [] alloc_kmem_pages_node+0x4b/0xc0
Feb 19 20:32:37 kernel: [258198.535619] [] copy_process+0x1be/0x1b70
Feb 19 20:32:37 kernel: [258198.535623] [] ? sock_sendmsg+0x38/0x50
Feb 19 20:32:37 kernel: [258198.535625] [] ? SYSC_sendto+0x101/0x190
Feb 19 20:32:37 kernel: [258198.535627] [] _do_fork+0x80/0x360
Feb 19 20:32:37 kernel: [258198.535629] [] SyS_clone+0x19/0x20
Feb 19 20:32:37 kernel: [258198.535632] [] entry_SYSCALL_64_fastpath+0x16/0x71
Feb 19 20:32:37 kernel: [258198.535634] Mem-Info:
Feb 19 20:32:37 kernel: [258198.535637] active_anon:55728 inactive_anon:62585 isolated_anon:0
Feb 19 20:32:37 kernel: [258198.535637] active_file:417178 inactive_file:416336 isolated_file:0
Feb 19 20:32:37 kernel: [258198.535637] unevictable:929 dirty:6 writeback:0 unstable:0
Feb 19 20:32:37 kernel: [258198.535637] slab_reclaimable:34325 slab_unreclaimable:5712
Feb 19 20:32:37 kernel: [258198.535637] mapped:8525 shmem:3376 pagetables:2259 bounce:0
Feb 19 20:32:37 kernel: [258198.535637] free:8451 free_pcp:0 free_cma:0
Feb 19 20:32:37 kernel: [258198.535640] Node 0 DMA free:15788kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:8kB active_file:12kB inactive_file:4kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:32kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535644] lowmem_reserve[]: 0 3745 3934 3934 3934
Feb 19 20:32:37 kernel: [258198.535647] Node 0 DMA32 free:16888kB min:7528kB low:9408kB high:11292kB active_anon:211756kB inactive_anon:227384kB active_file:1603828kB inactive_file:1599912kB unevictable:3376kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3835152kB mlocked:3376kB dirty:24kB writeback:0kB mapped:32608kB shmem:8864kB slab_reclaimable:125296kB slab_unreclaimable:20152kB kernel_stack:2656kB pagetables:8312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535651] lowmem_reserve[]: 0 0 189 189 189
Feb 19 20:32:37 kernel: [258198.535654] Node 0 Normal free:1128kB min:380kB low:472kB high:568kB active_anon:11156kB inactive_anon:22948kB active_file:64872kB inactive_file:65428kB unevictable:308kB isolated(anon):0kB isolated(file):0kB present:393216kB managed:193908kB mlocked:308kB dirty:0kB writeback:0kB mapped:1480kB shmem:4640kB slab_reclaimable:11984kB slab_unreclaimable:2656kB kernel_stack:784kB pagetables:724kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535657] lowmem_reserve[]: 0 0 0 0 0
Feb 19 20:32:37 kernel: [258198.535660] Node 0 DMA: 5*4kB (ME) 5*8kB (MEH) 3*16kB (MH) 2*32kB (EH) 2*64kB (UE) 3*128kB (UME) 3*256kB (UME) 2*512kB (ME) 3*1024kB (MEH) 1*2048kB (E) 2*4096kB (M) = 15788kB
Feb 19 20:32:37 kernel: [258198.535670] Node 0 DMA32: 625*4kB (UME) 1553*8kB (UME) 3*16kB (H) 1*32kB (H) 0*64kB 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 16924kB
Feb 19 20:32:37 kernel: [258198.535679] Node 0 Normal: 142*4kB (UME) 76*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1176kB
Feb 19 20:32:37 kernel: [258198.535686] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 19 20:32:37 kernel: [258198.535687] 838165 total pagecache pages
Feb 19 20:32:37 kernel: [258198.535696] 627 pages in swap cache
Feb 19 20:32:37 kernel: [258198.535697] Swap cache stats: add 87692, delete 87065, find 64623/66921
Feb 19 20:32:37 kernel: [258198.535698] Free swap = 8350600kB
Feb 19 20:32:37 kernel: [258198.535699] Total swap = 8388604kB
Feb 19 20:32:37 kernel: [258198.535700] 1081245 pages RAM
Feb 19 20:32:37 kernel: [258198.535700] 0 pages HighMem/MovableOnly
Feb 19 20:32:37 kernel: [258198.535701] 70004 pages reserved
Feb 19 20:32:37 kernel: [258198.535702] 0 pages cma reserved
Feb 19 20:32:37 kernel: [258198.535703] 0 pages hwpoisoned
And debug.log before and after crash (20:32):
Code:
2017-02-19 20:32:17 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=412192, us=127.0.0.1:8333, peer=321
2017-02-19 20:32:18 connection from 138.197.197.152:31044 dropped (banned)
2017-02-19 20:32:20 connection from 138.197.197.164:60006 dropped (banned)
2017-02-19 20:32:26 connection from 138.68.10.138:57854 dropped (banned)
2017-02-19 20:32:26 connection from 138.197.197.108:15696 dropped (banned)
2017-02-19 20:32:28 connection from 138.197.197.152:33434 dropped (banned)
2017-02-19 20:32:29 connection from 138.197.195.32:38820 dropped (banned)
2017-02-19 20:32:32 connection from 138.197.194.32:8144 dropped (banned)
2017-02-19 20:32:33 connection from 138.197.197.174:43606 dropped (banned)
2017-02-19 20:32:35 connection from 138.197.197.179:4758 dropped (banned)
2017-02-19 20:32:36 connection from 138.197.197.108:18216 dropped (banned)
2017-02-19 20:35:01
2017-02-19 20:35:01 Bitcoin version v0.13.2.0-g0d71914
2017-02-19 20:35:01 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2017-02-19 20:35:01 Default data directory /home/ubuntu/.bitcoin
2017-02-19 20:35:01 Using data directory /home/ubuntu/.bitcoin
2017-02-19 20:35:01 Using config file /home/ubuntu/.bitcoin/bitcoin.conf
2017-02-19 20:35:01 Using at most 125 connections (1024 file descriptors available)
2017-02-19 20:35:01 Using 2 threads for script verification
2017-02-19 20:35:01 scheduler thread start
Thank you for your help.
you should have a look at the mommory usage bars in htop, if it goes over 100% or not, and you should also check the swap usage which is bellow memory usage.
Have you checked debug.log ?
Could you try to find the lines writen there before the crash happens ?
Have you checked debug.log ?
Could you try to find the lines writen there before the crash happens ?
Here is the list of process running at the time of the crash.
I have an apache2 server running in a docker (looks like they use a lot of memory?).
Blockchain is 100% downloaded.
Code:
Feb 19 20:32:37 kernel: [258198.535703] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 19 20:32:37 kernel: [258198.535708] [ 406] 0 406 10180 1229 21 4 558 0 systemd-journal
Feb 19 20:32:37 kernel: [258198.535710] [ 432] 0 432 23693 245 16 3 3 0 lvmetad
Feb 19 20:32:37 kernel: [258198.535711] [ 468] 0 468 10748 800 23 3 2 -1000 systemd-udevd
Feb 19 20:32:37 kernel: [258198.535713] [ 550] 100 550 25081 539 20 4 13 0 systemd-timesyn
Feb 19 20:32:37 kernel: [258198.535715] [ 965] 0 965 4030 622 12 3 22 0 dhclient
Feb 19 20:32:37 kernel: [258198.535716] [ 1085] 0 1085 1306 29 8 3 0 0 iscsid
Feb 19 20:32:37 kernel: [258198.535718] [ 1086] 0 1086 1431 879 9 3 0 -17 iscsid
Feb 19 20:32:37 kernel: [258198.535720] [ 1090] 0 1090 68652 1038 36 3 23 0 accounts-daemon
Feb 19 20:32:37 kernel: [258198.535721] [ 1092] 0 1092 6511 452 19 3 1 0 atd
Feb 19 20:32:37 kernel: [258198.535723] [ 1094] 0 1094 6932 589 18 3 46 0 cron
Feb 19 20:32:37 kernel: [258198.535724] [ 1096] 104 1096 65158 738 29 3 18 0 rsyslogd
Feb 19 20:32:37 kernel: [258198.535726] [ 1098] 0 1098 77216 352 20 3 7 0 lxcfs
Feb 19 20:32:37 kernel: [258198.535728] [ 1102] 0 1102 141470 3735 66 6 1492 -500 dockerd
Feb 19 20:32:37 kernel: [258198.535729] [ 1104] 107 1104 10758 732 26 3 27 -900 dbus-daemon
Feb 19 20:32:37 kernel: [258198.535731] [ 1115] 0 1115 1100 298 7 3 1 0 acpid
Feb 19 20:32:37 kernel: [258198.535732] [ 1143] 0 1143 7545 1059 18 3 21 0 systemd-logind
Feb 19 20:32:37 kernel: [258198.535734] [ 1150] 0 1150 16380 901 36 3 30 -1000 sshd
Feb 19 20:32:37 kernel: [258198.535736] [ 1161] 0 1161 3344 35 11 3 2 0 mdadm
Feb 19 20:32:37 kernel: [258198.535737] [ 1165] 0 1165 69278 843 39 3 8 0 polkitd
Feb 19 20:32:37 kernel: [258198.535739] [ 1242] 0 1242 4868 51 14 3 13 0 irqbalance
Feb 19 20:32:37 kernel: [258198.535741] [ 1256] 0 1256 3619 460 12 3 5 0 agetty
Feb 19 20:32:37 kernel: [258198.535742] [ 1257] 0 1257 3665 365 11 3 4 0 agetty
Feb 19 20:32:37 kernel: [258198.535744] [ 1259] 0 1259 72894 1833 28 5 318 -500 docker-containe
Feb 19 20:32:37 kernel: [258198.535745] [ 1680] 0 1680 49036 955 23 5 82 -500 docker-proxy
Feb 19 20:32:37 kernel: [258198.535747] [ 1686] 0 1686 51736 229 21 5 147 -500 docker-containe
Feb 19 20:32:37 kernel: [258198.535748] [ 1705] 0 1705 4544 88 15 3 119 0 bash
Feb 19 20:32:37 kernel: [258198.535750] [ 1803] 0 1803 63847 691 98 3 1065 0 apache2
Feb 19 20:32:37 kernel: [258198.535752] [ 1809] 33 1809 83011 1473 101 3 1011 0 apache2
Feb 19 20:32:37 kernel: [258198.535753] [ 1813] 33 1813 82994 1242 101 3 1048 0 apache2
Feb 19 20:32:37 kernel: [258198.535755] [ 1814] 33 1814 82995 1214 101 3 1044 0 apache2
Feb 19 20:32:37 kernel: [258198.535756] [ 1815] 33 1815 82976 1216 101 3 1023 0 apache2
Feb 19 20:32:37 kernel: [258198.535758] [ 1816] 33 1816 82995 1401 101 3 1020 0 apache2
Feb 19 20:32:37 kernel: [258198.535759] [ 1817] 33 1817 83058 1314 101 3 1047 0 apache2
Feb 19 20:32:37 kernel: [258198.535761] [ 1818] 33 1818 82988 1208 101 3 1044 0 apache2
Feb 19 20:32:37 kernel: [258198.535762] [ 6992] 0 6992 35607 1924 26 5 146 0 snapd
Feb 19 20:32:37 kernel: [258198.535764] [ 8550] 33 8550 82968 1439 101 3 993 0 apache2
Feb 19 20:32:37 kernel: [258198.535765] [ 8552] 33 8552 82962 1286 101 3 1021 0 apache2
Feb 19 20:32:37 kernel: [258198.535767] [ 8553] 33 8553 82968 1247 101 3 1023 0 apache2
Feb 19 20:32:37 kernel: [258198.535768] [ 8554] 33 8554 82960 1244 101 3 1020 0 apache2
Feb 19 20:32:37 kernel: [258198.535769] [ 8555] 33 8555 63973 1187 94 3 1008 0 apache2
Feb 19 20:32:37 kernel: [258198.535771] [16697] 1000 16697 413383 104742 350 5 60 0 bitcoind
I have an apache2 server running in a docker (looks like they use a lot of memory?).
Blockchain is 100% downloaded.
You can paste this command in a terminal : df -h --total
you will have the amount of free disk space.
Now you can monitore the computer/VPS using htop: sudo apt-get install -y htop && htop
You will have several informations displayed, like the CPU usage, mommory usage, the ressources used by each process ...
You have some solutions to check what is going on, could you try to add "prune = 50000" to your bitcoin.conf file after deleting the blockchain ?
This command will let you download the last 50k blocks instead of downloading the full blockchain, this way you will spare some space for the next blocks.
As you have a 8gb swap, i don't think the problem is comming from there, but you will for sure notice a big usage of disc space, so your best chance is to prune the blockchain.
you will have the amount of free disk space.
Now you can monitore the computer/VPS using htop: sudo apt-get install -y htop && htop
You will have several informations displayed, like the CPU usage, mommory usage, the ressources used by each process ...
You have some solutions to check what is going on, could you try to add "prune = 50000" to your bitcoin.conf file after deleting the blockchain ?
This command will let you download the last 50k blocks instead of downloading the full blockchain, this way you will spare some space for the next blocks.
As you have a 8gb swap, i don't think the problem is comming from there, but you will for sure notice a big usage of disc space, so your best chance is to prune the blockchain.
I am curious what else is running on the system. In my experience that error is from the Linux kernel killing a process because the system is low on memory as a whole. Then the out of memory killer picks a process to kill based on various variables rolled into a score - hence your score 33.
Is something else on the machine memory and swap intensive? If so, it may be that process that is causing the issue and bitcoin just happens to be the process that the OOM killer is picking based on its heuristics to kill.
Obviously much of this is a guess without more info.
Is something else on the machine memory and swap intensive? If so, it may be that process that is causing the issue and bitcoin just happens to be the process that the OOM killer is picking based on its heuristics to kill.
Obviously much of this is a guess without more info.
Can you provide more data?
Like:
Your system;
Can you replicate the error in some way?
Your bitcoin.conf settings.
What is the load of your system when running the node.
Like:
Your system;
Can you replicate the error in some way?
Your bitcoin.conf settings.
What is the load of your system when running the node.
Hello,
I have a Full Node running on 0.13.2. However, every 24 hours it crashes with this message:
I have 4 GB of ram and 8 GB of swap.
Thank you for your help.
I have a Full Node running on 0.13.2. However, every 24 hours it crashes with this message:
Code:
Feb 19 20:32:37 kernel: [258198.535771] [16697] 1000 16697 413383 104742 350 5 60 0 bitcoind
Feb 19 20:32:37 kernel: [258198.535773] Out of memory: Kill process 16697 (bitcoind) score 33 or sacrifice child
Feb 19 20:32:37 kernel: [258198.540063] Killed process 16697 (bitcoind) total-vm:1653532kB, anon-rss:402216kB, file-rss:16752kB
Thank you for your help.
Jump to: