Author

Topic: Armory 0.95.1 out of memory rebuilding database (Read 851 times)

legendary
Activity: 3640
Merit: 1345
Armory Developer
January 28, 2017, 03:13:58 AM
#9
Are the stale logins related to Armory at all? That's the first time I hear anything like that.

As for the virtual memory, Armory mmaps all blkXXXXX.dat files when building and scanning. No copy of that data is performed in memory though, it's basically just a caching hint for the OS.

The 4GB figure looks high. I usually don't break 1.5GB on my machine, but that's very dependent on your setup. Basically you are running into this:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BlockchainScanner.cpp#L195

In full/bare node, there is very little data to write during scans (you can check verify by looking the size of your ssh file in the database folder, it should be tiny compared to the 100GB+ worth of blockchain data), so I just went with the assumptions the writer thread would always outpace the scanner threads (i.e there would never be any write queue depth).

I could fix the write queuing in 30 min but I don't want to release DB changes this time around, as very few people have run into your kind of issue (you are probably the worst case so far). I plan on speeding up the DB a bit in a later release (gotta keep up with the ever growing blockchain), I'll deal with it then.

Quote
More info - it seems to have nothing to do with the box also running bitcoind.

I only suggested that because Core can eat up a lot of RAM (2~3GB)
newbie
Activity: 13
Merit: 0
More info - it seems to have nothing to do with the box also running bitcoind. There were two stale logins (/usr/lib/systemd/systemd --user) with a bunch of child processes under them. Killing off both of those, the rebuild works with a simple --db-type="DB_FULL" --rebuild command. It eventually gets over 80GB virtual memory as shown by 'top', but only uses about 4GB of real memory and no swap.

I am not sure why those two stale logins caused this - none of the child processes seemed to be using up much memory.

newbie
Activity: 13
Merit: 0
Yes, bitcoind was running. I stopped that, and the armory rebuild worked. Thanks!

legendary
Activity: 3640
Merit: 1345
Armory Developer
Are you running Core along ArmoryDB? Try without it.
newbie
Activity: 13
Merit: 0
Same results with --db-type="DB_BARE" --rebuild --ram-usage=1 --thread-count=1

Of course, "rm -rf ~/.armory/databases; mkdir ~/.armory/databases" before trying that rebuild.

This is packaged from the source:
Fetch URL: https://github.com/goatpig/BitcoinArmory.git

commit cc9eccf690c8534af1502f66176e2ca73a9f083c
Merge: 2db2622 0816758
Author: goatpig
Date:   Tue Nov 8 18:04:50 2016 +0100

    Merge pull request #128 from achow101/master
   
    Remove bitcoinarmory.com link from multisig dialog


Is that the current source location/version? changelog.txt says 0.95.1

legendary
Activity: 3640
Merit: 1345
Armory Developer
Reduce the thread count then.
newbie
Activity: 13
Merit: 0
Nope. uname -a shows:

4.8.10-300.fc25.x86_64 #1 SMP Mon Nov 21 18:59:16 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

file /usr/bin/python2.7
/usr/bin/python2.7: ELF 64-bit LSB shared object, x86-64,

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 22761
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 22761
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

legendary
Activity: 3640
Merit: 1345
Armory Developer
Are you using a 32bit OS?
newbie
Activity: 13
Merit: 0
Running on Fedora 25, python 2.7, the box has 6GB memory, 6GB swap.

Armory 0.95.1 was working months ago, but failed to start today. The underlying bitcoind seems to be happy:

bitcoin-cli getinfo
{
  "version": 130100,
  "protocolversion": 70014,
  "walletversion": 60000,
  "balance": 0.00000000,
  "blocks": 450159,
  "timeoffset": 0,
  "connections": 124,
  "proxy": "",
  "difficulty": 392963262344.3704,
  "testnet": false,
  "keypoololdest": 1448647753,
  "keypoolsize": 105,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "errors": ""
}

Armory launched ArmoryDB, with DB_FULL. Based on other messages here, I tried to reduce the memory footprint with:

/usr/bin/ArmoryDB --db-type="DB_BARE" --spawnId="6RYQUcvejFYTxiPjDGhXABeP2pppQQyMXAv3bouBQNtE" --satoshi-datadir="/home/carl/.bitcoin/blocks" --datadir="/home/carl/.armory/" --dbdir="/home/carl/.armory/databases" --rebuild --ram-usage=1

That still dies with about 8GB resident.

dbLog.txt is the same as the console output:

Log file opened at 1485470827: /home/carl/.armory/dbLog.txt
-INFO  - 1485470827: (main.cpp:23) Running on 4 threads
-INFO  - 1485470827: (main.cpp:24) Ram usage level: 1
-INFO  - 1485470827: (BlockUtils.cpp:1338) blkfile dir: /home/carl/.bitcoin/blocks
-INFO  - 1485470827: (BlockUtils.cpp:1339) lmdb dir: /home/carl/.armory/databases
-INFO  - 1485470827: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 1485470827: (BlockUtils.cpp:1540) Executing: doInitialSyncOnLoad_Rebuild
-INFO  - 1485470827: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 1485470827: (BitcoinP2P.cpp:783) Connected to Bitcoin node
-INFO  - 1485470827: (DatabaseBuilder.cpp:169) Reading headers from db
-WARN  - 1485470827: (lmdb_wrapper.cpp:1175) No headers in DB yet!
-INFO  - 1485470827: (DatabaseBuilder.cpp:208) Found 1 headers in db
-INFO  - 1485470827: (DatabaseBuilder.cpp:51) updating HEADERS db
-INFO  - 1485470829: (DatabaseBuilder.cpp:268) parsed block file #1
-INFO  - 1485470829: (DatabaseBuilder.cpp:268) parsed block file #2
-INFO  - 1485470830: (DatabaseBuilder.cpp:268) parsed block file #3
....
-INFO  - 1485470944: (DatabaseBuilder.cpp:268) parsed block file #45
-INFO  - 1485470950: (DatabaseBuilder.cpp:268) parsed block file #47
-INFO  - 1485470974: (DatabaseBuilder.cpp:268) parsed block file #49

And it gets killed at that point. "journalctl -r" shows:

Killed process 14915 (ArmoryDB) total-vm:7897844kB, anon-rss:677480kB, file-rss:3882984kB
Out of memory: Kill process 14915 (ArmoryDB) score 386 or sacrifice child

Substantially the same results with DB_FULL or DB_BARE, ram-usage 1 or 4.


Jump to: