Author

Topic: Memory fills up to 100% ArmoryDB (Read 521 times)

newbie
Activity: 8
Merit: 0
June 05, 2017, 03:59:06 AM
#13
Thanks Goatpig!

The memory issue is fixed in build armory_0.96.0.2-testing_win64.exe.
Used memory stays around 1,5 - 2 GB, and never gets filled up now. No pagefile swapping anymore.

Armory finishes in 9 minutes for first build of the databases and getting online!

Thank you for fixing the memory issue!
newbie
Activity: 8
Merit: 0
June 04, 2017, 03:02:05 PM
#12
Before the start, ArmoryDB does not run as a process.

When I let it run for a long time (hours), it finally crashes, as if it has no more connection with the Bitcoin Core.
But the crash is only after a long time.
Probably because the whole computer is unresponsive.
newbie
Activity: 53
Merit: 0
June 04, 2017, 02:57:21 PM
#11
@treeset , what happens, does it crash with you like it does with me?
Before you start armory, you should check if the armorydb.exe process is still running in the background. If that's the case then armory starts but nothing further happens....hopefully the new test version solves this problem
newbie
Activity: 8
Merit: 0
June 04, 2017, 01:03:01 PM
#10
It still fills up. Not as fast though  Wink

I've added screenshots to the folder.

https://drive.google.com/drive/folders/0BzDo3gCEmMEVaEktNGFmWEFkRXc
legendary
Activity: 3640
Merit: 1345
Armory Developer
June 04, 2017, 12:38:12 PM
#9
Try 1/1 for thread and ram settings. Get me a screenshot of the Task Managers process tab as well.
newbie
Activity: 8
Merit: 0
June 04, 2017, 12:01:08 PM
#8
I have started the database with the following command:

C:\Program Files (x86)\Armory>ArmoryDB.exe --thread-count=4 --ram-usage=8 --satoshi-datadir="g:\Virtual Machines SSD\BitconCoreDB\"

It still fills up the memory. When the program runs in the beginning, the CPU is high, no problem.
After the memory is filled up, the CPU utilization reduces, and the pc starts swapping to the pagefile and is terrible unresponsive at this moment (with less CPU utilization).

I made screenshots which would explain more than a lot of words ;-)
But I have no place to put them online unfortunately.

Edit: created google drive link:
https://drive.google.com/drive/folders/0BzDo3gCEmMEVaEktNGFmWEFkRXc
Thank you for your time.

The dblog.txt just looks like it is processing blocks...

Code:
Log file opened at 17:23:09.000: P:\Users\avt\AppData\Roaming/Armory/dbLog.txt
-INFO  - 17:23:09.000: (..\main.cpp:29) Running on 4 threads
-INFO  - 17:23:09.000: (..\main.cpp:30) Ram usage level: 8
-INFO  - 17:23:09.000: (..\BlockUtils.cpp:907) blkfile dir: g:\Virtual Machines SSD\BitconCoreDB/blocks
-INFO  - 17:23:09.015: (..\BlockUtils.cpp:908) lmdb dir: P:\Users\avt\AppData\Roaming/Armory/databases
-INFO  - 17:23:09.015: (..\lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 17:23:09.046: (c:\users\goat\code\armory3\cppforswig\BDM_Server.h:248) Listening on port 9001
-INFO  - 17:23:10.076: (..\BitcoinP2P.cpp:947) Connected to Bitcoin node
-INFO  - 17:23:13.088: (..\BlockUtils.cpp:1091) Executing: doInitialSyncOnLoad
-INFO  - 17:23:13.182: (..\DatabaseBuilder.cpp:169) Reading headers from db
-WARN  - 17:23:13.198: (..\lmdb_wrapper.cpp:1175) No headers in DB yet!
-INFO  - 17:23:13.198: (..\DatabaseBuilder.cpp:208) Found 1 headers in db
-INFO  - 17:23:13.198: (..\DatabaseBuilder.cpp:51) updating HEADERS db
-INFO  - 17:23:15.647: (..\DatabaseBuilder.cpp:268) parsed block file #1
-INFO  - 17:23:15.678: (..\DatabaseBuilder.cpp:268) parsed block file #2
-INFO  - 17:23:15.865: (..\DatabaseBuilder.cpp:268) parsed block file #3
-INFO  - 17:23:17.034: (..\DatabaseBuilder.cpp:268) parsed block file #7
-INFO  - 17:23:20.358: (..\DatabaseBuilder.cpp:268) parsed block file #11
-INFO  - 17:23:22.682: (..\DatabaseBuilder.cpp:268) parsed block file #13
-INFO  - 17:23:22.807: (..\DatabaseBuilder.cpp:268) parsed block file #15
-INFO  - 17:23:25.272: (..\DatabaseBuilder.cpp:268) parsed block file #17
-INFO  - 17:23:25.880: (..\DatabaseBuilder.cpp:268) parsed block file #19
-INFO  - 17:23:28.330: (..\DatabaseBuilder.cpp:268) parsed block file #21
-INFO  - 17:23:28.032: (..\DatabaseBuilder.cpp:268) parsed block file #23
-INFO  - 17:23:30.997: (..\DatabaseBuilder.cpp:268) parsed block file #25
-INFO  - 17:23:31.809: (..\DatabaseBuilder.cpp:268) parsed block file #27
-INFO  - 17:23:33.634: (..\DatabaseBuilder.cpp:268) parsed block file #29
-INFO  - 17:23:34.414: (..\DatabaseBuilder.cpp:268) parsed block file #31
-INFO  - 17:23:36.348: (..\DatabaseBuilder.cpp:268) parsed block file #33
-INFO  - 17:23:37.097: (..\DatabaseBuilder.cpp:268) parsed block file #35
-INFO  - 17:23:38.969: (..\DatabaseBuilder.cpp:268) parsed block file #37
-INFO  - 17:23:39.702: (..\DatabaseBuilder.cpp:268) parsed block file #39
-INFO  - 17:23:41.652: (..\DatabaseBuilder.cpp:268) parsed block file #41
-INFO  - 17:23:42.339: (..\DatabaseBuilder.cpp:268) parsed block file #43
-INFO  - 17:23:44.335: (..\DatabaseBuilder.cpp:268) parsed block file #45
-INFO  - 17:23:44.053: (..\DatabaseBuilder.cpp:268) parsed block file #47
-INFO  - 17:23:46.831: (..\DatabaseBuilder.cpp:268) parsed block file #49
-INFO  - 17:23:47.611: (..\DatabaseBuilder.cpp:268) parsed block file #51
-INFO  - 17:23:49.218: (..\DatabaseBuilder.cpp:268) parsed block file #53
-INFO  - 17:23:49.045: (..\DatabaseBuilder.cpp:268) parsed block file #55
-INFO  - 17:23:51.026: (..\DatabaseBuilder.cpp:268) parsed block file #57
-INFO  - 17:23:52.822: (..\DatabaseBuilder.cpp:268) parsed block file #59
-INFO  - 17:23:54.741: (..\DatabaseBuilder.cpp:268) parsed block file #61
-INFO  - 17:23:55.489: (..\DatabaseBuilder.cpp:268) parsed block file #63
-INFO  - 17:23:57.923: (..\DatabaseBuilder.cpp:268) parsed block file #65
-INFO  - 17:23:58.344: (..\DatabaseBuilder.cpp:268) parsed block file #67
-INFO  - 17:24:00.357: (..\DatabaseBuilder.cpp:268) parsed block file #69
-INFO  - 17:24:01.074: (..\DatabaseBuilder.cpp:268) parsed block file #71
-INFO  - 17:24:02.931: (..\DatabaseBuilder.cpp:268) parsed block file #73
-INFO  - 17:24:03.664: (..\DatabaseBuilder.cpp:268) parsed block file #75
-INFO  - 17:24:05.598: (..\DatabaseBuilder.cpp:268) parsed block file #77
-INFO  - 17:24:06.410: (..\DatabaseBuilder.cpp:268) parsed block file #79
-INFO  - 17:24:08.328: (..\DatabaseBuilder.cpp:268) parsed block file #81
-INFO  - 17:24:08.062: (..\DatabaseBuilder.cpp:268) parsed block file #83
-INFO  - 17:24:10.824: (..\DatabaseBuilder.cpp:268) parsed block file #85
-INFO  - 17:24:11.604: (..\DatabaseBuilder.cpp:268) parsed block file #87
-INFO  - 17:24:13.710: (..\DatabaseBuilder.cpp:268) parsed block file #89
-INFO  - 17:24:14.100: (..\DatabaseBuilder.cpp:268) parsed block file #91
-INFO  - 17:24:16.331: (..\DatabaseBuilder.cpp:268) parsed block file #93
-INFO  - 17:24:16.737: (..\DatabaseBuilder.cpp:268) parsed block file #95
-INFO  - 17:24:18.827: (..\DatabaseBuilder.cpp:268) parsed block file #97
-INFO  - 17:24:19.451: (..\DatabaseBuilder.cpp:268) parsed block file #99
-INFO  - 17:24:21.276: (..\DatabaseBuilder.cpp:268) parsed block file #101
-INFO  - 17:24:21.932: (..\DatabaseBuilder.cpp:268) parsed block file #103
-INFO  - 17:24:23.663: (..\DatabaseBuilder.cpp:268) parsed block file #105
-INFO  - 17:24:24.381: (..\DatabaseBuilder.cpp:268) parsed block file #107
-INFO  - 17:24:26.097: (..\DatabaseBuilder.cpp:268) parsed block file #109
-INFO  - 17:24:26.861: (..\DatabaseBuilder.cpp:268) parsed block file #111
-INFO  - 17:24:28.796: (..\DatabaseBuilder.cpp:268) parsed block file #113
-INFO  - 17:24:29.186: (..\DatabaseBuilder.cpp:268) parsed block file #115
-INFO  - 17:24:31.354: (..\DatabaseBuilder.cpp:268) parsed block file #117
-INFO  - 17:24:31.822: (..\DatabaseBuilder.cpp:268) parsed block file #119
-INFO  - 17:24:33.741: (..\DatabaseBuilder.cpp:268) parsed block file #121
-INFO  - 17:24:34.474: (..\DatabaseBuilder.cpp:268) parsed block file #123
-INFO  - 17:24:36.159: (..\DatabaseBuilder.cpp:268) parsed block file #125
-INFO  - 17:24:36.955: (..\DatabaseBuilder.cpp:268) parsed block file #127
-INFO  - 17:24:38.608: (..\DatabaseBuilder.cpp:268) parsed block file #129
-INFO  - 17:24:39.201: (..\DatabaseBuilder.cpp:268) parsed block file #131
-INFO  - 17:24:41.167: (..\DatabaseBuilder.cpp:268) parsed block file #133
-INFO  - 17:24:41.713: (..\DatabaseBuilder.cpp:268) parsed block file #135
-INFO  - 17:24:43.850: (..\DatabaseBuilder.cpp:268) parsed block file #137
-INFO  - 17:24:44.177: (..\DatabaseBuilder.cpp:268) parsed block file #139
-INFO  - 17:24:46.127: (..\DatabaseBuilder.cpp:268) parsed block file #141
-INFO  - 17:24:46.767: (..\DatabaseBuilder.cpp:268) parsed block file #143
-INFO  - 17:24:48.530: (..\DatabaseBuilder.cpp:268) parsed block file #145
-INFO  - 17:24:48.029: (..\DatabaseBuilder.cpp:268) parsed block file #147
-INFO  - 17:24:51.291: (..\DatabaseBuilder.cpp:268) parsed block file #149
-INFO  - 17:24:51.665: (..\DatabaseBuilder.cpp:268) parsed block file #151
-INFO  - 17:24:53.553: (..\DatabaseBuilder.cpp:268) parsed block file #153
-INFO  - 17:24:54.863: (..\DatabaseBuilder.cpp:268) parsed block file #155
-INFO  - 17:24:56.189: (..\DatabaseBuilder.cpp:268) parsed block file #157
-INFO  - 17:24:56.063: (..\DatabaseBuilder.cpp:268) parsed block file #159
-INFO  - 17:24:58.795: (..\DatabaseBuilder.cpp:268) parsed block file #161
-INFO  - 17:24:59.746: (..\DatabaseBuilder.cpp:268) parsed block file #163
-INFO  - 17:25:02.211: (..\DatabaseBuilder.cpp:268) parsed block file #165
-INFO  - 17:25:02.757: (..\DatabaseBuilder.cpp:268) parsed block file #167
-INFO  - 17:25:04.629: (..\DatabaseBuilder.cpp:268) parsed block file #169
-INFO  - 17:25:05.128: (..\DatabaseBuilder.cpp:268) parsed block file #171
-INFO  - 17:25:07.515: (..\DatabaseBuilder.cpp:268) parsed block file #175
-INFO  - 17:25:16.532: (..\DatabaseBuilder.cpp:268) parsed block file #179
-INFO  - 17:25:22.944: (..\DatabaseBuilder.cpp:268) parsed block file #181
-INFO  - 17:25:24.034: (..\DatabaseBuilder.cpp:268) parsed block file #183
-INFO  - 17:25:29.433: (..\DatabaseBuilder.cpp:268) parsed block file #185
-INFO  - 17:25:32.460: (..\DatabaseBuilder.cpp:268) parsed block file #186
-INFO  - 17:25:33.396: (..\DatabaseBuilder.cpp:268) parsed block file #187
-INFO  - 17:25:35.486: (..\DatabaseBuilder.cpp:268) parsed block file #189
-INFO  - 17:25:40.416: (..\DatabaseBuilder.cpp:268) parsed block file #191
-INFO  - 17:25:40.447: (..\DatabaseBuilder.cpp:268) parsed block file #193
-INFO  - 17:25:49.557: (..\DatabaseBuilder.cpp:268) parsed block file #195
-INFO  - 17:26:00.493: (..\DatabaseBuilder.cpp:268) parsed block file #198
-INFO  - 17:26:01.195: (..\DatabaseBuilder.cpp:268) parsed block file #199
-INFO  - 17:26:04.487: (..\DatabaseBuilder.cpp:268) parsed block file #201
-INFO  - 17:26:08.574: (..\DatabaseBuilder.cpp:268) parsed block file #203
-INFO  - 17:26:14.096: (..\DatabaseBuilder.cpp:268) parsed block file #205
-INFO  - 17:26:18.589: (..\DatabaseBuilder.cpp:268) parsed block file #206
-INFO  - 17:26:19.447: (..\DatabaseBuilder.cpp:268) parsed block file #207
-INFO  - 17:26:28.121: (..\DatabaseBuilder.cpp:268) parsed block file #209
-INFO  - 17:26:30.586: (..\DatabaseBuilder.cpp:268) parsed block file #211
-INFO  - 17:26:46.685: (..\DatabaseBuilder.cpp:268) parsed block file #213
-INFO  - 17:26:47.683: (..\DatabaseBuilder.cpp:268) parsed block file #215
-INFO  - 17:26:59.196: (..\DatabaseBuilder.cpp:268) parsed block file #219
-INFO  - 17:27:15.483: (..\DatabaseBuilder.cpp:268) parsed block file #221
-INFO  - 17:27:18.338: (..\DatabaseBuilder.cpp:268) parsed block file #223
-INFO  - 17:27:23.829: (..\DatabaseBuilder.cpp:268) parsed block file #227
-INFO  - 17:27:32.048: (..\DatabaseBuilder.cpp:268) parsed block file #231
-INFO  - 17:27:44.296: (..\DatabaseBuilder.cpp:268) parsed block file #235
-INFO  - 17:27:56.823: (..\DatabaseBuilder.cpp:268) parsed block file #239
-INFO  - 17:28:08.429: (..\DatabaseBuilder.cpp:268) parsed block file #243
-INFO  - 17:28:27.586: (..\DatabaseBuilder.cpp:268) parsed block file #247
-INFO  - 17:28:42.453: (..\DatabaseBuilder.cpp:268) parsed block file #249
-INFO  - 17:28:43.421: (..\DatabaseBuilder.cpp:268) parsed block file #251
-INFO  - 17:28:59.629: (..\DatabaseBuilder.cpp:268) parsed block file #255
-INFO  - 17:29:19.379: (..\DatabaseBuilder.cpp:268) parsed block file #259
-INFO  - 17:29:48.738: (..\DatabaseBuilder.cpp:268) parsed block file #263
-INFO  - 17:30:12.575: (..\DatabaseBuilder.cpp:268) parsed block file #267
-INFO  - 17:30:35.053: (..\DatabaseBuilder.cpp:268) parsed block file #271
-INFO  - 17:31:00.171: (..\DatabaseBuilder.cpp:268) parsed block file #275
-INFO  - 17:31:19.874: (..\DatabaseBuilder.cpp:268) parsed block file #279
-INFO  - 17:31:39.265: (..\DatabaseBuilder.cpp:268) parsed block file #283
-INFO  - 17:31:53.929: (..\DatabaseBuilder.cpp:268) parsed block file #287
-INFO  - 17:32:10.434: (..\DatabaseBuilder.cpp:268) parsed block file #291
-INFO  - 17:32:31.307: (..\DatabaseBuilder.cpp:268) parsed block file #295
-INFO  - 17:32:52.866: (..\DatabaseBuilder.cpp:268) parsed block file #299
-INFO  - 17:33:14.316: (..\DatabaseBuilder.cpp:268) parsed block file #303
-INFO  - 17:33:43.348: (..\DatabaseBuilder.cpp:268) parsed block file #307
-INFO  - 17:34:03.831: (..\DatabaseBuilder.cpp:268) parsed block file #311
-INFO  - 17:34:23.159: (..\DatabaseBuilder.cpp:268) parsed block file #315
-INFO  - 17:34:49.914: (..\DatabaseBuilder.cpp:268) parsed block file #319
-INFO  - 17:35:20.724: (..\DatabaseBuilder.cpp:268) parsed block file #323
-INFO  - 17:35:41.049: (..\DatabaseBuilder.cpp:268) parsed block file #327
-INFO  - 17:36:13.203: (..\DatabaseBuilder.cpp:268) parsed block file #331


legendary
Activity: 3640
Merit: 1345
Armory Developer
June 04, 2017, 10:20:33 AM
#7
mmap'ing is to hint the OS to use unaddressed memory to cache on disk data. It doesn't prevent other processes from requesting memory. If the DB was actually addressing 100% RAM, it would get killed by the OS.

The slowing down has more to do with CPU load than memory load. You can reduce the thread count and memory usage of the DB with the --thread-count=X and --ram-usage=X command line arguments.

At any rate, post dbLog.txt for the good measure.
newbie
Activity: 8
Merit: 0
June 04, 2017, 10:09:07 AM
#6
OK, but building database never gets finished, and my computer is not responsive due to the memory filled up.

In beginning of the database build, the estimated finish time is about 2 minutes, and then I see memory getting filled up.
When memory is filled up, the estimated time to finish only increases and never finishes...
After the memory is filled up, I also see a lot of disk activity to the pagefile.sys; the computer is swapping out memory to disk...

edit:
mapped memory, is'nt that the 'standby' memory (blue color in resource monitor).
The ArmoryDB takes up the 'in use' part (green color).
legendary
Activity: 3640
Merit: 1345
Armory Developer
June 04, 2017, 07:21:38 AM
#5
That's just mmap'ed memory, and that only happens when bootstrapping the DB.
newbie
Activity: 8
Merit: 0
June 04, 2017, 06:38:15 AM
#4
In resource manager, the memory fills up with 100% 'In use' (green bar).
The proces ArmoryDB notes: Hard faults: 535 Commit KB: 825,596 Working Set KB:22,128,568 Harable: 21,347,784 privat: 780,784.

When ArmoryDB is stopped, the used memory falls bak to 1 GB.
I dont know if I can post a picture....

Thanks
legendary
Activity: 3640
Merit: 1345
Armory Developer
June 04, 2017, 06:29:22 AM
#3
Are you looking at the system memory load or the process addressed memory? Cause these 2 aren't the same.
newbie
Activity: 1
Merit: 0
June 04, 2017, 12:33:34 AM
#2
I'm having the same problem.

Windows 7 x64
i7
16GB ram
newbie
Activity: 8
Merit: 0
June 03, 2017, 04:49:50 PM
#1
I am trying to get Armory up and running.

When I start Armory and go past the first screens, the database starts building.
Then the memory starts filling up to 100%, with ArmoryDB using about all of it, which makes the computer barely usable off course.
Building database never finishes.


System:
Bitcoin core v0.14.1 up and running
Tried Armory 0.96 and '0.96.0.1 testing', both the same problem
Windows 7
Core i7
24 GB RAM

Thanks
Jump to: