Topic: Armory 0.93 testing release! (with 0.05 BTC bug bounty) - page 23. (Read 35768 times)

What's your RAM and CPU usage at again?

69% now after continuing to run overnight, now nearly 2 days

This is my very first time to report an Armory issue so if I'm doing it wrong please be gentle and forgive me.

OS: Windows 7 Premium Home Edition.
Armory Version: 0.92.99.1
Bitcoin Version 0.1+

For starters I have Bitcoin and Armory set up to write to a USB 1TB drive due to space restrictions on my local hard drive. I'm using the commands:

Armory: "C:\Program Files (x86)\Armory\ArmoryQt.exe" --settings="G:\Armory\ArmorySettings.txt" --satoshi-datadir="G:\Armory" --datadir="G:\Bitcoin"

Bitcoin: "C:\Program Files\Bitcoin\bitcoin-qt.exe" -conf="G:\Bitcoin\bitcoin.conf" -datadir="G:\Bitcoin"

What I am seeing is that if you let Armory call bitcoind (via Armory Settings) then everything works as it should. But If I try to run in the configuration where Armory expects Bitcoin-QT or bitcoind to already be running as a separate process and attach to it on start-up, the Armory screen flashes up briefly and then disappears. By doing a Ctrl-Alt-Del to see the process table, I can see that Armory-QT is still running even though the GUI has died. I have to manually kill Armory-QT in order to stop the process. I have tried this about 5 times in order to make sure the problem is re-producible. After killing the process I can restart Armory-QT and tell it to call bitcoind itself setting the Bitcoin Install Dir to "C:\Program Files\Bitcoin" and the Bitcoin Home Dir to "G:\Bitcoin" and everything works as it should.

I hope I reported this correctly.

I completely understand And btw, I'm just running normal armory, not the supernode. still scanning transactions, up to 46% now and reporting 11 hours to go

There is a wealth of solutions and yours certainly is an acceptable one. Thank you for your concern. However, the issue is about segregation of duties, not implementation. This simply is not a part of the codebase I deal with right now, and every time I ran across this bug in the past few months, I failed to pass it on to the right person.

CircusPeanut is keeping track of everything being reported in this thread and assigning tasks to the proper personnel. This may sound selfish, but I deal with the DB issues, not the rest. Maybe 90% of the DB code was changed this time around and I am solely focusing on getting that part solid. Also I don't usually answer unless the issue is dire or I have at least some sort of a fix worked in already. However I do read all the posts in this thread.

I'm already working on a couple bugs I found on my own a few days ago, although I don't expect any of you will run into those. I'm also paying attention to DB building speed with the different machines you guys run on. From that I will decide if there is need to make anything faster. Keep in mind that I expect most private users will be running fullnode, not supernode, so give that DB type some love too =)

This takes 1/2 an hour to fix, if you need a walkthrough, I'm happy to help You just need to get the links that are broken, create page's, configure them to be the URL that's now broken, and then use page links to, to send them to the correct (working) page. This would broken links across old builds, as well as this new build.

Links have been down for a while now. We're aware of the snafu but have been lazy about it I guess =(

I stopped it while it was at 41% scanning transactions, going to get rid of the swapfile, reboot, and then try again. fingers crossed! It's initialising the bitcoin engine, so hopefully it'll get through initialising the bitcoin engine, building the db's and picking up from scanning the transaction history.

It's just synced with network now, and flying through loading database headers, took about 4 minutes. Lets see how long it takes to build the database. Not very long at all, went straight from 0% to 99% straight away. Went out for a few minutes and when I came back, it's back to scanning transaction history at 41%. There isn't any noticeable difference at all whether pagefile is on or off, still taking forever. I'll leave it running overnight and hopefully my wallets will be up to date in the morning.

But



+1

Just another minor bug, I noticed noticed that the link on the welcome message isn't working

Links to https://bitcoinarmory.com/faqs (which shows page not found), should be https://bitcoinarmory.com/faq/



Btw, a quick way to fix all of these broken links is to install something like https://wordpress.org/plugins/page-links-to/ , and then you can set up the pages to link to the right one.

It's not helping that the OS is constantly moving stuff in and out of the swap, it eats disk I/O for no reason. You can try without.

You can halt the scan process at any time, even by killing the process. The DB is resilient to segfaults.

16gb ram here and quad core i7, but not ssd

now up to 34% into scanning transaction history. my cpu  is low, but disk and ram usage nearly 100%

I have a large swapfile though, so I wonder if that's what's causing the delay in my case?

As I said LMDB is a mapped DB. File mapping is entirely controlled by the OS. Depending on how much resources your other active processes require, the OS will throttle down the amount of physical RAM it makes available to hold the mapped file. This in return will crush the write speed (which needs to resolve the page on which the key is or will be before it can write in the data).

My scans take usually an extra 30-40% speed if I play old steam games in the meantime.

To give you an idea, I use a laptop with an i7 haswell, 32GB DDR3 and 3 micro SSDs in RAID0 and no swapping file. It takes me about 1h30 to copy the blocks over and 3h30 to scan supernode (last time I did it was around block 335k). About 45 minutes to scan fullnode, but that's cause the processing is performed in a single thread. Fullnode scans could be lightning fast if I multithreaded that part, as they require very little writing past the block copying. However, that again was not a priority.

I can't wait for DDR4 in laptops so that I can move to 128GB RAM

The DB creation works in 2 phases:

1) Building, which is simply copying the blockchain as a whole over to the DB. This is the phase you are in, which takes very little resources. This code is single threaded and was barely modified in this release (only to support headers first Core). It could be made faster but this isn't a priority in any way.

2) Scanning the transaction history: this part requires heavy IO and processing, and has been largely modified. It will eat all the RAM on your system and I/O capacity of your drives, take several hours/days, and runs on 2 to 4 threads (possible more to come).

I have 16 GB of RAM as well, and my swap file is on my SSD. However, I have another task using just under 9GB. Armory uses about 900MB in the scanning stage. Is Armory limiting itself because it sees that there isn't a lot of RAM to spare (still over 2GB free that Armory isn't touching), or is something else the cause of this 900MB vs +2GB thing?

But why does it use such little resources? That's what I don't understand.
About 40% of RAM is free on my system at any given time, why doesn't Armory take it? Same with IO.

You are still building the DB (copying block data over), not scanning (actually parsing the transaction data). When scanning supernode, you will be using over 2GB RAM at least.

I have 16 GB of RAM and a 1 GB swap file on my SSHD(85 MB/s read/write), it still goes extremely slowly, and I see Armory using ≤41 MB of the swap file and ≤200 MB of RAM. Armory only uses about 1 MB/s of IO right now.

Hi,

I'm responsible for the DB code. For those who care, it takes me 3h30 to build supernode, which approximates 90GB with the current blockchain size.

If you have less than 16GB of RAM and a HDD, do not expect supernode to build under a week. Your machine can't handle it. This DB type tracks ALL transactions in the blockchain, it's meant for heavy duty servers. You should at least try and see how long it will take your system to build fullnode before thinking of going with supernode.

The obvious bottleneck is I/O. Moreover, LMDB maps its whole dataset in RAM, so a machine with little RAM and a swapping file on a HDD will get crushed.

Just back in now, says it's scanning transaction history and done 26% so far, reporting 18 hours to go. So looks like it'll take from lunchtime Sunday, till tomorrow morning (Tuesday) to have it installed, downloaded the bootstrap torrent, synced from there, build the database and scan the transaction history. 48 hours is a LONG time. I know it has to be done once, but is it just me on a normal hard drive having this?

That can cause issues, whether it's the source of this problem or not. It's not a good idea to leave your system disk with anything less than about 20% free space, more so with SSD's.

Still, with plenty room on the system disk and the Armory datadir on a separate disk to that, it took me 6 hours to build the 0.93.x database, and about 5 hours for the full scan.

This isn't the issue in my case: my 2TB HDD with the Armory DB and Bitcoin block chain has at least 745GB free when doing this. My OS hard drive is a smaller SSD with 36.7 GB (out of 232) free.