Topic: Upgraded to 0.96 building database takes long (Read 1752 times)

All is clear thanks again

It means it does not maintain the dataset to resolve arbitrary tx hashes. On the GUI side that means you can't input tx nor fee of incoming transactions (people paying you). This is the state 0.94 was in.


Well yes and no. You can keep on using this DB type but due to a mess up on my end, you have to specify the --db-type on each run. I fixed that issue but there's no build out with it yet. You're not supposed to be able to swap types once a DB is instantiated, but again, I messed up the check.

If you don't specify it, it will corrupt the DB on the next run. You can use armoryqt.conf to force that arg for now. Once the fix goes into the next build, you won't have to worry about the arg after bootstrapping.


I can't tell. Clearly there is an issue that only exists with slow/old machines, since I never run into it, and all cases I can remember like yours are on old/weak machines. For now there seems to be a critical point at which a machine is too aged/weak and can't bootstrap the DB off of defaults. Until the point I figure out what's going on, you are stuck with using custom settings. What you ran is basically the fail safe setting.

I don't think the issue is related to block size but rather block load. Smaller blocks with lighter transactions (up around block #350k) are easy enough to parse. It's the recent stuff that's grinding some machines to a halt. Seeing you eventually got past it, I'm gonna guess your machine can keep on going.


False positives. A bunch of debug verbose I left there cause people on OSX are complaining about slow shutdowns. This helps me figure out what's going on. Ignore on anything but OSX.

It works! Thanks so much for the help!  

I tried to look up what DB_BARE means and I think it means I'm still a fill node (through bitcoin core) and everything is validated fully but some hash information is not stored making it some searches from Armory less rich. Is that correct? Can I just keep running like this?

Considering the difficulty I had with getting this to run does that mean relatively soon I won't be able to run Armory online on this computer or shouldn't I worry about that?


Edit:

When I close the client I see the following errors in the log (and it takes a while to close the client meanwhile it's not responding). Is that all ok?

Yes.

That works but it's done quite fast:


I ran it like: ArmoryDB.exe --datadir="XXX\Armory" --satoshi-datadir="XXX\Bitcoin" --db-type=DB_BARE --ram-usage=1 --thread-count=3

Next start the client?

Start the DB manually. Ignore the client for now.

It shows this pop-up:



But dblog.txt doesn't show anything. In Armorylog.txt I found:

2017-07-08 14:11:50 (ERROR) -- ArmoryQt.py:1819 - Failed to start Armory database: cannot concatenate 'str' and 'int' objects
Traceback (most recent call last):
  File "ArmoryQt.py", line 1797, in startArmoryDBIfNecessary
  File "SDM.pyc", line 392, in spawnDB
TypeError: cannot concatenate 'str' and 'int' objects

Try with --ram-usage=1 --thread-count=3. No need to wipe the DB this time.

Yes and yes.

I didn't build it but downloaded the 0.96.0.4 release from github.

If you mean whether I'm starting Armory with --db-type==DB_BARE as a command line argument, no I'm not. Should I try that? Do I need to delete the database folder again?

Are you building with --db-type==DB_BARE?

This is with 0.96.0.4?

At this point it's probably still failing. Delete dbLog.txt and your databases folder, then start from scratch and post the log.


Legacy from the older versions. I'll fix that one of these days. All versions I've put out are x64 indeed.

Ok, I tried the latest #4 testing build last night until this morning. It ran for 12 hours and got to 4% (1.5 months left) and slowed my PC down to a very slow pace like before and was consuming most of the memory.

Does this mean the bug is still there or should I just let it run for >24 hours (or much more?)

BTW: I saw that the Armory installer is installing into "program files (x86)" folder by default while it's a 64 bit application. Is that correct?

Ok thanks for all the help so far.

I found 2 more lines in my log that might help so I'm going to leave them here:

Not when it triggers this issue.



I don't know that they are related yet.


Once it's ready and tested. Can't tell you more.

Ok, so if I leave it running like this it will never finish?

Does this also have something to do with the weird big that started all this? (The, non-existing (?) outgoing transaction that Armory showed with the associated lower balance)

When are you planning to release the testing build? (I have to travel in a few days).

No that means you synced your chain with a super old version of Core and never had to nuke it since. Nothing wrong with that, but the size of the early block files is a tell.


No this means your computer is so weak that you can 100% trigger this issue even on top of a completed build. I've got a VM gimped enough that I ran in the issue every other 5 scans now (I've never managed to reproduce the issue before), so I think I can get it under control. You'll have to wait for testing build with the fix or build it yourself once I push.

Okay, is having an old copy a problem? I always run a bit behind for security reasons (want to give it some time in the wild for bugs to be found). Version 0.13 is only 10 months old. Or is this because my blockchain is based on even older versions of Bitcoin and upgraded every time?

Meanwhile I started ArmoryQT by itself. I blazed passed the first steps (organizing chain, building database) and again is hanging my computer while scanning transaction history. Anything I can see or do?

Edit:

ArmoryDB memory usage is 3.6GB. Total memory usage is 93-99% ArmoryQT UI is hanging (like always, the little spinning banner is frozen)

Edit2:

Memory usage dropped to 1.4GB, 67% of total system memory usage.

You have an old copy of the chain, that explains it. Just start Armory the regular way now, see how that goes.

I'm using Bitcoin core version 0.13.0.

The blkxxxxx.dat files under Bitcoin data dir /blocks are 127-128 MB except for blk00000, blk00001, blk00002 and blk00003 (which are 1.95GB, 1.95GB, 1.95GB and 1.53 GB respectively). The last blk file is indeed blk00852.data

Are you using a very old copy of the chain? How big are the first blkXXXXX.dat files in your blockchain data folder?

Bitcoin core is fully synced. What can be wrong? Why is it not seeing the block data?

You are short like 5GB worth of block data.

Ok I'm running the bitcoin core client now to sync back up (15 hours behind) and then I'll try it and report back.

I have an alternative datadir and satoshi datadirr so I run it like:

ArmoryDB.exe --datadir=LOCATION --satoshi-datadir=LOCATION  ?

BTW:
I went into the settings (ArmorySettings.txt) and saw that a setting called: Wallet_MYWALLETNAME_BelongsTo is unset. Is that related in anyway?

Yes

So only armorydb.exe (not armoryqt.exe)? With ArmoryDB.exe --datadir=LOCATION --satoshi-datadir=LOCATION

My specs:

AMD Phenom II X6 1055T Processor @ 2.80 GHz (6-core)
8 GB RAM
AMD Radeon HD6800 video card

For some reason you managed to get the mem leak bug. Shutdown Bitcoin Core and start the DB alone. What are your system specs btw?

Processor was low (a few percent) RAM of the system was changing between 95-99%. The Armorydb.exe process started at 3 GB and slowly bled down until 0.7 GB over time but the total system RAM stayed at 95-99% (I have 8 GB of RAM).

Anyway just now, windows gave me a popup asking me to close armorydb.exe to prevent information loss since the system was low on resources. I declined but it crashed slightly later anyway. Is 8GB of RAM not enough anymore to run Armory?

What's the process CPU load? RAM load?

It just changed to 1% (1.5 months). Is this really going to take 100 times longer? Can I run with lower RAM usage (maybe also CPU?) as the PC is completely unusable now. If I move the mouse it doesn't react for a few minutes.

Edit:

After a lot of waiting I managed to check the write time stamps of the logs. Dblog.txt has not been written to in 17 hours, armorycpplog.txt and armorylog.txt have not been written to in 16.5 hours.

Edit:

Over an hour later then my previous post and 2% 3 weeks now

The process is a bit further (it passed build databases) but at least since 7 hours ago (maybe more it was in the am) it's been stuck on "scanning transaction history" 0% 4 months.

Anything I can do? Is it likely still going ok? Any idea how much more it should take? (I can't use my main computer).

Thanks

There is nothing in your log indicating an issue, just let it finish.

Post your dbLog. If you want to see the command prompt, you'll have to start ArmoryDB from within a prompt.

It seemed to run pretty well, up until the moment it says it takes 4 seconds more for building databases. It's been hanging there for an hour now. There is no open command line window like last time to see what the progress is. Is it still continuing or did it hang here?  (I have the task manager running and I see the RAM usage of ArmoryDB is still slowly going up: about 2.55 GB).

Yes


No

Ok, doing that now. All default settings? (so e.g. for RAM)

Edit:

And do I need to re-delete the database or can I keep the current state?

Another 300 or so. Kill the process and use the version I pointed you to.

Sorry you did. After I started the process already though (I thought I removed this topic). Did you want me to kill the process and start over?

It's now at block #610, how many more are there?

I explicitly told you to use the testing builds.

Hi, roughly 14 hours ago I upgraded to Armory 0.96 from an old version, deleted the databases from the datadir folder and started rebuilding the database.

By now armoryDB got to block file 106 (there are over 500 right?). At this pace it will take almost 3 days. Is there any way to speed it up? I ran with default settings (RAM mode 4 and it's using all 6 cores of my machine). I have 8 GB of RAM.

Thank you

Edit:

I run Windows 7 as the OS