Topic: Extremely long engine initialization, v 0.95.1 (Read 2427 times)

The "Tower PC" was using its main SSD for the Bitcoin database but initially, the laptop PC was also using its SSD (and its faster than the one in the Tower PC) and it was just as slow on the laptop as when using the NAS.

The NAS has write speeds similar to what many people have in their PCs unless they have SSDs, so that's probably not the issue.

One thing I noticed - the Tower PC CPU usage was around 30% and it did not feel overloaded at all. I could browse the net, for example, without noticing any performance degradation. On the less powerful laptop PC, the CPU usage was only around 5-7% according to task manager, but it felt much slower than that. Badly managed hyperthreading maybe?

Really odd.

Both pc running against the NAS? Can you post both specs next to each other for comparison?

Update: So I installed Bitcoin Core on the other PC I mentioned (Core i7 4790K etc) Yesterday and it's almost finished syncync now, downloaded over 75GB since Yesterday and most of the time the PC actually was sleeping as Windows 10 in their eternal wisdom did not believe Bitcoin Core was doing anything and let the PC enter hibernation.

In other words, on the more powerful PC, connected to the same network as the core i7 laptop, it's exponentially faster. Probably around 15x+ faster...

I dont recall anyone complaining about that in recent memory. At this point you should ask Core directly.


LevelDB is flaky on Windows. It has gotten a lot better with the years but a bad shutdown or basically anything else can completely destroy Core's DB.

I start Armory every day and I had my version .93 stuck on "Initializing Bitcoin Engine" 4th December 8 hours after I had shut it down the night before, never been initializing for so long so I thought maybe it is because it has something to do with the new version and I should get it.
So I did and still the same.
I wish the initializing stage would tell us that something is in fact happening as we don't know if the rotating icon means it is only looping forever in some hopeless routine.
Fibre internet with plenty of PC grunt.
Not sure what to do with it, have never had to fiddle with it, always shuts down correctly.

Decided to start Bitcoin Core up, it wanted to re-index the whole lot, why would that be ? Not very convenient having to wait "4 years 18 weeks" so I can use it again.
Hoping this is the solution to the Armory problem however.
Has to be a good reason why this can fall over in this way.

That's the part I do not fully understand - the CPU is not weak as we discussed before, it's a Core i7, albeit a laptop version in a relatively old Lenovo Thinkpad T520 with 8GB RAM and my Internet connection is 240/25Mbps cable - although I can get only around 180Mbps due to old router but that's more than good still I suppose.

I mean, it does not feel like it's broken, it's definitely syncync as the NAS is basically unusable now and working on the laptop is a pain. It's just slow. Extremely slow. Couldn't perhaps Windows Defender or firewall be interfering with the process?

I can also test it on a game rig I have access to (ex-BTC mining machine, though upgraded to a very fast core i7 4790K with 16GB RAM).

This is basically your last resort, albeit I can't think of any other issue causing your grief. Your only other solution is to let me SSH/TeamViewer into your machine so I can debug the code directly.

10 days is a bit excessive, but if you have bad connectivity to the network and/or a slow CPU, it will take days. It would prolly take my machine over 12h to sync the full chain now (for info, it builds & scans Armory's DB in 4min total, and I'm on fiber internet).


Since the headers first change in Core, the chain data commit is random, which will grind a HDD to a crawl. The process is very CPU intensive too, courtesy of ECDSA. It will routinely max your cores.

It *almost* worked for me in the end, but it crashed at the armoryDB.exe part still. I reinstalled Armory and without wallet it worked, then when I imported the old wallet it happened again after trying to sync.

So i thought maybe the blockchain got corrupt (Bitcoin Core was synced but there were no wallets in it). Anyway, I am now running Bitcoin Core getting fresh blockchain. Problem is, it seems it will take at least 10 days to fully sync. Is it normal these days?

I am running it on the NAS and while the CPU and RAM usage is relatively low, the NAS is hammered badly. I can hear its 6 RAID-6 connected HDDs working all the time now. The NAS has write speed of around 100MB/s.

When I was syncing the blockchain on the internal SSD, the laptop was unusable almost completely.

There's a clickable figure in the wallet's properties dialog next to "Addresses Used", in export mode.


That's benign. Still, pastebin your log file.

Sigh. Thank you for continuing to maintain Armory, hadn't realized the tribulations.

Sorry to hear your tale of woe, Mine is far simpler.

Life happens, been busy, haven't touched BTC in about 12 months. Consolidating coin. Forgot about Armory wallet with big pile of coin in it. Start Armory 0.91, too old, needs bitcoin/etc updates, Update, hangs/crashes repeatedly. Backup wallet (digital, unencrypted), remove all traces of Armory, install fresh, import wallet, have repeated troubles with 0.95.1 taking forever to start, like others in this thread. Failing to start at all, like others in the thread, but I've been trying for a week..

.. and here we are.


Been all over the client, in expert mode, can't find the "address chain length", either where to change it, or extend it.

I do see a decent amount of "-ERROR - 1480706621: (..\BitcoinP2P.cpp:865) caught BitcoinP2P_Exception in processDataStackThread: invalid header size" in the ArmoryDB console however. I'll work on getting you logs. Pastebin ok?

Block count in lower right corner is: {will update once we're not OFFLINE again from restarting the client to get to expert mode}

Your problem is different from the OP's. You also didn't post your log files as requested.

Consider that for each vocal user seeking help here, there is a silent majority that parses these threads and fix their setup on their own. Isolating each issue in its own thread enhances search results and readability.


In this case Armory is allocing <1GB memory but caching its reads in whatever left over RAM (mmap memory does NOT spill over the swap). This in turn reduces the load on your SSD, as this data is read over several times. As for the SSD load, there's some 90GB of data to process at this time. It will have to be read one way or another.

I don't particularly care to change your appreciation of the code. This is more of an informative point for the silent majority. You obviously made your opinion of me before you started posting.


Do you understand mine? This project went from being funded, with 6 full time developers and 2 interns, to getting bombed and leaving me working on it alone, out of my pocket. There was not even a plan to further development of the FOSS version past 0.93. Armory would be nowhere if I didn't volunteer over a year of my time and money to keep it going. And with that I have to catch up on the technological debt cumulated over 2 years FOSS development stagnation.

Imagine my joy when people who did not participate to the testing phases, provide no feedback or code reviews, let alone contribute some code or help with maintaining the webpage, inaugurate their first post in this forum will all spite and venom.

But at any rate, this path doesn't really achieve anything. Reread yourself. Then put yourself in the position of the people that could help you. Do you really think your post did anything but deter them from helping?


This is not a product but a project. It's always in development, always trying to get better, always in need of feedback, testing and code reviews. This is also beta software that you run at your own discretion. You are approaching this from an entitled perspective. The Bitcoin world will be full of bad experiences if you stick to that view.


Wrong again. A "free product" still has a monetization scheme. You may not be aware of it, but it is still there. Facebook and Google data mine you. Youtube feeds you advertisement. Free to play games come with micro transactions and pay to win schemes.

With Armory, there is none of that. There was quite a push under ATI to go down that path, but under my rule it has strayed as far as possible from this path, to deliver the leanest solution possible. The first I did was to get rid of the phone home code, which was intended to track user statistics for revenue projections and attracting new investors.


Jesus... try syncing 0.93 with the current chain, let's talk again after.


Your coins are not being held hostage. Your private keys are in your wallet, free for you to use. You can access them anytime with offline Armory, with versions as old as 0.90. Your private keys are your coins. Armory is a framework among others that deploys services to interface with the bitcoin network. Why do I even bother trying to explain this...


What is the address chain length on your wallets? Did you restore them from a paper backup before syncing 0.95? Try extending the address chain in offline mode before rescanning. Also, what is the top block displayed in the bottom right? Log files would go a long way.

People providing data about about the same problem isn't a hijack, it's an attempt to help find a solution to a problem.


You are correct, I misread it.. it hammers the machine so hard it -acts- like it's completely out of ram and into swap.




True.

I hope you understand peoples frustration here...a product, free or not, they've been using, has become, for whatever reason, impossible to use, monopolizing their machines, and most importantly, holding their BTC hostage.

Speaking of which, after another over-night run, the client finally appears happy and caught up, except all the wallets have zero balance. Rescan gave zero change.

help?

1) Please don't hijack somebody else's thread, create your own and post your log files.

2) There is a difference between allocated memory (malloc) and mapped files (mmap). The usage figures show you total RAM load, but the distinction is significant: allocated memory belongs to the process while mapped files boils down to hinting the OS that you want these files cached in RAM for faster access. Mapped memory does not prevent new allocations, it simply uses otherwise free resources to speed up I/O.

You are better off documenting yourself on these topics than just jumping to conclusion. Armory has been using this model for nearly 3 years now.

3) This is an open source project, feel free to submit code if you have a better solution, or at least participate to the testing phases and provide constructive feedback and bug tracking.

Armory doesn't reproduce the entire block chain anymore, so the first clue that something is wrong with your set up is that you are using 200GB of storage (the block chain is currently about 90GB).

I recently did a clean install (no wallet) of Armory on a 12 year old PC with a Q6600, 4GB of RAM, and an 120GB value tier SSD. It was synced and running in 15 minutes (I already had the block chain since I run a full node).

Have you posted your log files? That's the first step in getting informed help on this sub-forum.

What I find unreasonable is people who have certain expectations when dealing with the block chain and the software surrounding it. 90GB is a sizeable amount of data to work with on home PCs with countless combinations of hardware. If you want the security that comes with having a local copy of the block chain, you have to expect some trade-offs. These devs (Core and Armory) are providing users with free software that is capable of protecting large sums of money. While the software may not run perfectly every time for every user, after some basic troubleshooting, usually the problem can be found.

Been having a hell of a time the last week trying to get Armory to start.. Same symptoms, taken the same steps as others, still haven't been able to get it ever "complete" so I could move coins around.

FWIW, even with the suggested command line switches, ArmoryDB still uses 100% of the system RAM within 60 seconds.

Running on a 8 core 32G machine, all SSD.. shouldn't be this bad.

Running Bitcoin-QT/BitcoinD on their own, just fine, current and up to date on the block count.
ArmoryDB.exe --db-type=DB_BARE --ram-usage=1 == ram flatlined, sits on Organizing Chain a *LONG* time.. have waited a couple times then it's building TX -- FOREVER.

Once I get my coins out, I'm not -ever- running Armory again. To have to dedicate 2x the blockchain size just to run the client is unreasonable. 200G of SSD, 8 cores 32G to run a wallet?

I have just left prime95 running the Blend test for over 2 hours, no problems at all.

These types of errors have 4 different causes:

1) You are running a same DB folder against different blockchain folders.
2) You are running different versions of the code against the same DB folder. You should get an error output in the console and GUI in this case.
3) You have a bad/corrupted copy of the blockchain.
4) Your machine is unstable.


There is no knowing that without passing an hour worth of memtest and prime95 without errors. If you can actually pass these tests, then consider 3).


The DB code does not use shared libraries, it's all in the one binary. The only possible "left over issue" is using a DB dataset built with a previous version (i.e. 0.94 data vs 0.95 binary).

Unfortunately, no luck. Armory still crashes at the ArmoryDB.exe part. These are the last records in dbLog.txt:

-ERROR - 1480209973: (..\lmdb_wrapper.cpp:1435) Headers DB has no block at height: 0
-ERROR - 1480209973: (..\lmdb_wrapper.cpp:1415) No headers at height 0

Before the command prompt window crashed I also saw error about missing key pairs as before. The PC is really stable, I don't think that's the issue. Perhaps something is left from the old version installation somewhere still?

I am not sure if I am the only person with such problems, though I may be the only person who reported it at the forums but that's a different thing. I am not sure what caused these issues but the PC is actually an extremely stable machine running Windows 7 which I restart only about once a month. There is definitely no malware etc.

The last time I used Armory was the 0.93.x version several months ago and I did not experience any of the issues at all. Of course, there can always be another application or previous installation interfering.

Hopefully it will work in the end.

DB is done building. You have to start the client to scan it now.


People are having issues with upgrading to 0.95 because they do not wipe out the existing DB or use a different dbdir.

The issues you are experiencing are in a whole different class, and you are the only one with these. Surely if this was an in issue with the code, it would have manifested itself several times since April of this year, when 0.94 came out.

I believe the PC is actually very stable, I have not experienced a similar issue before. But now it seems it has finished building the database, well maybe.

In the command prompt window i know see this:

-INFO  - 1480142143: (..\DatabaseBuilder.cpp:268) parsed block file #688
-INFO  - 1480142189: (..\DatabaseBuilder.cpp:268) parsed block file #689
-DEBUG - 1480142189: (..\Blockchain.cpp:242) Organizing chain
-INFO  - 1480142227: (..\DatabaseBuilder.cpp:56) updated HEADERS db in 2603.85s
-INFO  - 1480142227: (..\BlockUtils.cpp:1636) Enabling zero-conf tracking


That is the last line I see in there and it does not update. There is no error recorded either though so I suppose it finished? The Armory database has 142MB.

It was all so simple last year...

Somehow, your system appears to be very unstable. This isn't the first time you report errors that can be summed up to LMDB failing entire sets of I/O operations.

I would suggest you try to sanitize and isolate your setup as much as you can. This is what I would do:

1) Uninstall Armory, make sure the binaries are gone, download 0.95.1, verify the hash and signature, then install fresh. You don't need to wipe your datadir for that purpose, only the installation folder.

2) Do not run BitcoinQt when Armory is performing it's original build & scan. This should free up a lot of resources. Maybe taxing your system puts it over in terms of stability. You can start BitcoinQt once Armory is fully built and scan.

Running the DB alone gets you through the build phase. You have to run a client against that DB at least once so it has some wallet data to scan the blockchain against.

Your steps would be:

a) Start DB alone, let it complete. Once it says "Enabling zero-conf tracking", it's ready.
b) Start the client, you will know within the client when the scan is complete.
c) Start BitcoinQt.

3) Do not build the DB on your NAS. Create a folder on your SSD, and point to it specifically with --dbdir. The final DB_BARE db is ~200MB, DB_FULL is ~1GB, you shouldn't have trouble affording the space.

4) Consider running a few benchmarks on your system to test for stability. At least one RAM, one CPU and one storage stress test.

5) If you find out your CPU isn't stable underload, you can use --thread-count to manually force the level of parallelism during build & scan operations. This defaults to maximum available cores in your system.

OK, thank you for spotting that! Not sure how it got there. However, the DB creating did not finish I am afraid. It stopped at this error:

-DEBUG - 1480128093: (..\Blockchain.cpp:242) Organizing chain
-INFO  - 1480128131: (..\DatabaseBuilder.cpp:56) updated HEADERS db in 2484.12s
-INFO  - 1480128131: (..\BlockUtils.cpp:1636) Enabling zero-conf tracking
-ERROR - 1480128133: (..\BDM_mainthread.cpp:257) caught exception in main thread: no sdbi at this key

That second prefix dash looks weird. It should be 2 minus (-) signs.

I tried running this command:

armorydb.exe --db-type=DB_BARE --ram-usage=1 -–satoshi-datadir="X:\Other\BTC\Bitcoin" --datadir="X:\Other\BTC\Armory"

but it seems there is a syntax error with the -–satoshi-datadir="X:\Other\BTC\Bitcoin" part but I just can't see it.

Given your custom folders, it should be:

X:\Other\BTC\Armory\databases

You don't have to use that folder, you can point to another one using the --dbdir command line argument as well.

OK, will do. Just to clarify - the database folder I should delete is under "Armory\databases", correct?

Create a fresh db folder, run armorydb against that with --db-type=DB_BARE --ram-usage=1. Once it's done building, start the client and let it scan.

The Errors recorded in log before the crash look like:

-ERROR - 1447113712: (..\lmdb_wrapper.cpp:152) Returning dirty key ref

Log file opened at 1480059815: X:\Other\BTC\Armory\armorycpplog.txt
-ERROR - 1480060049: (..\SocketObject.cpp:438) POLLERR error in readAndWrite

Hmm... armoryDB.exe is just crashing now right after headers scan. Any idea? The error right before the crash seem to be always somewhat different.

That's what I am doing - the databases folder was deleted so Armory is building it from scratch. It seems to be working OK now, in 12 minutes the database should be built it says.

Armory's db is matched to the copy of the blockchain it was built upon. You cannot split the 2. This means in your case you should rebuild your DB from scratch.

Update: It seems to be building the database now, I restarted Armory 3 times, each time a different problem with armoryDB.exe, but now it's building so it looks OK. Quite sensitive though...

Thanks everyone for the help! I was able to finally get the BTC database synchronized. Bitcoin Core is working. Unfortunately, Armory is not. After armoryDB.exe opens its command prompt window, Armory crashes with error "tried to use invalid lmdb iterator".

At first there was a problem with database version mismatch so I deleted the "databases" folder under Armory, then the new error happened. Any suggestions? I will post the log in a short while if I can save it before Armory crashes.

https://bitnodes.21.co/nodes/leaderboard/ should be helpful.

I believe at this point you should take the discussion to Core specialists. I'm not too sure what is the bast place to get that info, I only know of the dev channels personally. If anything, a post in D&TD or Technical Support will put you on the right path:

https://bitcointalk.org/index.php?board=6.0

https://bitcointalk.org/index.php?board=4.0

How do I find those? I tried Googling it before but most of the time it was just old info.

In my opinion, this is a major Bitcoin issue though. I do realize most users will never use Bitcoin Core but still, there if there is lack of reliable nodes, it has to cause problems. A year or so ago this was working smoothly.

Then you should consider finding a list of strong well known nodes (explorers, miners, high speed ones) and add them to your bitcoin.conf by IP just to finish bootstrapping.

The drive in the laptop is 480GB SSD but I moved the Bitcoin as well as Armory databases to my Synology NAS which has multiple 3TB HDDs connected in RAID6. When the DBs were on the SSD, the laptop was basically unusable.

Bad nodes is all I get to be honest. Or no nodes. Right now there is even no response listed for most of them. From time to time I do get a good node and transfer even 1GB in a few hours, then it disappears and I am left with the bad ones again.

Well, the bitcoind process is currently consuming 1-3% of CPU time. It does occupy the NAS quite a lot though, and my NAS is capable of 100MB/s+ writing.

That's a decent CPU. The drive size suggests SSD. I'm guessing the machine isn't the issue. Could be you are getting connected a lot of weak/bad nodes.

You should consider actively managing your connections. I'd ban any node that:

- has >500 ms ping
- is not Core
- has a version number <10.0

It's probably not a Bitcoin issue either.

The blocks need 2 major time consuming treatments:

• Downloading
• Processing

Your laptop is almost certainly choking on the processing part, not the downloading part. Bitcoin needs to d/l the blocks, but it also needs to check that the Bitcoin peers you d/l'ed them from didn't give you bad data. The amount of time it takes to process 1 block goes up noticeably as you get above around 300,000 blocks. That's because the blocks started to become very full from thereon in, there are alot more transactions to check in those blocks.

Update: Since I last posted I am still trying to sync Bitcoin Core. I wanted to start from scratch but was able to download only approximately 15 GB in a day and then it got almost stuck, so I used the old blocks database, which was over 90GB so that it would not have to download everything again.

I am able to download like 1GB *or less* per day now (I have 200Mbps+ Internet connection)! It's useless really. What is going on with Bitcoin these days? There seem to be almost no available nodes! In the debug window of Bitcoin Core I can see how the app struggles to find some that work.

I do realize it's not an Armory issue...

It seems the bitcoin-qt.exe I had was version 0.85.1 from 2013, I thought Armory updated it but I was wrong. I have downloaded the newest Bitcoin Core and will see how that works. Also, the new Bitcoin Core is installed into a different folder for some reason.

I will report back any progress.

The log file does not show any evidence of that, so something is failing either way. Try these settings and let me know.

Well, this is under File->Settings->Bitcoin Home Dir: X:\Other\BTC\Bitcoin\

I was referring to Armory not seeing that at all.

OK, I will change those settings. However, the new bitcoind path is apparently working as the blocks files are saved into the new directory etc. And beside, a few days ago I was running with the default settings and paths, and had the same issue, so I am quite not sure if it will help at all.

Tomas

There is no trace in your log file indicating that you have properly pointed Armory to your new Bitcoin data folder. There are indications that the auto managed instance of bitcoind is failing to start under these new settings.

I would suggest you change your setup to the following to clarify your custom setup:

1) In Armory, go to File -> Settings and turn off auto bitcoind management (in the first section).

2) Create a shortcut of ArmoryQt.exe. Right click it, pick properties. In the properties dialog, go to the General tab, and in the target input box, append the following:

--satoshi-datadir="X:\Other\BTC\Bitcoin" --datadir="X:\Other\BTC\Armory"

3) Create a shortcut in the same fashion for BitcoinQt.exe (you should find it here: C:\Program Files (x86)\Bitcoin\bin\). Append the following to its target:

-datadir="X:\Other\BTC\Bitcoin"

With that done, start BitcoinQt from your new shortcut, then Armory from its new shortcut. Report here afterwards. Keep in mind that armory command line arguments take 2 hyphens (--) and Bitcoin Core takes 1 (-)

You have been using the default auto bitcoind setup so far, and that setup is really only functional with all settings at their default value. I would suggest you stray away from that arrangement if you wish to use custom paths, at least for this version.

OK, here is the pastebin link:

http://pastebin.com/gbXSA9nN

I had to remove the older records as it was over 512KB but all of the current version info should be there.

Just copy and paste the contents of the file into a post. If the post is too big, put it on https://pastebin.com and post the link to the paste here.

It seems I can't attach the logfile? I cannot see the button for it, perhaps because I have too few posts?

Also, the bitcoind process consumes around 5-8% of the CPU now. I had a problem when the BTC database was on my main SSD drive, the laptop was extremely stressed and I thought it was because of the CPU usage but apparently the SSD writing was the issue.

I moved both the bitcoin and armory directories to NAS with approximately 100MB/s write speed and since then the laptop can breathe again. The CPU usage is very low now as I said, though the bitcoind process takes almost a GB of RAM.

I will post the log if someone tells me how...

Tomas

Post your logs.

You are limited by 2 main factors:

• Download speed
• Processor max clock speed

1st issue should be ok if you're using Bitcoin Core 0.10 minimum, I'm assuming you are. 2nd issue shouldn't be so terrible on an i7 laptop, but maybe the reality is different. I don't think additional processing cores help, the best results are achieved with desktop machines with the PSU wattage that can safely support > 3 GHz. The processor clockspeed matters because you've got a whole multitude of cryptographic proofs to compute for all that blockchain data you're downloading.

Hello everyone,

I used Armory for the last time several months ago, a few days ago I launched it again, version 0.93.x which is now upgraded to the latest 0.95.1.

The problem is, it has been initializing the engine for days now! I do realize there is a lot of new data to download, but it's crawling and I have very fast connection. I do see new block files are being downloaded so it's not stuck. Just very very slow.

Any idea what could be causing this? The OS is Windows 7 64bit with 8GB RAM and Core i7 (albeit an older laptop version).

Thanks!

Tomas