Topic: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. (Read 1495 times)

Thank - I will try it!

It looks like the DB is sane, you should start ArmoryQt now while ArmoryDB is running.

Hi goatpig,

I think I let it run about 18 hours.

It appeared to have concluded a process, and I created another log file - ArmoryDB_log2.txt in my updated pastebin.

https://github.com/truebar/pastebin

The last line in the terminal was:

-INFO  - 01:10:12: (BDM_supportClasses.cpp:1891) Enabling zero-conf tracking

Is this Ok, along with the early log line:

-INFO  - 00:06:28: (BDM_Server.h:263) Listening on port 9001

Also, I noticed that running ArmoryDB on the terminal did not hang in the usual 2-4 hours as it does with Armory GUI running and bitcoind
running in the background. It just kept running on without stopping.

Lastly, when I killed the terminal running ArmoryDB, it looked done, but there was still some kind of process running in it, though nothing new
came on screen.

Also updated in my pastebin is a new file armory_log3.txt.

On December 17th, I ran first BitcoinCore Qt, and let up update the blockchain. Then I launched Armory. Armory still froze, but it got farther
than usual. The Scanning Transactions progress bar went all the way to 100%, and the timer said only 1 second left. But at that point, Armory froze.

Any ideas from this as to get Armory back up and running fully together with bitcoind? It has been a pretty frustrating process so far.

Thanks much,
cobbler7

It looks like it updated the chain but has had time to do much more. How long did you let it run?

Hi goatpig,

Please see terminal output after running ArmoryDB at terminal.  File is ArmoryBD_log.txt

https://github.com/truebar/pastebin

The only thing I see in this I see of possible interest is:

-INFO  - 00:06:28: (BDM_Server.h:263) Listening on port 9001

Any ideas? Thanks much.
cobbler7

From the the terminal, run ArmoryDB. Post the output from the terminal. You don't need bitcoind running for that.

I uploaded txt files of the odt ones to the pastebin, did you get/see them?

Thanks, cobbler7

Can you please make it all text files. ODT I can't read in browser =/. I don't really want to have to download users' log files locally.

Thanks goatpig and HCP -- this updated directory below should have two netstat files, debug.log, and bitcoind last
run terminal log. Any ideas much appreciated.
Thanks,
cobbler7

https://github.com/truebar/pastebin

Thank s much goatpig and HCP for your continued help.

I have updated my pastebin https://github.com/truebar/pastebin
with the following 4 files:

-netstat run 1: scanning port 9001 with BOTH Armory and bitcoind running

-netstat run 2: scanning port 9001 with NEITHER Armory NOR bitcoind running

- armory debug.log

- screenshot of latest bitcoind terminal run

Re netstat, I still can't tell from these two files if anything is there that shouldn't be, or otherwise
is interfering with either Armory or Bitcoin.

Re armory debug.log, it is much like terminal output of bitcoind. Nothing jumps out to me as a
problem.

Re bitcoind run screenshot, this appears to me to indicate that the local bitcoin blockchain copy
is complete, or near complete, and not 100k blocks short of being complete. Any further ideas as
to what is going wrong in making Armory hang on "Organizing Blockchain" or "Scanning Tx"?

One further background fact, in case it matters. In doing re-installs of Armory and BitcoinCore,
I have been making use of a backup copy of the bitcoin blockchain that I keep on a external USB
hard drive. (This is specifically the contents of the .bitcoin directory.) For example, I get a new
computer, install BitcoinCore on it, but don't yet run it. Instead, I copy the contents of my
back up .bitcoin directory from the USB external drive to the .bitcoin directory on my new PC.
Only then do i run BitcoinCore Qt. Does/might this matter?

Thanks much,
cobblert7

You have to start bitcoind and let it sync. 100k blocks short means you're missing some 25% of the chain? The later blocks are a lot heavier then the earlier ones. This would mean a few extra hours worth of bitcoind syncing to catch up.


I checked the github page your linked in the previous thread, can't find any netstat output.


Type ArmoryDB or ArmoryQt in the terminal to start them. You are "introducing something new" in that approach in that you're not letting ArmoryQt control ArmoryDB, which reduces the degrees of freedom in the stack.

Armory isn't 100k blocks behind... your bitcoind database is.

ArmoryDB reads your bitcoin block files directly... and these lines of debug from ArmoryDB show that ArmoryDB is only able to see 626760 blocks in your bitcoin block files.


If you're not running Bitcoin Core GUI, you might want to find the bitcoin "debug.log" file... and check in that to make sure that bitcoind is actually syncing properly... you should see "update tip" messages in debug.log if it is.... like this:

Otherwise, it might also show errors as to why it isn't syncing properly.

Thanks goatpig. I will try it.

Some further comments/questions below.

goatpig
Insert Quote
1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned

cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory.
No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before
I launched Armory. I don't know why Armory should be 100k blocks behind.

2. Make sure there are no zombie ArmoryDB instances
oat
cobbler7: pretty sure no zombies. I always hard boot before trying to re-run and reconstitute
bitcoincore/Armory.

3. Make sure nothing is listening on 9001

cobbler7: I tried netstat :9001, both with bitcoin/Armory, and then on clean boot without them. Both
netstat logs are now in my updated pastebin. I can't make heads or tails of these logs. Is something
there that shouldn't be there? And if so, what should I do about it?

Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt.

cobbler7: so I don't introduce anything new, can you tell me what the "clean" default terminal commands are to launch ArmoryDB, and then after that, ArmoryQt?

Thanks much.
cobbler7

1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned
2. Make sure there are no zombie ArmoryDB instances
3. Make sure nothing is listening on 9001

Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt.

cobbler7:

My Armory logfiles can be viewed at:

https://github.com/truebar/pastebin

Any further advice most appreciated.
Thanks,
cobbler7

As goatpig has mentioned, without being able to see the log files, it's going to be very difficult to understand what exactly is happening with your setup. Are there any obvious errors in the log files?


Also, no idea why your whole computer is "freezing"... That isn't a normal situation, so possibly indicative of something other than Armory or Bitcoin Core causing issues I've had Armory basically "freeze" (or seem like it has frozen) during the initial sync/scan... but the rest of the PC was functioning just fine.

Hard to help you without logs.

Hi,
I have a quite similar issue. I am by this point a fairly experienced BitcoinCore and Armory user (since 2014 or 2015). So I have tried my usual tricks
and tilts on the Armory pinball machine, but nothing seems to work now.

I have been running Ubuntu 20.4, Armory 0.96.5, and BitcoinCore 0.21.1, on a one year old System76 laptop with 64GB ram and an
Intel® Core™ i7-10870H CPU @ 2.20GHz × 16. Plus Verizon Fios for Internet. I don't think my system is underpowered to run a BitcoinCore/
Armory combination.

My specific problem is, BitcoinCore and Armory repeatedly hang/freezes after running for 2-6 hours. It hangs on either "Organizing Blockchain".
Or if that process completes, it hangs on "Scanning Tx".

I have run it (the boot up) maybe 20x plus over the past 3 weeks, to no avail.  One time, when I wasn't there/didn't see it, it showed me my Armory wallet balances,
but when I got there, the whole computer was frozen, so I couldn't use Armory or anything, and had to reboot. A couple of other times, it completed the "Scanning
Tx" part (at 100%), but then froze my whole computer, and did NOT show me my wallet balances. But that is as far as I have got, after 3 weeks.

I have tried the usual tilts, which have worked before. Like:
- rebuild and rescan databases
- letting, and not letting, Armory run bitcoind in the background
- running bitcoind and bitcoin-qt first, and then manually launching Armory

Also, when I run bitcoind or bitoin-qt manually, BitcoinCore 0.21.1 runs normally and updates the bitcoin blockchain apparently up to
current state.

So the problem appears to be the handshaking of Armory 0.96.5. with BitcoinCore.

I realized tonight that BitcoinCore has been upgraded to 0.22.0. I am currently running an update of blockchain with that. Maybe that will solve the problem,
But I dunno if that will work.

Any thoughts/ideas on what I can do to fix this? Any feedback most appreciated.
Thanks in advance, cobbler7.



Follow-up (by cobbler7): my Armory log files can be viewed at: https://github.com/truebar/pastebin

Thank you

armorycpplog.txt may not show anything of great interest... I'd check out armorylog.txt and dbLog.txt in the first instance.

If you want more assistance, you'd need to post the contents of those log files (in full) to pastebin.com and then link the generated pastebin URL here so we can view the logs.

If you don't see any obvious errors in the log files, then you might want to use the "Rebuild and Rescan Databases" option in Armory's "Help" menu and see if that helps.

Thank you NC501c,
I will check that out.
Cobbler7

The pointers should be in the logs like goatpig requested in the previous replies.

"armorycpplog.txt" is in Armory's datadir; default paths: btcarmory.com/docs/pathing#armorydb

Hi,

I am experiencing a similar problem: Armory hangs apparently endlessly in the "organizing blockchain"
stage, and never completes despite being run all day.

I am running BitcoinCore 0.21.1 and Armory 0.96.5.

This combination worked fine for me previously, but now something somewhere must have changed.
(I am thinking maybe from sudo apt-get update and upgrade etc.)

The only new clue (perhaps) as to what has changed is I get a new error message when I try
to quit the hung instance of Armory. This new error message says:

"ArmoryQt.py is not responding"
 
Any help or pointers would be most appreciated!

Thanks much,
Cobbler7

Just to say this worked, and everything is good now. Thanks a lot for your help.

OK.  Have deleted (actually moved) contents of .bitcoin folder - assuming will all be recreated when restart bitcoin-qt.  Rebuilding now and looks like it's going ok. 

Thanks a lot for your help.

-ERROR - 1455537157: (BlockWriteBatcher.cpp:980) No block in DB at height 195803
-INFO  - 1455537160: (BlockUtils.cpp:1518) Scanned Block range in 411.828s

That's your error. Your copy of the blockchain is corrupt, you need to redownload it.

I have deleted contents of .armory/databases/, and restarted Armory.  Currently "organizing blockchain" as before.

The armorycpplog.txt is here:  http://pastebin.com/ruuVVeNK.

Thanks a lot for your help.

Again these log lines are irrelevant. You should clean up your armorycpplog.txt or all personal info and pastebin it so I can look at it. In the mean time, delete the content of Armory's database folder and try again.

Thanks.  Have done as you said:

• Started Bitcoin-qt directly
• Blockchain seemed synced, looking at the block height and activity.
• Started Armory first in offline mode, and unchecked box to stop Armory managing bitcoind
• Shutdown and restarted Armory in normal mode (leaving bitcoin-qt running). Now only 2 progress bars rather than 3
• Result appears the same - hanging on "Organizing Blockchain" - though so far it's only been 20 min
• Stuff in logs appears same as before, to my inexpert eye

(Noticed this in log, fwiw):
2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1323 - 8 attempts to load blockchain failed.  Remove mempool.bin.
2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1328 - File mempool.bin does not exist. Nothing deleted.

None of these lines are relevant to your issue. You should start BitcoinQt manually and let it fully sync. In Armory, turn off auto bitcoind management then start it after BitcoinQt. Report what happens here.

I'm trying to install Armory in Ubuntu 15.04, using the version at https://github.com/goatpig/BitcoinArmory/releases/tag/v0.93.3.  I have a few wallets already, from an installation on a laptop I had stolen.  So I'm trying to get these coins back.

Dashboard shows "Synchronizing with Network" complete.  It has finished "reading block headers", but now hangs on Organizing Blockchain.  It's been that way for maybe 18h overall (with a couple of attempted restarts in between).

Might be relevant:  I tried to give the new install a headstart by copying over some of the database files from a backup I had of the old installation (and also to the .bitcoin installation).

Also:  when I quit, it hangs on "Preparing to shut down", and I have to quit again.  When that happens I get the following repeated many times in  .armory/armorylog.txt:

2016-02-14 09:52 (WARNING) -- SDM.py:768 - Overriding not-available state. This should happen 0-5 times


Heres the cpplog.txt after re-starting, up to present moment where it hangs.

Log file opened at 1455443804: ~/.armory/armorycpplog.txt
-INFO  - 1455443847: (BlockUtils.cpp:918) blkfile dir: /home/gav/.bitcoin/blocks
-INFO  - 1455443847: (BlockUtils.cpp:919) lmdb dir: /home/gav/.armory/databases
-INFO  - 1455443847: (lmdb_wrapper.cpp:478) Opening databases...
-INFO  - 1455443847: (BlockUtils.cpp:1250) Executing: doInitialSyncOnLoad
-INFO  - 1455443847: (BlockUtils.cpp:1321) Total number of blk*.dat files: 444
-INFO  - 1455443847: (BlockUtils.cpp:1322) Total blockchain bytes: 59,390,537,918
-INFO  - 1455443847: (BlockUtils.cpp:1695) Reading headers from db
-WARN  - 1455443847: (lmdb_wrapper.cpp:1478) No headers in DB yet!
-INFO  - 1455443847: (BlockUtils.cpp:1721) Found 1 headers in db
-DEBUG - 1455443847: (Blockchain.cpp:211) Organizing chain w/ rebuild
-WARN  - 1455443847: (BlockUtils.cpp:1351) --- Fetching SSH summaries for 1342 registered addresses
-INFO  - 1455443847: (BlockUtils.cpp:1364) Left off at file 0, offset 0
-INFO  - 1455443847: (BlockUtils.cpp:1367) Reading headers and building chain...
-INFO  - 1455443847: (BlockUtils.cpp:1368) Starting at block file 0 offset 0
-INFO  - 1455443847: (BlockUtils.cpp:1370) Block height 0
-DEBUG - 1455444332: (Blockchain.cpp:211) Organizing chain w/ rebuild


And the last few lines of armorylog.txt

2016-02-14 09:56 (INFO) -- ArmoryQt.py:5887 - Dashboard switched to auto-InitSync
2016-02-14 09:57 (INFO) -- ArmoryQt.py:2525 - loadBlockchainIfNecessary
2016-02-14 09:57 (INFO) -- ArmoryQt.py:2546 - Setting netmode: 1
2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode
2016-02-14 09:57 (INFO) -- Networking.py:65 - Connection initiated.  Start handshake
2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode
2016-02-14 09:57 (INFO) -- Networking.py:156 - Received version message from peer:
2016-02-14 09:57 (INFO) -- Networking.py:157 -    Version:     70002
2016-02-14 09:57 (INFO) -- Networking.py:158 -    SubVersion:  /Satoshi:0.11.2/
2016-02-14 09:57 (INFO) -- Networking.py:159 -    TimeStamp:   1455443848
2016-02-14 09:57 (INFO) -- Networking.py:160 -    StartHeight: 398373
2016-02-14 09:57 (INFO) -- Networking.py:350 - Handshake finished, connection open!


Edit:  seems I get this error on starting from command line:

/usr/lib/armory/armoryengine/Transaction.py:2649: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(ERROR) announcefetch.py:287 - Unspecified error downloading URL
Traceback (most recent call last):
  File "/usr/lib/armory/announcefetch.py", line 277, in __fetchFile
    urlobj = urllib2.urlopen(url, timeout=CLI_OPTIONS.nettimeout)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1200, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: ('The read operation timed out',)


Grateful for any help.