Topic: Starting preliminary 0.94 testing - "Headless fullnode" - page 2. (Read 15232 times)

OK, I will do that ... but I find it really weird that neither Satoshi,
nor Armory detects the missing/corrupted block data.

(Or at least none of them complains.)

You are missing some block data starting 353055, which is why Armory can't scan any futher.

In your blocks folder, you should delete the last few blk files, probably up to blk00320.dat (315 to be on safe side) and the associated rev files (rev00320.dat and so on). Then start BitcoinQt, it will tell you its DB is corrupt and rebuild it from scratch, download the last few missing files in the process.

Once that is done, start Armory, it should recover from that.

After restarting Armory, and waiting for a long, long time again, the same situation returned.
(Scanning finished at block 353055)

This is the end of of the output:

-INFO  - 1440596237: (BlockUtils.cpp:395) reading blocks from file 322
-INFO  - 1440596239: (BlockUtils.cpp:395) reading blocks from file 323
-INFO  - 1440596241: (BlockUtils.cpp:395) reading blocks from file 324
-INFO  - 1440596243: (BlockUtils.cpp:395) reading blocks from file 325
-INFO  - 1440596244: (BlockUtils.cpp:1418) Wrote blocks to DB in 24.3227s
-WARN  - 1440596245: (BlockUtils.cpp:1114) Scanning from 353055 to 353055
-INFO  - 1440596247: (BlockUtils.cpp:1458) checking scan integrity
-INFO  - 1440596247: (BlockUtils.cpp:1646) --- bwbDtor: 0s
-INFO  - 1440596247: (BlockUtils.cpp:1647) Scanned Block range in 0.300441s
-INFO  - 1440596247: (BlockUtils.cpp:1653) Finished loading at file 325, offset 73062124
-INFO  - 1440596247: (BlockDataViewer.cpp:157) Enabling zero-conf tracking
-DEBUG - 1440596422: (Blockchain.cpp:214) Organizing chain

And indeed, it says the status is "Connected (353055 blocks)", ignoring the last ~20k blocks in the network.
(While my Satoshi's log says:  height=371612 )

currently feature-not-bug.

On the list of transactions, clicking on the header fields of the table (saying "Date", Wallet", "Comments", etc.), I can see an arrow being moved, switched up/down, moved to the column I clicked, etc, but the displayed transactions are not re-ordered accordingly.

In fact, they don't change at all.

Blockchain not scanned up to the top!

So, I have compiled Armory from github, using the ffreeze branch.
(My exact revision is 65dafec4ad888219abaa8897929364423f71ea05 )

I could successfully sync up my DB, and I also did two successful transactions.
I know they were successful, because they show up (with plenty of confirmations) on blockchain.info.

But then, after restarting Armory, the transactions don't show up.

Here is the full story:

 - I did a successful transaction, which was immediately confirmed a few times.
 - However, I noticed that armory still showed 0 confirmations.
 - I did another transaction, and Armory said that this transaction might have
   been refused by the network. I want to blockchain.info to check, and
   I saw that the transaction waiting there with 0 confirmations.
   A few blocks have gone by, and it wasn't confirmed, so I suspected that
   maybe I should have set higher fees, and so I wanted to "undo" this
   transactions ... so I told Armory (in the help menu) to
   "Clear All Unconfirmed".
 - Meanwhile, I see that eventually, my transaction _were_ confirmed OK
 - However, after restarting armory, these transactions don't show up at all.
   Neither the first one, nor the second one. Likewise, the displayed
   balance is the one before the transactions. According to armory, it's
   as if those transactions have never happened.
 - Also, the label in the bottom right corner says "Connected (353000 blocks)",
   whereas according to the .bitcoin/debug.log, the current block number is
   371598. Ha Armory forgot to scan the last 20000 blocks?

That is going on here?

What's the total size of your DB? The headers DB alone?

https://github.com/etotheipi/BitcoinArmory/blob/ffreeze/cppForSwig/mdb/mdb.c#L588

Reduce this value to 16MB and rebuild a new DB with that, see if it suffers from the bad_alloc.

I got a third crash at 20150825_2016. It looks like Armory has to have been running for 24++ hrs in order to
get to the condition that triggers the std::bad_alloc. The approximate 48 hr interval that I see is probably due to
the fixed times of day that I use the PC. I sent 1.5BTC from bitcoin-qt to Armory-watch. Armory popped up the
std::bad_alloc window a few seconds after bitcoin-qt said it had sent the TX. I think the problem occurs when
Armory receives the raw TX. After restarting, it gets the TX from the block without a problem (but that's not after
a 24++ hr delay).

The memory usage remained fairly constant for the 48 hrs, including after the exception.
  Armory:   VIRT = 1052M  RES = 288M   SHR = 19M
  Bitcoin-qt  VIRT = 1121M  RES = 404M   SHR = 12M

I'm not having any success debuging ArmoryQt.py with gdb. I'll just try to get a crash in less than 48 hrs.

So you are saying it takes 24h+ to crash? Can you keep an eye on the process memory? I'll run some tests on my own.

I've been running Armory on the laptop for about a week. The initial scan happened without incident.
It took me a few days to sort out the maxconnections issue. After that I've been running Armory
continuously. The laptop is only used to run bitcoind and Armory.

I got a first std::bad_alloc crash at 20150821_1449.
I restarted Armory under gdb.
I got the second std::bad_alloc crash at 20150823_2125.
Python had exited cleanly so there was nothing for gdb to display.
I restarted Armory under gdb, set a breakpoint at BDM_mainthread.cpp:453, and did run with --netlog.
I am expecting to have to wait a couple of days for another crash. I'll keep on making small value
transfers between Armory and bitcoin-qt. I want to see if the crash is spontaneous, or if it happens during
some user initiated activity.

We don't use realloc directly, and very little malloc anyways.

It is most likely a std container failing to get enough memory. The issue isn't so much that the addressing failed but whether it is because it ran out of RAM or it was passed a bad ptr. If this is happening in a std container, which we rely heavily upon, then it's the former.

Did the issue happen after scanning the whole chain or was it in a freshly started process?

_CppBlockUtils.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped

The machine only has 2GB dram, but it also has 12GB swap.
I think you can also get this exception from a realloc with a bad ptr.

bad_alloc usually means the process failed to allocate more RAM. I'd like to make sure this is not happening on a x86 machine first before I investigate.

it does actually say x86_64 on the readout, apologies if that's what you meant.

Could it be something to do with the instruction set on the Core 2 era? I seem to remember that Core 2 was better specified than Core 2 Duo somehow, possibly not

I'm pretty sure Core 2 can handle both x86 and x64 OSes, so this isn't really helping me. Browse to Armory's binary folder, and run this command and paste the result back:

I'm using a home-built Linux running on a HP laptop.
Linux sys 2.6.37.6 #11 SMP Sun Apr 14 21:52:49 EDT 2013 x86_64 Intel(R) Core(TM)2 CPU T5300 @ 1.73GHz GenuineIntel GNU/Linux
I used gcc-4.9.2 to build the c++11 bits of Armory. Packages: swig-3.0.6 sip-4.16.9 PyQt-x11-gpl-4.11.4 psutil-2.1.3 Twisted-15.3.0 qt-4.8.6.

Are you using a x86 OS?

When running ffreeze and bitcoind 0.11.0 in Linux I got a "BDM thread failed: std::bad_alloc" crash.
Python traps the exception so gdb has nothing to report.
I was doing a send from bitcoin-qt to Armory-watch-wallet. The exception happened when Armory received
the broadcast transaction. I restarted Armory and it then saw the transaction when the first confirmation block
arrived. Are there any debug printf's for identifying where the bad_alloc is occuring?

Oh right, there's that too. I personally do it the other way and add "addnode=127.0.0.1" to guarantee Core tries to keep the localhost connection with Armory alive.

I found that "maxconnections=6" in bitcoin.conf was causing bitcoind to reject connections
from Armory on the localhost interface. When I changed it to "maxconnections=30", Armory
started working as expected, and bitcond says it has 9 open connections.