Topic: CppnotificationHandler problem (Read 1028 times)

This is just gdb reporting threads starting and ending, nothing to see here. When the code crashes, you will see a SIGSEGV or SIGTERM at the bottom then prompt back at gdb> ready for your input.

Then post only the output for "bt" and wait on instructions.

I have changed number of threads to 15, then

1) from command line "make DEBUG=1"
2) from command line "gdb ArmoryDB"
3) run

now every couple of seconds log in gdb change and looks like:

[New Thread 0x7fff8f7fe700 (LWP 26326)]
[Thread 0x7fff8f7fe700 (LWP 26326) exited]
[New Thread 0x7fff8f7fe700 (LWP 26327)]
[Thread 0x7fff8f7fe700 (LWP 26327) exited]
[New Thread 0x7fff8f7fe700 (LWP 26328)]
[Thread 0x7fff8f7fe700 (LWP 26328) exited]
[New Thread 0x7fff8f7fe700 (LWP 26329)]
[Thread 0x7fff8f7fe700 (LWP 26329) exited]
[New Thread 0x7fff8f7fe700 (LWP 26330)]
[Thread 0x7fff8f7fe700 (LWP 26330) exited]
[New Thread 0x7fff8f7fe700 (LWP 26331)]
[Thread 0x7fff8f7fe700 (LWP 26331) exited]
[New Thread 0x7fff8f7fe700 (LWP 26332)]
[Thread 0x7fff8f7fe700 (LWP 26332) exited]
[New Thread 0x7fff8f7fe700 (LWP 26333)]
[Thread 0x7fff8f7fe700 (LWP 26333) exited]
[New Thread 0x7fff8f7fe700 (LWP 26334)]
[Thread 0x7fff8f7fe700 (LWP 26334) exited]
[New Thread 0x7fff8f7fe700 (LWP 26335)]
[Thread 0x7fff8f7fe700 (LWP 26335) exited]
[New Thread 0x7fff8f7fe700 (LWP 26336)]
[Thread 0x7fff8f7fe700 (LWP 26336) exited]
[New Thread 0x7fff8f7fe700 (LWP 26337)]
[Thread 0x7fff8f7fe700 (LWP 26337) exited]
[New Thread 0x7fff8f7fe700 (LWP 26338)]
[Thread 0x7fff8f7fe700 (LWP 26338) exited]
[New Thread 0x7fff8f7fe700 (LWP 26339)]
[Thread 0x7fff8f7fe700 (LWP 26339) exited]
[New Thread 0x7fff8f7fe700 (LWP 26340)]
[Thread 0x7fff8f7fe700 (LWP 26340) exited]
[New Thread 0x7fff8f7fe700 (LWP 26341)]
[Thread 0x7fff8f7fe700 (LWP 26341) exited]
[New Thread 0x7fff8f7fe700 (LWP 26342)]
[Thread 0x7fff8f7fe700 (LWP 26342) exited]


armoryD is running. Is it all right? Should I wait now for error and the gdb will stop? then I will may type "bt all"?

Thank You in advance for Your help

m.

I will try.

m.

If the chance for the code to fail is not reduced, that's a way to go about it.

should I decrease amount of threads to for example 20 ?

m.

You can build with DEBUG=1 and run ArmoryDB in gdb. You can then post the backtrace after it hangs, although that will require some extra work to be really useful, as you would need to show me the bt for all relevant threads (seeing you run 100 ZC threads, it will be tedious to filter through).

I have patched the code. Now Armory deamon was terminated after c.a 9 hours of running, then I have started it again and it took c.a 8-9 hours and the error happen again:

In my opinion the patch helped but there must be something more which cuase error:

New Block:  446454
New Block:  446455
New Block:  446456
New Block:  446457
New Block:  446458
New Block:  446459
New Block:  446460
New Block:  446461
New Block:  446462
New Block:  446463
New Block:  446464
-INFO  - 1483465297: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465299: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465309: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465314: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465316: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465319: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465321: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465325: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465333: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465334: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465350: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465398: (SocketObject.cpp:517) POLLIN recv return 0


Can I help You with investigating problem by running my deamon with some kinde of debug etc? I don't know wheter it will be easy to find a reason of problem beause each time I had to wait couple of hours for it.


m.

Try to patch this PR to your code base:

https://github.com/goatpig/BitcoinArmory/pull/158

If that doesn't fix it, I'll have to reproduce the issue myself and deliver the fix for 0.96.

I send 100 outgoing transactions from wallet (in c.a 30 seconds all):

and the daemon log an error like below:

(INFO) armoryd.py:3370 - New ZC tx: 2000000
(INFO) armoryd.py:3370 - New ZC tx: 33608
(INFO) armoryd.py:3370 - New ZC tx: -10527220
(INFO) armoryd.py:3370 - New ZC tx: -12913772
(INFO) armoryd.py:3370 - New ZC tx: -1990000
(INFO) armoryd.py:3370 - New ZC tx: -4193393
(INFO) armoryd.py:3370 - New ZC tx: -2020640
(INFO) armoryd.py:3370 - New ZC tx: -18500000
(INFO) armoryd.py:3370 - New ZC tx: -106172
(INFO) armoryd.py:3370 - New ZC tx: -735709
(INFO) armoryd.py:3370 - New ZC tx: -5020640
(INFO) armoryd.py:3370 - New ZC tx: -790640
(INFO) armoryd.py:3370 - New ZC tx: -320640
(INFO) armoryd.py:3370 - New ZC tx: -73220
(INFO) armoryd.py:3370 - New ZC tx: -517525
(INFO) armoryd.py:3370 - New ZC tx: -2257937
(INFO) armoryd.py:3370 - New ZC tx: -1013920
(INFO) armoryd.py:3370 - New ZC tx: -8194190
(INFO) armoryd.py:3370 - New ZC tx: -2324457
(INFO) armoryd.py:3370 - New ZC tx: -11291727
(INFO) armoryd.py:3370 - New ZC tx: -7860640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -4300000
(INFO) armoryd.py:3370 - New ZC tx: -1151953
(INFO) armoryd.py:3370 - New ZC tx: -11920640
(INFO) armoryd.py:3370 - New ZC tx: -318831
(INFO) armoryd.py:3370 - New ZC tx: -2143039
(INFO) armoryd.py:3370 - New ZC tx: -1020640
(INFO) armoryd.py:3370 - New ZC tx: -42221280
(INFO) armoryd.py:3370 - New ZC tx: -10532429
(INFO) armoryd.py:3370 - New ZC tx: -28539483
(INFO) armoryd.py:3370 - New ZC tx: -2257059
(INFO) armoryd.py:3370 - New ZC tx: -113000
(INFO) armoryd.py:3370 - New ZC tx: -2020640
(INFO) armoryd.py:3370 - New ZC tx: -4107234
(INFO) armoryd.py:3370 - New ZC tx: -9704437
(INFO) armoryd.py:3370 - New ZC tx: -15220640
(INFO) armoryd.py:3370 - New ZC tx: -4920640
(INFO) armoryd.py:3370 - New ZC tx: -79360
(INFO) armoryd.py:3370 - New ZC tx: -100000
(INFO) armoryd.py:3370 - New ZC tx: -52462500
(INFO) armoryd.py:3370 - New ZC tx: -138673605
(INFO) armoryd.py:3370 - New ZC tx: -10566460
(INFO) armoryd.py:3370 - New ZC tx: -10544487
(INFO) armoryd.py:3370 - New ZC tx: -1400000
(INFO) armoryd.py:3370 - New ZC tx: -1081640
(INFO) armoryd.py:3370 - New ZC tx: -3020640
(INFO) armoryd.py:3370 - New ZC tx: -33072396
(INFO) armoryd.py:3370 - New ZC tx: -945195
(INFO) armoryd.py:3370 - New ZC tx: -5030640
(INFO) armoryd.py:3370 - New ZC tx: -252600
(INFO) armoryd.py:3370 - New ZC tx: -223920640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -11229000
(INFO) armoryd.py:3370 - New ZC tx: -9990000
(INFO) armoryd.py:3370 - New ZC tx: -14114489
(INFO) armoryd.py:3370 - New ZC tx: -7862640
(INFO) armoryd.py:3370 - New ZC tx: -14163165
(INFO) armoryd.py:3370 - New ZC tx: -11648277
(INFO) armoryd.py:3370 - New ZC tx: -2670298
(INFO) armoryd.py:3370 - New ZC tx: -2520640
(INFO) armoryd.py:3370 - New ZC tx: -2330640
(INFO) armoryd.py:3370 - New ZC tx: -1013640
(INFO) armoryd.py:3370 - New ZC tx: -1302327
(INFO) armoryd.py:3370 - New ZC tx: -481734
(INFO) armoryd.py:3370 - New ZC tx: -18670640
(INFO) armoryd.py:3370 - New ZC tx: -5650879
(INFO) armoryd.py:3370 - New ZC tx: -10490638
(INFO) armoryd.py:3370 - New ZC tx: -931080
(INFO) armoryd.py:3370 - New ZC tx: -11244500
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -380640
(INFO) armoryd.py:3370 - New ZC tx: -1020640
(INFO) armoryd.py:3370 - New ZC tx: -1485205
(INFO) armoryd.py:3370 - New ZC tx: -2091110
(INFO) armoryd.py:3370 - New ZC tx: -1078640
-INFO  - 1483206291: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206293: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206304: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206309: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206310: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206314: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206316: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206319: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206327: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206328: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206344: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206365: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206367: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206382: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206396: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206397: (SocketObject.cpp:517) POLLIN recv return 0
terminate called after throwing an instance of 'DbErrorMsg'

I have to start daemon again.

m.

This isn't how it works. This code consumes a thread in the parser thread FIFO pile whenever it receives a ZC from your node (i.e. it receives an inv_msg_tx packet and follows up with a successful getdata on the hash).

The thread parses the tx, i.e. it performs all tasks that can be parallelized (like deser and hash operations) then sends the processed tx to the main ZC thread for evaluation (since ZC need to be processed in order of appearance). Once a parser thread is done with its task, it re-registers itself with the FIFO pile.

In other words, these threads are constantly recycled, and they do not wait on the tx receiving a confirmation. That's handled by the main scanner code instead.

GETZC_THREADCOUNT defines how many of these threads are created on startup. The current code is built to ignore a ZC if there is no parser thread available to handle it. It is built around a custom container (Stack) that throws if it's empty.

There is another container, BlockingStack, that as the name suggests, will block when it is empty. I could have had you simply swap to that and ignore the thread count, but since this is untested behavior, I'd rather you bump the thread count for now.

Quote

2) Is there any parameter which allow me to spend unconfirmed outputs now (especially I mean unconfirmed change from previous unconfirmed transaction)?

All getUtxo methods take a IGNOREALLZC bool argument (usually the last one, defaulting to true). Pass false there and it should return ZC that are your change.

Quote

3) For 2-3 times since yesterday I had some error like "POOLIN:0". It was sometking about connection to DB. Do You know it?  I will try to provide You a log for the next time when it will appear.

These are false positives 99% of the time, left that verbose there for my own benefit. Regardless, feel free to post the log if you get to catch another of those.

It seems much better after changing  GETZC_THREADCOUNT to 100.

Now I have 3 questions:

1) If I set GETZC_THREADCOUNT for 100 and will have 100 incoming unconfirmed transfers for 10 hours (for example 100 transfers with low fees) does it mean that for 10 hours other ZC transactions will be ignored?

2) Is there any parameter which allow me to spend unconfirmed outputs now (especially I mean unconfirmed change from previous unconfirmed transaction)?

3) For 2-3 times since yesterday I had some error like "POOLIN:0". It was sometking about connection to DB. Do You know it?  I will try to provide You a log for the next time when it will appear.

m. 

Thank You. I will test it and let You know.

m.

I think I've figured it out.

I've wrote the ZC code so that each transaction gets processed in its own thread. To prevent lockups on older machines, it is meant to skip ZCs if there are no processing threads available to parse them (i.e. the machine is getting overwhelmed with ZC).

There are multiple solutions to this problem and I'll think about adding a CLI arg to force deterministic ZC parsing in the future. For now you can modify this value to increase the amount of parser threads:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BDM_supportClasses.h#L28

You will have to rebuild the binary for this change to take effect. These threads are sleeping until there is data to parse, so increasing that figure shouldn't tax your CPU unless there is a lot of ZC traffic (or you are pushing 50 ZC in bulk to your node).

tab = TheBDM.bdv().getLedgerDelegateForWallets().getHistoryPage(0)

for above (is it global ledger for You?) result are exaclty the same as for

tab =self.curWlt.cppWallet.getHistoryPage(0)

I have also called below code. Confirmed transactions looks fine but unconfirmed are absent in result.

tab = TheBDM.bdv().getLedgerDelegateForScrAddr(self.curWlt.uniqueIDB58,Hash160ToScrAddr(a160)).getHistoryPage(0)


My problem is not even to sent ZC inputs but when I get unspent outputs for some address and I wanna prepare transaction from it I am not sure whether maybe they had been already used and transaction which included them is waiting in mempool for confirmation. Of course Inputs are not availabe when transaction get at least one confirmation but sometimes it takes 30 minutes or more time and inputs should not be in "getunspentoutputsforaddres" available during this time.

What exactly do You mean rescan & rebuild database? delete database and start client (or Armorydb) with some parameters? Or maybe I may do this when deamon is running?

m.

What does the global ledger return? If you are missing transactions there too, I'd strongly suggest you try a rebuild & rescan. This looks like a case of DB desync to me so far.


After investigation, this call was deprecated. While the list is being constructed on the DB side, the underlying data isn't passed along over the socket. That call was only used to resolve address comments in an expensive way on the Python side, so I gutted it.

In other parts of the code, Tx details are grabbed this way:

1) Grab the tx hash from the ledger entry

2) Get the tx for that ledger entry and create a PyTx object out of it. Your code will look something like this:


3) The PyTx object will have a list of PyTxIn and PyTxOut objects, the pretty print methods for both classes show you how to extract data out of those:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/Transaction.py#L513
https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/Transaction.py#L577

Bitcoind 0.13.1 (txindex=1)
Armory 0.95.1
Watchingonly wallet (25000 addresses) - wallet generated with 0.93.1


I have tested function provided by You and I have couple of question:

My code:

 tab =self.curWlt.cppWallet.getHistoryPage(0)

      for tx in tab:
          if tx.getValue()>0:
              print tx.getValue()
              print '\t', binary_to_hex(tx.getTxHash(),BIGENDIAN)
              print '\t', time.ctime(int(tx.getTxTime()))
              addr = tx.getScrAddrList()
              for a160 in addr:
                 print '\t',hash160_to_addrStr(a160)


Belowe some LedgerElelementDatas - output from above code:


300000
        47a99e5e1e58b2c89b604bd6d1da634e4815f527eca2a761d631d7f393a47d22
        Fri Dec 30 09:33:20 2016
11477731
        23f65038e91b7bee1b4f6d734ba848708a19d3c4ab17aeb538a7c95ebbde7ada
        Fri Dec 30 09:33:20 2016
26687823
        1b58c8db6e190f78597ff82985b7c2ba2b00e5b0c017e55c086fad1671082158
        Fri Dec 30 09:33:20 2016
2588182
        819b13d1431940e172c21cdce5c793e8d7f44c2da97d530f7725f704e1bc27ac
        Fri Dec 30 09:33:20 2016


Question 1) Why getScrAddrList() return always empty list? How can I get information aboute addresses (source and destination) from each transaction.

Question 2) getHistoryPage(0) returns:
                       1 unconfirmed transaction when I have 3-6 unconfirmed transaction in wallet
                       4 unconfirmed transatcions when I have 50 unconfirmed transactoin in wallet (I am sending 50 transaction from one address to different addresses in wallet or I am sending out 50 transactions from wallet - each from different address to one destination address).

Is it  a bug? Why all unconfirmed transactions are not visible in history? Can You investigate it or verify whether the problem exists.


I see that getHistoryPage calls c++ code so the problem is rather not in python layer.


  

Try to get the address like this instead:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/PyBtcWallet.py#L3217

You can then use all public methods of the ScrAddrObj class:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L82

These are available to Python through SWIG.


Armory was never meant to let you spend ZC (only your own ZC change). This is because Armory was developed as a GUI wallet. armoryd was donated by a user, ATI only maintained it because it got some traction with our more advanced users.

I do intent to change that to allow for CPFP and RBF (basically adding double spend features in the expert GUI) in the upcoming version. Under the hood, the methods will let you fetch ZC in bulk.

If you want to fetch ZC right now, you will have to go through a few hops:

1) Fetch the wallet's tx ledger (or the global ledger). These are the getHistoryPage methods.

You can fetch that stuff straight from wallet object:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L115

Or you can use LedgerDelegates, which let you combine several wallets for the DB to output the sorted tx history ledger. Delegates are fairly easy to use, you can see the definition here:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L55

And an example of how to instantiate them here:

https://github.com/goatpig/BitcoinArmory/blob/master/ArmoryQt.py#L6708

Basically, to get the global ledger delegate (for all wallets), you call TheBDM.bdv().getLedgerDelegateForWallets() on a valid bdv object.

2) For ZC, you only ever want to fetch page 0. That page always exists so this call will never fail if the object you are calling it on is valid. All ZC are always prepended to the top of the first history page. You can tell ZC apart by their height, which is always UINT32_MAX (2^32 -1).

Keep in mind that you can also get the history ledger for a given address this way:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L301

Here's an example in code:

https://github.com/goatpig/BitcoinArmory/blob/master/qtdialogs.py#L3688

3) If you want more data, you can get the whole tx. Ledger entries come with txhashes, you can use the txhash to fetch the whole tx with this method:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/SwigClient.h#L311

--------------------

All these methods I am listing are in SwigClient.h, which gets SWIG'd (ie, it is C++ code made available to Python through the use of SWIG). SWIG only parses public members/methods, so only those are made available. If you want to read some private members, usually there will be a public method to do that (instead of just accessing it at is).

SWIG doesn't modify naming, so as long as you instantiate a SWIG'd class, you can use it as a native Python class.

Let me know if you need anything else. If you have some patience (maybe a lot =O), I'm confident I can get armoryd back to a place where you can build your applications around it.

Fine,so I have an address with some unconfirmed transactions. This code return 0.0 balance  when I run:

      self.curWlt.getAddrDataFromDB()
      self.curWlt.getBalancesAndCountFromDB()
      atype,a160 = addrStr_to_hash160("1Ne9KrmTLuePACNXND29j3exxxxxxxxx")
      print self.curWlt.getAddrBalance(a160,"unconf")

also getFullUTXOList return only inputs with at least 1 confirmation from the wallet.

How to make it to return also unconfirmed tx data?

m.

It's not that they are deprecated, rather that armoryd has not received much attention since 0.93. You should use the wallet methods instead of relying directly on the BDV:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/PyBtcWallet.py#L443

was this  TheBDM.bdv().getUnspentTxoutsForAddr160List(scrAddrList, IGNOREZC) deprecated in 0.95.1 version ? Which should I use instead ?

m.