Armory 0.95 testing phase - page 3. | Bitcointalksearch.org

droark

sr. member

Activity: 525

Merit: 282

Quote from: goatpig on September 06, 2016, 03:28:34 AM

droak: I've managed to reproduce the error code 22 issue with valgrind and the new test. I've split a stress test that doesn't fail on valgrind. Please build the unit tests then run DB1kIterTest with valgrind massif. This shouldn't fail. I'll be looking at error 22 in the mean time.

Thanks. Had to write a patch to get everything to compile.

Anyway, when running valgrind w/ massif, it's slow as hell. I suspect this one will be running for awhile. I'll report back when it's done.

Also, just to confirm, I'll still getting the ArmoryDB crash. Haven't run the other tests yet.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: Carlton Banks on September 02, 2016, 07:52:37 PM

Looking good so far. Previous issues seem to have been dealt with, other than the incomplete list of addresses in Wallet Properties (coin control consistently gets the list of in-use addresses correct, however). Input-fine coin control seems good (maybe using tree layout to expand and collapse the inputs pertaining to a given address would make long lists of inputs more manageable). Satoshi/byte seems to work Ok. Progress bar overhaul works well (functionally and visually). Provokes the same Gtk "object drawn out of bounds" errors we've seen in the past, so if it could be fixed then, presumably a new fix exists in this instance (it's never been more than a concerning looking warning in the logs anyhow).

I was going for the cheapest GUI implementation possible. If someone feels like beefing up the GUI in a PR, by all means be my guest. Otherwise I'll start taking suggestions once the testing builds are out.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

knightdk submitted a PR that makes use of new cookie auth feature from Core 0.12 to setup an RPC socket with the local node in case there is no password setup in bitcoin.conf,

The means Armory can now setup an RPC connection to your node regardless of auto bitcoind management. The bottom line, most evident effect is that it should display your node's calculated minimum acceptable fee per byte when selecting that fee option in the send dialog (as opposed to a plain 0).

This feature will only work if your client has access to a local Core instance. The feature is disabled for remote clients. This is because this feature was done through the path of least resistance, i.e. leveraging the existing RPC code in the Python client.

I will leverage the full benefits of this change as well as make it available to remote clients once I migrate this code to the C++ DB. For now, this is an added bonus to "default" users. On a side note, this should also work on OSX (once I get the db rolling on Macs that is)

Please test this feature thoroughly with both self managed and auto bitcoind. I didn't have time to test it directly yet so I'm hoping you guys can list the issues before hand.

-----

droak: I've managed to reproduce the error code 22 issue with valgrind and the new test. I've split a stress test that doesn't fail on valgrind. Please build the unit tests then run DB1kIterTest with valgrind massif. This shouldn't fail. I'll be looking at error 22 in the mean time.

Carlton Banks

legendary

Activity: 3430

Merit: 3074

Quote from: goatpig on September 04, 2016, 12:30:47 AM

Quote

Provokes the same Gtk "object drawn out of bounds" errors we've seen in the past, so if it could be fixed then, presumably a new fix exists in this instance (it's never been more than a concerning looking warning in the logs anyhow).

I have no idea what the underlying issue actually is, besides that it triggers with older version of PyQt4. I can see it on my Debian7 VM, I don't think I can on Debian 8. What version are you using Qt are you using?

Version 4.8.6, Debian 8.5. Pretty sure that's the most up to date version in the Debian 8 standard repo.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: josephbisch on September 03, 2016, 05:45:54 PM

A solution is to modify self.dbExecutable in SDM.py to be prepended by the following path.

Code:

path = os.path.dirname(os.path.abspath(__file__))
self.dbExecutable = os.path.join(path, 'ArmoryDB')
if OS_WINDOWS:
    # add .exe to self.dbExecutable

Done. You should just PR these kind of quick fixes in the future.

Quote

Provokes the same Gtk "object drawn out of bounds" errors we've seen in the past, so if it could be fixed then, presumably a new fix exists in this instance (it's never been more than a concerning looking warning in the logs anyhow).

I have no idea what the underlying issue actually is, besides that it triggers with older version of PyQt4. I can see it on my Debian7 VM, I don't think I can on Debian 8. What version are you using Qt are you using?

josephbisch

member

Activity: 75

Merit: 10

Quote from: achow101 on September 03, 2016, 05:28:14 PM

Quote from: josephbisch on September 03, 2016, 04:48:34 PM

Quote from: achow101 on September 03, 2016, 01:14:27 AM

This is strange. If I use the shortcut I made to launch Armory, it doesn't work.

This is the error from the log:

Code:

2016-09-03 01:02 (INFO) -- ArmoryUtils.py:639 - Executing popen: ['./ArmoryDB', '--db-type="DB_FULL"', '--spawnId="5uWPvi3P2wvYyo2xNdpeWKU4spxWvHuDRuCJ9gfxevYd"', '--satoshi-datadir="/media/andy/Data/Programs/Bitcoin/data/blocks"', '--dbdir="/media/andy/Data/Data/bitcoin/ArmoryData/databases"']
2016-09-03 01:02 (ERROR) -- Traceback (most recent call last):
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6631, in method_signal
    method()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6668, in completeBlockchainProcessingInitialization
    gotDB = self.startArmoryDBIfNecessary()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 2198, in startArmoryDBIfNecessary
    spawnId = TheSDM.spawnDB(TheBDM.armoryDBDir)
  File "/home/andy/bitcoin/BitcoinArmory/SDM.py", line 490, in spawnDB
    launchProcess(pargs, **kargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/ArmoryUtils.py", line 642, in launchProcess
    return Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

However if I run the same command from the terminal, I have no problems whatsoever.

I am running Ubuntu 16.04. This is probably an issue with my system, but I think it is something to note.

Armory is trying to open ArmoryDB as a relative path. That only works if you are have used cd to enter the Armory source tree first. The solution is for Armory to use absolute paths, so people don't need to be in the same directory as Armory to run it.

So this update breaks shortcuts right now. That is a bit of an issue.

It breaks shortcuts or any use case where someone is trying to run Armory from somewhere other than the Armory source tree. So if they are above a level and try to run "python BitcoinArmory/ArmoryQt.py" that should fail too.

A solution is to modify self.dbExecutable in SDM.py to be prepended by the following path.

Code:

path = os.path.dirname(os.path.abspath(__file__))
self.dbExecutable = os.path.join(path, 'ArmoryDB')
if OS_WINDOWS:
    # add .exe to self.dbExecutable

achow101

staff

Activity: 3374

Merit: 6530

Just writing some code

Quote from: josephbisch on September 03, 2016, 04:48:34 PM

Quote from: achow101 on September 03, 2016, 01:14:27 AM

This is strange. If I use the shortcut I made to launch Armory, it doesn't work.

This is the error from the log:

Code:

2016-09-03 01:02 (INFO) -- ArmoryUtils.py:639 - Executing popen: ['./ArmoryDB', '--db-type="DB_FULL"', '--spawnId="5uWPvi3P2wvYyo2xNdpeWKU4spxWvHuDRuCJ9gfxevYd"', '--satoshi-datadir="/media/andy/Data/Programs/Bitcoin/data/blocks"', '--dbdir="/media/andy/Data/Data/bitcoin/ArmoryData/databases"']
2016-09-03 01:02 (ERROR) -- Traceback (most recent call last):
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6631, in method_signal
    method()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6668, in completeBlockchainProcessingInitialization
    gotDB = self.startArmoryDBIfNecessary()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 2198, in startArmoryDBIfNecessary
    spawnId = TheSDM.spawnDB(TheBDM.armoryDBDir)
  File "/home/andy/bitcoin/BitcoinArmory/SDM.py", line 490, in spawnDB
    launchProcess(pargs, **kargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/ArmoryUtils.py", line 642, in launchProcess
    return Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

However if I run the same command from the terminal, I have no problems whatsoever.

I am running Ubuntu 16.04. This is probably an issue with my system, but I think it is something to note.

Armory is trying to open ArmoryDB as a relative path. That only works if you are have used cd to enter the Armory source tree first. The solution is for Armory to use absolute paths, so people don't need to be in the same directory as Armory to run it.

So this update breaks shortcuts right now. That is a bit of an issue.

josephbisch

member

Activity: 75

Merit: 10

Quote from: achow101 on September 03, 2016, 01:14:27 AM

This is strange. If I use the shortcut I made to launch Armory, it doesn't work.

This is the error from the log:

Code:

2016-09-03 01:02 (INFO) -- ArmoryUtils.py:639 - Executing popen: ['./ArmoryDB', '--db-type="DB_FULL"', '--spawnId="5uWPvi3P2wvYyo2xNdpeWKU4spxWvHuDRuCJ9gfxevYd"', '--satoshi-datadir="/media/andy/Data/Programs/Bitcoin/data/blocks"', '--dbdir="/media/andy/Data/Data/bitcoin/ArmoryData/databases"']
2016-09-03 01:02 (ERROR) -- Traceback (most recent call last):
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6631, in method_signal
    method()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6668, in completeBlockchainProcessingInitialization
    gotDB = self.startArmoryDBIfNecessary()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 2198, in startArmoryDBIfNecessary
    spawnId = TheSDM.spawnDB(TheBDM.armoryDBDir)
  File "/home/andy/bitcoin/BitcoinArmory/SDM.py", line 490, in spawnDB
    launchProcess(pargs, **kargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/ArmoryUtils.py", line 642, in launchProcess
    return Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

However if I run the same command from the terminal, I have no problems whatsoever.

I am running Ubuntu 16.04. This is probably an issue with my system, but I think it is something to note.

Armory is trying to open ArmoryDB as a relative path. That only works if you are have used cd to enter the Armory source tree first. The solution is for Armory to use absolute paths, so people don't need to be in the same directory as Armory to run it.

droark

sr. member

Activity: 525

Merit: 282

ArmoryDB crash still happening on OSX, even with the new code. Will double check to see if it's in the same location but I'd imagine this is the case.

achow101

staff

Activity: 3374

Merit: 6530

Just writing some code

This is strange. If I use the shortcut I made to launch Armory, it doesn't work.

This is the error from the log:

Code:

2016-09-03 01:02 (INFO) -- ArmoryUtils.py:639 - Executing popen: ['./ArmoryDB', '--db-type="DB_FULL"', '--spawnId="5uWPvi3P2wvYyo2xNdpeWKU4spxWvHuDRuCJ9gfxevYd"', '--satoshi-datadir="/media/andy/Data/Programs/Bitcoin/data/blocks"', '--dbdir="/media/andy/Data/Data/bitcoin/ArmoryData/databases"']
2016-09-03 01:02 (ERROR) -- Traceback (most recent call last):
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6631, in method_signal
    method()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 6668, in completeBlockchainProcessingInitialization
    gotDB = self.startArmoryDBIfNecessary()
  File "/home/andy/bitcoin/BitcoinArmory/ArmoryQt.py", line 2198, in startArmoryDBIfNecessary
    spawnId = TheSDM.spawnDB(TheBDM.armoryDBDir)
  File "/home/andy/bitcoin/BitcoinArmory/SDM.py", line 490, in spawnDB
    launchProcess(pargs, **kargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/ArmoryUtils.py", line 642, in launchProcess
    return Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

However if I run the same command from the terminal, I have no problems whatsoever.

I am running Ubuntu 16.04. This is probably an issue with my system, but I think it is something to note.

Carlton Banks

legendary

Activity: 3430

Merit: 3074

Looking good so far. Previous issues seem to have been dealt with, other than the incomplete list of addresses in Wallet Properties (coin control consistently gets the list of in-use addresses correct, however). Input-fine coin control seems good (maybe using tree layout to expand and collapse the inputs pertaining to a given address would make long lists of inputs more manageable). Satoshi/byte seems to work Ok. Progress bar overhaul works well (functionally and visually). Provokes the same Gtk "object drawn out of bounds" errors we've seen in the past, so if it could be fixed then, presumably a new fix exists in this instance (it's never been more than a concerning looking warning in the logs anyhow).

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Added ZC notifications. It will give you an specific error message on transactions that failed to broadcast.
Fixed some progress bars, added one for the tx resolution. It's still a bit whacky, mainly the ETA, but the progress reporting is decent.
Added fee per byte option to the send dialog.

Also added a bunch of tentative fixes for the other issues listed here (that I sadly can't reproduce T_T). Please test away.

droark

sr. member

Activity: 525

Merit: 282

Quote from: goatpig on August 31, 2016, 01:09:50 AM

Quote from: droark on August 30, 2016, 09:20:11 PM

Hmmm. I'm seeing something different. After one successful run on Linux (Ubuntu 16.04), I kept getting the "Error Code: 22" error on Load3Blocks_ZC_Plus3_TestLedgers and had to disable it. After that, I did see this once.

Clearly there's something off with the comparison function, or it is fed an empty pointer eventually. Is it happening at random in any test or always the same spot?

The error code? It happens consistently on Linux when valgrind is used.

Code:

[----------] 18 tests from BlockUtilsBare
[ RUN      ] BlockUtilsBare.Load5Blocks
[       OK ] BlockUtilsBare.Load5Blocks (1398 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_DamagedBlkFile
[       OK ] BlockUtilsBare.Load5Blocks_DamagedBlkFile (7102 ms)
[ RUN      ] BlockUtilsBare.Load4Blocks_Plus2
[       OK ] BlockUtilsBare.Load4Blocks_Plus2 (1251 ms)
[ RUN      ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2
[       OK ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2 (10502 ms)
[ RUN      ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
Failed to create map of file. Error Code: 22 (Invalid argument)

On its own, the tests pass. Sorry. Didn't make that clear the first time.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: droark on August 30, 2016, 09:20:11 PM

Hmmm. I'm seeing something different. After one successful run on Linux (Ubuntu 16.04), I kept getting the "Error Code: 22" error on Load3Blocks_ZC_Plus3_TestLedgers and had to disable it. After that, I did see this once.

Clearly there's something off with the comparison function, or it is fed an empty pointer eventually. Is it happening at random in any test or always the same spot?

Quote

Also, I forgot to run ContainerTests. No crashes, at least not yet. First is the normal version, second is the Valgrind version, which is still running after almost an hour. (I'll update it if it finishes anytime soon.) This is on Linux. I'll update if OSX has significant differences.

All those are benign or false positives, nothing alarming here at least

Quote

EDIT: Interestingly, on OSX, ContainerTests passes every time, yet when running under valgrind, BlockingStackTest_Concurrent seems to get stuck, just like on Linux.

Yep, expected, forgot to rework the BlockingStack clean up method, which will end up hanging on heavy loads. It's only used when shutting down the DB, so I'm not in a hurry to fix it yet.

droark

sr. member

Activity: 525

Merit: 282

Quote from: goatpig on August 30, 2016, 02:56:45 PM

Quote

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785==
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785== Non-existent physical address at address 0x1027D6000
==6785== at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785== by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785== by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785== by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function) (DatabaseBuilder.cpp:381)
==6785== by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr) (DatabaseBuilder.cpp:327)
==6785== by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr, bool) const (DatabaseBuilder.cpp:217)
==6785== by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785== by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785== by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785== by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785== by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785== by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785==
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Now that's something I can't reproduce on my end. My valgrind tests all pass 100%. Can you try reproducing this on a linux setup too? Definitely worth looking into.

Hmmm. I'm seeing something different. After one successful run on Linux (Ubuntu 16.04), I kept getting the "Error Code: 22" error on Load3Blocks_ZC_Plus3_TestLedgers and had to disable it. After that, I did see this once.

Code:

[ RUN ] BlockUtilsBare.Load3Blocks_ZCchain
==20207==
==20207== Process terminating with default action of signal 7 (SIGBUS)
==20207== Non-existent physical address at address 0x403B000
==20207== at 0x5A5D851: __memcmp_sse4_1 (memcmp-sse4.S:812)
==20207== by 0x88E929: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==20207== by 0x88E982: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==20207== by 0x9BCB7D: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::function) (DatabaseBuilder.cpp:381)
==20207== by 0x9BC5F1: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::shared_ptr) (DatabaseBuilder.cpp:328)
==20207== by 0x9BB9FB: DatabaseBuilder::updateBlocksInDB(std::function const&, bool, bool)::{lambda(unsigned short, unsigned long, std::shared_ptr, bool)#1}::operator()(unsigned short, unsigned long, std::shared_ptr, bool) const (DatabaseBuilder.cpp:217)
==20207== by 0x9BBEA7: DatabaseBuilder::updateBlocksInDB(std::function const&, bool, bool) (DatabaseBuilder.cpp:258)
==20207== by 0x9BD2CD: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==20207== by 0x8B97C8: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==20207== by 0x92A2F5: BlockDataManagerThread::run()::{lambda()#2}::operator()() const (BDM_mainthread.cpp:178)
==20207== by 0x92AAEF: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==20207== by 0x92AFFB: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
==20207==
Killed

Unlike the OSX crash, which is consistent, this one only happens occasionally.

Also, I forgot to run ContainerTests. No crashes, at least not yet. First is the normal version, second is the Valgrind version, which is still running after almost an hour. (I'll update it if it finishes anytime soon.) This is on Linux. I'll update if OSX has significant differences.

Code:

[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 8 tests from ContainerTests
[ RUN ] ContainerTests.TransactionalMap
[ OK ] ContainerTests.TransactionalMap (216 ms)
[ RUN ] ContainerTests.PileTest_Sequential
[ OK ] ContainerTests.PileTest_Sequential (508 ms)
[ RUN ] ContainerTests.PileTest_Concurrent
[ OK ] ContainerTests.PileTest_Concurrent (934 ms)
[ RUN ] ContainerTests.StackTest_Sequential
[ OK ] ContainerTests.StackTest_Sequential (718 ms)
[ RUN ] ContainerTests.StackTest_Concurrent
[ OK ] ContainerTests.StackTest_Concurrent (590 ms)
[ RUN ] ContainerTests.BlockingStackTest_Sequential
[ OK ] ContainerTests.BlockingStackTest_Sequential (689 ms)
[ RUN ] ContainerTests.BlockingStackTest_Concurrent
ContainerTests.cpp:576: Failure
Expected: (theStack.count()) != (0), actual: 0 vs 0
[ FAILED ] ContainerTests.BlockingStackTest_Concurrent (1099 ms)
[ RUN ] ContainerTests.TimedStackTest_Concurrent
[ OK ] ContainerTests.TimedStackTest_Concurrent (2850 ms)
[----------] 8 tests from ContainerTests (7604 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 1 test case ran. (7604 ms total)
[ PASSED ] 7 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ContainerTests.BlockingStackTest_Concurrent

1 FAILED TEST

Code:

==23036== Massif, a heap profiler
==23036== Copyright (C) 2003-2015, and GNU GPL'd, by Nicholas Nethercote
==23036== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==23036== Command: ./ContainerTests
==23036==
[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 8 tests from ContainerTests
[ RUN ] ContainerTests.TransactionalMap
[ OK ] ContainerTests.TransactionalMap (9033 ms)
[ RUN ] ContainerTests.PileTest_Sequential
[ OK ] ContainerTests.PileTest_Sequential (6640 ms)
[ RUN ] ContainerTests.PileTest_Concurrent
ContainerTests.cpp:249: Failure
Value of: poptally
Actual: 540229271904773
Expected: pushtally
Which is: 858672733906290
ContainerTests.cpp:250: Failure
Value of: 0
Expected: thePile.count()
Which is: 296875
[ FAILED ] ContainerTests.PileTest_Concurrent (71804 ms)
[ RUN ] ContainerTests.StackTest_Sequential
[ OK ] ContainerTests.StackTest_Sequential (3947 ms)
[ RUN ] ContainerTests.StackTest_Concurrent
ContainerTests.cpp:400: Failure
Value of: poptally
Actual: 809112423041638
Expected: pushtally
Which is: 859016864469035
ContainerTests.cpp:401: Failure
Value of: 0
Expected: theStack.count()
Which is: 46511
[ FAILED ] ContainerTests.StackTest_Concurrent (106924 ms)
[ RUN ] ContainerTests.BlockingStackTest_Sequential
[ OK ] ContainerTests.BlockingStackTest_Sequential (11527 ms)
[ RUN ] ContainerTests.BlockingStackTest_Concurrent

EDIT: Interestingly, on OSX, ContainerTests passes every time, yet when running under valgrind, BlockingStackTest_Concurrent seems to get stuck, just like on Linux.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: droark on August 30, 2016, 12:33:47 PM

Code:

[ RUN ] BlockDir.HeadersFirst
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
[ OK ] BlockDir.HeadersFirst (21467 ms)

Never seen these. Sounds like OSX specific, not particularly concerned if they don't result in a segfault.

Quote

Code:

Failed to create map of file. Error Code: 22

Can happen sometimes when the OS doesn't create file handles fast enough. Each test completely wipes the db. On some rare occasions (like a system on full load), lmdb will create the db file and try to access it after the call returns but before the OS actually flushes the file to the system (which is a very rare and unexpected window to hit). Not a concern per se, file creation only takes place with empty db folders.

Quote

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785==
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785== Non-existent physical address at address 0x1027D6000
==6785== at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785== by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785== by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785== by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function) (DatabaseBuilder.cpp:381)
==6785== by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr) (DatabaseBuilder.cpp:327)
==6785== by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr, bool) const (DatabaseBuilder.cpp:217)
==6785== by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785== by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785== by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785== by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785== by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785== by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785==
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Now that's something I can't reproduce on my end. My valgrind tests all pass 100%. Can you try reproducing this on a linux setup too? Definitely worth looking into.

droark

sr. member

Activity: 525

Merit: 282

Quote from: goatpig on August 30, 2016, 12:36:05 AM

Quote from: droark on August 30, 2016, 12:25:18 AM

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

Try the unit tests in the gtest folder. Both test binaries should pass valgrind --tool=massif

Hmmm. I'm seeing a couple of odd things after running Valgrind a few times.

Code:

[ RUN ] BlockDir.HeadersFirst
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
[ OK ] BlockDir.HeadersFirst (21467 ms)

These messages pop up consistently at this point. These messages seem to be considered spurious by a lot of valgrind users. They apparently pop up quite a bit when running on OSX, which valgrind is slow to support. (In fact, AFAIK, they still haven't implemented full OSX 10.11 support.) I believe they sometimes come up when valgrind doesn't fully understand system calls. It's hard to say but it might be worth looking at this code anyway.

Code:

Failed to create map of file. Error Code: 22

Saw this once last night but can't reproduce it now for unknown reasons. I think it happened during the Load3Blocks_ZCchain test but I can't remember. Anyway, I believe 22 = EINVAL on OSX. I've written a small patch that uses strerror() in the future. The patch might be why this only came up once. (See below.)

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785==
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785== Non-existent physical address at address 0x1027D6000
==6785== at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785== by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785== by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785== by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function) (DatabaseBuilder.cpp:381)
==6785== by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr) (DatabaseBuilder.cpp:327)
==6785== by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr, bool) const (DatabaseBuilder.cpp:217)
==6785== by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785== by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785== by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785== by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785== by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785== by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785==
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Code:

[----------] 18 tests from BlockUtilsBare
[ RUN ] BlockUtilsBare.Load5Blocks
[ OK ] BlockUtilsBare.Load5Blocks (30152 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_DamagedBlkFile
[ OK ] BlockUtilsBare.Load5Blocks_DamagedBlkFile (6538 ms)
[ RUN ] BlockUtilsBare.Load4Blocks_Plus2
[ OK ] BlockUtilsBare.Load4Blocks_Plus2 (44924 ms)
[ RUN ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2
[ OK ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2 (82404 ms)
[ RUN ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
[ OK ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers (89746 ms)
[ RUN ] BlockUtilsBare.Load3Blocks_ZCchain
[ OK ] BlockUtilsBare.Load3Blocks_ZCchain (11110 ms)
[ RUN ] BlockUtilsBare.Load3Blocks_RBF
[ OK ] BlockUtilsBare.Load3Blocks_RBF (497 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_FullReorg
[ OK ] BlockUtilsBare.Load5Blocks_FullReorg (3000 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_DoubleReorg
[ OK ] BlockUtilsBare.Load5Blocks_DoubleReorg (4670 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_ReloadBDM_Reorg
[ OK ] BlockUtilsBare.Load5Blocks_ReloadBDM_Reorg (19694 ms)
[ RUN ] BlockUtilsBare.CorruptedBlock
[ OK ] BlockUtilsBare.CorruptedBlock (8482 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_RescanOps
[ OK ] BlockUtilsBare.Load5Blocks_RescanOps (136481 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_RescanEmptyDB
[ OK ] BlockUtilsBare.Load5Blocks_RescanEmptyDB (22092 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_RebuildEmptyDB
[ OK ] BlockUtilsBare.Load5Blocks_RebuildEmptyDB (68415 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_SideScan
[ OK ] BlockUtilsBare.Load5Blocks_SideScan (39226 ms)
[ RUN ] BlockUtilsBare.Load5Blocks_GetUtxos
[ OK ] BlockUtilsBare.Load5Blocks_GetUtxos (55087 ms)
[ RUN ] BlockUtilsBare.Load4Blocks_ZC_GetUtxos
[ OK ] BlockUtilsBare.Load4Blocks_ZC_GetUtxos (43231 ms)
[ RUN ] BlockUtilsBare.LedgerSeDer
[ OK ] BlockUtilsBare.LedgerSeDer (20928 ms)
[----------] 18 tests from BlockUtilsBare (686681 ms total)

When the tests do succeed under valgrind, the BlockUtilsBare tests take forever. On their own, they don't take too long.

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: droark on August 30, 2016, 12:25:18 AM

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

Try the unit tests in the gtest folder. Both test binaries should pass valgrind --tool=massif

droark

sr. member

Activity: 525

Merit: 282

Quote from: goatpig on August 30, 2016, 12:22:22 AM

Quote from: goatpig on August 29, 2016, 03:48:31 AM

Quote from: droark on August 28, 2016, 11:13:44 PM

Quote from: goatpig on August 28, 2016, 02:04:17 AM

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

Can you:

1) upload the incriminated block file

2) try to sync against another copy of the testnet

Thanks for the file. Can you build a DB from scratch using --db-type=DB_BARE?

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

goatpig

legendary

Activity: 3640

Merit: 1345

Armory Developer

Quote from: goatpig on August 29, 2016, 03:48:31 AM

Quote from: droark on August 28, 2016, 11:13:44 PM

Quote from: goatpig on August 28, 2016, 02:04:17 AM

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

Can you:

1) upload the incriminated block file

2) try to sync against another copy of the testnet

Thanks for the file. Can you build a DB from scratch using --db-type=DB_BARE?

Topic: Armory 0.95 testing phase - page 3. (Read 8294 times)