Pages:
Author

Topic: RAM-Reduction & Backup Center Testing (version 0.89.99.16) - page 19. (Read 41296 times)

legendary
Activity: 1795
Merit: 1208
This is not OK.
It appeared to be working... started up, took forever to scan then showed my balance... but then I hit exit and my machine immediately BSOD'ed :/

Wtf?  Certainly haven't heard that one yet...

System specs?


Sorry, was going to, but forgot... Win 8, 16Gb Ram, SSD + HDD, i7 Quad core 2.4GHz.

I'll try again tomorrow see what happens.
legendary
Activity: 1193
Merit: 1003
9.9.2012: I predict that single digits... <- FAIL
P.S -- Has no one played with the new backup center?  Come on!  It's so badass!

It's awesome!
cp1
hero member
Activity: 616
Merit: 500
Stop using branwallets
I look around in parsePrivateKeyData and figured out what I needed to add to my base58 conversion, thanks.

So what was it?  I just did the same thing as you did at first and got a wacky answer.  I haven't looked at that in forever, so I'm curious what I forgot.

I had to take the substring of the hex instead of the substring of the private key and then it worked.

line = "5KJvsngHeMpm884wtkJNzQGaCErckhHJBGFsvd3VyK5qMZXj3hS"
binEntry = base58_to_binary(line)
print prettyHex(binary_to_hex(binEntry[1:33]))
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
It appeared to be working... started up, took forever to scan then showed my balance... but then I hit exit and my machine immediately BSOD'ed :/

Wtf?  Certainly haven't heard that one yet...

System specs?
legendary
Activity: 1795
Merit: 1208
This is not OK.
It appeared to be working... started up, took forever to scan then showed my balance... but then I hit exit and my machine immediately BSOD'ed :/
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
I look around in parsePrivateKeyData and figured out what I needed to add to my base58 conversion, thanks.

So what was it?  I just did the same thing as you did at first and got a wacky answer.  I haven't looked at that in forever, so I'm curious what I forgot.
cp1
hero member
Activity: 616
Merit: 500
Stop using branwallets
I look around in parsePrivateKeyData and figured out what I needed to add to my base58 conversion, thanks.
sr. member
Activity: 287
Merit: 250
P.S -- Has no one played with the new backup center?  Come on!  It's so badass!
Great to see a M of N paper backup implemented so smoothly.  I've really been looking forward to that as a GUI feature.
Been poking around 0.89.99.3 in a Ubuntu 12.04 VM.  No issues so far.

legendary
Activity: 1428
Merit: 1093
Core Armory Developer
OK, so tracing the bug is difficult if one cannot single-step through the function unless it is done in less than 10 sec Smiley

It seems that it sometimes cause corruption of the database, so that a rescan is necessary on the next restart.  Othewise transactions may be missing.


Modify the variable MT_WAIT_TIMEOUT_SEC in armoryengine.py.  Set it to UINT32_MAX or something.   That will help with the debugging.

Also, be aware that I explicitly try to avoid corrupting the wallet history by only saving it if there's a clean shutdown.  If you kill Armory, or you close it while it's scanning, offline, or dirty, it will not save any history and will require rescanning. 
hero member
Activity: 547
Merit: 500
Decor in numeris
OK, so tracing the bug is difficult if one cannot single-step through the function unless it is done in less than 10 sec Smiley

It seems that it sometimes cause corruption of the database, so that a rescan is necessary on the next restart.  Othewise transactions may be missing.
full member
Activity: 226
Merit: 100
P.S -- Has no one played with the new backup center?  Come on!  It's so badass!



One back from the conference I'll spend some time on it. Have some other things to to do also but I am also excited about the new features.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
P.S -- Has no one played with the new backup center?  Come on!  It's so badass!

legendary
Activity: 1428
Merit: 1093
Core Armory Developer
sr. member
Activity: 366
Merit: 258
Just wanted to report that this update is definitely Usable on our windows7 64-bit machine.  Typically we use ubuntu, but I've wanted to try out some watch only copies on windows.  While it does seem to hang regularly when closing the program, its been stable enough to test functionality and everything we typically use works great.  

Current RAM usage on this version is hovering around 315 Mb, which is at least an order of magnitude improvement.  I'll continue troubleshooting to provide more helpful feedback but for now I just wanted to say THANKS for this much needed fix and good luck in your troubleshooting.
hero member
Activity: 547
Merit: 500
Decor in numeris
I did a bit more debugging, putting a breakpoint in the crashing function and stepping line by line.  It looks like the call to KEY_NOT_IN_MAP triggers an exception in some python code.  It is a testnet wallet, so I could in principle send it to you.

Code:
Breakpoint 1, BlockDataManager_LevelDB::shutdownSaveScrAddrHistories (this=0x104ba23e0) at BlockUtils.cpp:3988
3988    LOGINFO << "Saving wallet history for next load";
(gdb) n
Current language:  auto; currently c++
-INFO  - 1380282523: (BlockUtils.cpp:3988) Saving wallet history for next load
3990    iface_->startBatch(BLKDATA);
(gdb) n
3992    uint32_t i=0;
(gdb) n
3993    set::iterator wltIter;
(gdb)
3994    for(wltIter  = registeredWallets_.begin();
(gdb)
3995        wltIter != registeredWallets_.end();
(gdb)
3998       for(uint32_t a=0; a<(*wltIter)->getNumScrAddr(); a++)
(gdb)
4000          ScrAddrObj & scrAddr = (*wltIter)->getScrAddrObjByIndex(a);
(gdb)
4001          BinaryData uniqKey = scrAddr.getScrAddr();
(gdb)
4003          if(KEY_NOT_IN_MAP(uniqKey, registeredScrAddrMap_))
(gdb)
(ERROR) armoryengine.py:12189 - Waiting for BDM output that didn't come after 20s.
(ERROR) armoryengine.py:12190 - BDM state is currently: Uninitialized
(ERROR) armoryengine.py:12191 - Called from: armoryengine.py:12314 (83837112)
(ERROR) armoryengine.py:12192 - BDM currently doing: Shutdown (83837112)
(ERROR) armoryengine.py:12193 - Direct traceback
  File "ArmoryQt.py", line 4981, in
    os._exit(QAPP.exec_())
  File "ArmoryQt.py", line 4835, in closeForReal
    TheBDM.execCleanShutdown(wait=True)
  File "/Users/schiotz/src/BitcoinArmory/armoryengine.py", line 12314, in execCleanShutdown
    return self.waitForOutputIfNecessary(expectOutput, rndID)
  File "/Users/schiotz/src/BitcoinArmory/armoryengine.py", line 12194, in waitForOutputIfNecessary
    traceback.print_stack()
(ERROR) armoryengine.py:12195 - Traceback:
Traceback (most recent call last):
  File "/Users/schiotz/src/BitcoinArmory/armoryengine.py", line 12185, in waitForOutputIfNecessary
    return self.outputQueue.get(True, self.mtWaitSec)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/Queue.py", line 176, in get
    raise Empty
Empty
4009          RegisteredScrAddr & rsa = registeredScrAddrMap_[uniqKey];
hero member
Activity: 547
Merit: 500
Decor in numeris
Hi etotheipi,

I have a strange crash when Armory quits on Mac OS.  LevelDB gives a segfault, the stack trace from gdb is below.

I am by the way almost done making a Mac OS X App.  But since this is not a race for a bounty, I prefer to sit on it for a few days to fix the worst issues  Wink  (No real testing yet, will it run on another machine?  It probably crashes on old hardware if built on new hardware.  The icon and other stuff is missing.  The app is beyond huge, 250M !!).  I think I know how to address all of them.

Anyway, here is the crash data.  It occurs when I quit the app, and it does not depend on whether I build a real app, or just compile Armory and run it from the command line.

Code:
-INFO  - 1380265090: (BlockUtils.cpp:3988) Saving wallet history for next load

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
[Switching to process 899 thread 0x1903]
0x0000000103e083c0 in InterfaceToLDB::commitBatch ()
(gdb) where
#0  0x0000000103e083c0 in InterfaceToLDB::commitBatch ()
#1  0x0000000103e5d064 in BlockDataManager_LevelDB::shutdownSaveScrAddrHistories ()
#2  0x00000001040250fe in _wrap_BlockDataManager_LevelDB_shutdownSaveScrAddrHistories ()
#3  0x000000010001d5a9 in PyEval_EvalFrameEx ()
#4  0x0000000100021869 in _PyEval_SliceIndex ()
#5  0x000000010001d63a in PyEval_EvalFrameEx ()
#6  0x0000000100021869 in _PyEval_SliceIndex ()
#7  0x000000010001d63a in PyEval_EvalFrameEx ()
#8  0x0000000100021869 in _PyEval_SliceIndex ()
#9  0x000000010001d63a in PyEval_EvalFrameEx ()
#10 0x0000000100021869 in _PyEval_SliceIndex ()
#11 0x000000010001d63a in PyEval_EvalFrameEx ()
#12 0x000000010001b147 in PyEval_EvalCodeEx ()
#13 0x0000000100054d7a in PyFunction_SetClosure ()
#14 0x00000001000136c6 in PyObject_Call ()
#15 0x00000001000309bf in PyMethod_New ()
#16 0x00000001000136c6 in PyObject_Call ()
#17 0x0000000100021018 in PyEval_CallObjectWithKeywords ()
#18 0x000000010007d7f6 in initthread ()
#19 0x00007fff96982772 in _pthread_start ()
#20 0x00007fff9696f1a1 in thread_start ()
(gdb) list
1 // algparam.cpp - written and placed in the public domain by Wei Dai
2
3 #include "pch.h"
4
5 #ifndef CRYPTOPP_IMPORTS
6
7 #include "algparam.h"
8
9 NAMESPACE_BEGIN(CryptoPP)
10
(gdb) up
#1  0x0000000103e5d064 in BlockDataManager_LevelDB::shutdownSaveScrAddrHistories () at basic_string.h:279
279       _M_data() const
Current language:  auto; currently c++
(gdb) list
274     private:
275       // Data Members (private):
276       mutable _Alloc_hider _M_dataplus;
277
278       _CharT*
279       _M_data() const
280       { return  _M_dataplus._M_p; }
281
282       _CharT*
283       _M_data(_CharT* __p)
(gdb) up
#2  0x00000001040250fe in _wrap_BlockDataManager_LevelDB_shutdownSaveScrAddrHistories ()
(gdb) list
284       { return (_M_dataplus._M_p = __p); }
285
286       _Rep*
287       _M_rep() const
288       { return &((reinterpret_cast<_Rep*> (_M_data()))[-1]); }
289
290       // For the internal use we have functions similar to `begin'/`end'
291       // but they do not call _M_leak.
292       iterator
293       _M_ibegin() const
(gdb) up
#3  0x000000010001d5a9 in PyEval_EvalFrameEx ()
(gdb) list
294       { return iterator(_M_data()); }
295
296       iterator
297       _M_iend() const
298       { return iterator(_M_data() + this->size()); }
299
300       void
301       _M_leak()    // for use in begin() & non-const op[]
302       {
303 if (!_M_rep()->_M_is_leaked())
cp1
hero member
Activity: 616
Merit: 500
Stop using branwallets
Success!  Thanks

parsePrivateKeyData worked great.  I still can't figure out base58_to_binary though (not a big deal).  I did try substringing at first using [1:-4], but it never worked for me, I always get too many bytes (and also the wrong answer).

For example:

line = "5JdeQ39z8NUkNVvB37tt74Cu2WSNVj7qb9PdY651UoQnqyCm937"
hardcodedPrivKey = base58_to_binary(line[1:-4])
print prettyHex(binary_to_hex(hardcodedPrivKey))

0x0000:  0d728779 7539eb09 87a0c18f 9e3fac6c 68dbe7f9 1eb106e5 0793d7b9 6e03697f
0x0020:  f673


But parsePrivateKeyData works perfect:

line = "5JdeQ39z8NUkNVvB37tt74Cu2WSNVj7qb9PdY651UoQnqyCm937"
pk = (parsePrivateKeyData(line)[0])
print prettyHex(binary_to_hex(pk))

0x0000:  6b88c087 247aa2f0 7ee1c595 6b8e1a9f 4c7f892a 70e324f1 bb3d161e 05ca107b

Ha, and funny enough it just loaded the transactions for this key in armoryqt and there was a small transaction in May.  People use the worst brainwallets.
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
Thanks, that worked great for a 32-byte hex key.  Is it possible to import from WIF?  I've tried:

wlt.importExternalAddressData(privKey=SecureBinaryData(base58_to_binary("5KaPWnaXTLUdv5WdbwbiXbjDqUFXEUT1aAgSGrYHtzuGD5mooWv")))

But I get: assert(plainPrivKey.getSize()==32)

Do I have to convert from WIF to 32-byte first?  Is there a function in armory?  I was snooping in bulkImportAddresses() and found snippets to deal with base-58 private key, but couldn't turn it into anything working.


Edit:  It must be in there somewhere because I think you can import WIF from armoryqt.  Now I'm on a hunt!

The "parsePrivateKeyData()" method autodetects basically every private key format, including Casascius' mini-private-keys.  You could manually read the keys using "base58_to_binary()", but the following will work just as well to convert the data into 32-byte hex that can imported per my previous example:

Code:
>>> encodePrivKeyBase58('\xaa'*32)
'5K7T2qR7K5MUMDmkJvCEPbUeALuPyc6LFEnBiwWLuKCEVdBp8qV'
>>> parsePrivateKeyData(_)
('\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa\xaa',
'Standard Base58 key with checksum')


Edit:  I just noticed you that your problem was actually that you crammed the whole 37 bytes into the import function.  The first byte is 0x80 which is the "network private key byte", and the last four bytes are a checksum: So you could've just done "base58_to_binary(...)[1:-4]" and that would've worked, too.
cp1
hero member
Activity: 616
Merit: 500
Stop using branwallets
Thanks, that worked great for a 32-byte hex key.  Is it possible to import from WIF?  I've tried:

wlt.importExternalAddressData(privKey=SecureBinaryData(base58_to_binary("5KaPWnaXTLUdv5WdbwbiXbjDqUFXEUT1aAgSGrYHtzuGD5mooWv")))

But I get: assert(plainPrivKey.getSize()==32)

Do I have to convert from WIF to 32-byte first?  Is there a function in armory?  I was snooping in bulkImportAddresses() and found snippets to deal with base-58 private key, but couldn't turn it into anything working.


Edit:  It must be in there somewhere because I think you can import WIF from armoryqt.  Now I'm on a hunt!
legendary
Activity: 1428
Merit: 1093
Core Armory Developer
I haven't seen that loop since I started running bitcoin-qt independent of armory.  But I've only open and closed it a few times, so it could just be a coincidence.

It really does only happen like once every 10 loads.  And then goes away.  Which will make it tough for me to track it down.  So it's more "expected" than "coincidence"...




I know you're busy, but I have (hopefully) a quick question -- I'm trying to add an address to a wallet following the add 500k address in extras, but only adding 1 key ...

Code:
...

Any ideas?

You just made me realize I should probably remove that example script... that script is actually "special" because someone had wanted to try running Armory with 500k addresses, and I said that it would be ridiculously slow to import them "the right way" (shown below).    So I recommended he manually modify the wallet files with the proper wallet format -- you should not import addresses that way!  

Instead, use the built-in function, and leave everything blank except for the privKey argument:

Code:
def importExternalAddressData(self, privKey=None, privChk=None, \
                                    pubKey=None,  pubChk=None, \
                                    addr20=None,  addrChk=None, \
                                    firstTime=UINT32_MAX,  firstBlk=UINT32_MAX, \
                                    lastTime=0,   lastBlk=0):

I haven't tested the following code, but it would look something like this (note that key data needs to be crammed into a "SecureBinaryData" object before being passed to any of these methods -- but still pretty simple)

Quote from: ExampleCode
# If run from extras dir, need to add parent dir to path
import sys
sys.path.append('..')
from armoryengine import *

# Create the SBD object for the private key which is "aaaaaaaa..."
hardcodedPrivKey = SecureBinaryData(hex_to_binary('aa'*32))

# Read the wallet file (it handles all the backup stuff automatically)
wlt = PyBtcWallet().readWalletFile('/path/to/my.wallet')

# Import the address using the built-in method
wlt.importExternalAddressData(privKey=hardcodedPrivKey)

# Nothing else needs to be done -- the wallet is closed after every operation already
print 'Done!'

# But let's verify it worked:
addr160 = convertKeyDataToAddress(hardcodedPrivKey)
addrStr = hash160_to_addrStr(addr160)
print 'The new address is', addrStr, 'and is in the wallet:', wlt.hasAddr(addr160)


Yeah, it's dead simple when you're using the right calls... sorry for the confusion!  

Edit updated the script with formatting and with the hex_to_binary() call which is needed if you are plugging in a hex private key.  Also, you may have damaged your previous wallet file trying the original script.  You might consider removing it and trying the new script on the original to make sure you're not mixing two different things.
Pages:
Jump to: