Armory - Discussion Thread - page 151.

chrisrico

hero member

Activity: 496

Merit: 500

etotheipi, I think the "parseNewBlockData did not get enough data..." error is coming from this loop.

Code:

      while(bsb.streamPull())
      {
         while(bsb.reader().getSizeRemaining() > 8)
         {
            ...
   
            BinaryRefReader brr(bsb.reader().getCurrPtr(), nextBlkSize);
            parseNewBlockData(brr, fnum-1, bsb.getFileByteLocation(), nextBlkSize);
            blocksReadSoFar_++;
            bytesReadSoFar_ += nextBlkSize;
            bsb.reader().advance(nextBlkSize);
         }
      }

I made this small change and it drastically reduces the output, my freshly imported wallet still shows the right balance, but I'm sure it's not a real fix, since it looks like there's a bunch of data left unread...

Code:

            if(!parseNewBlockData(brr, fnum-1, bsb.getFileByteLocation(), nextBlkSize))
            {
               cout << "Next block size: " << nextBlkSize << endl;
               cout << "Buffer size remaining: " << bsb.reader().getSizeRemaining() << endl;
               cout << "Reader size remaining: " << brr.getSizeRemaining() << endl;
               break;
            }

And here's some sample output with that change:

Code:

***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 2231
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 3349
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00001.dat
/home/bitcoin/.bitcoin/blocks/blk00001.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 9362
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 20808
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00002.dat
/home/bitcoin/.bitcoin/blocks/blk00002.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 16391
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 28286
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00003.dat
/home/bitcoin/.bitcoin/blocks/blk00003.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 25310
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 25309
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00004.dat
/home/bitcoin/.bitcoin/blocks/blk00004.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 3530
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 6970
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00005.dat
/home/bitcoin/.bitcoin/blocks/blk00005.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 15835
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 24656
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00006.dat
/home/bitcoin/.bitcoin/blocks/blk00006.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 27895
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 29726
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00007.dat
/home/bitcoin/.bitcoin/blocks/blk00007.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 14370
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 41116
Reader size remaining: 0

chrisrico

hero member

Activity: 496

Merit: 500

Ok, not to continue hijacking the threat, but I stayed up all night coding again and I vastly improved the serial communication.

Now, Armory tries to listen on the serial device as soon as it's started (rather than only in the offline tx dialog), and it encapsulates its messages to and from the server with protocol buffers. This allowed me to expand the types of messages available (as well as add more robust handling). Currently, you can import a watching only copy of a wallet directly from the offline device. I plan on extending the "Create new wallet" dialog to allow something similar - initiate the creation of a new wallet on the offline device, and send the watching only copy back. Also needed is to make the connection off by default, and enabled in the advanced or expert options, as well as setting the serial device and baud rate.

Also, I tested it on the Pi and a rate of 115200 works even better than 9600... Imagine that! I didn't run into any apparent slowness until trying to transfer a wallet over the wire...

The diff now has quite a few more changes, but it's still surprising to me how little code it took. I'm really liking python, and etotheipi, your code base is a pleasure to work with!

Red Emerald

hero member

Activity: 742

Merit: 500

Quote from: Ploo on December 12, 2012, 04:26:12 PM

Here's something that's on my mind that I hope to get cleared up.

I currently have a cold-storage set up thanks to Armory. Something that only just now came to my attention is the fact that the online watch-only wallet is managing to generate addresses without knowing their private keys. How does it do this? Does the watch-only copy derive a new seed that is only capable of generating public keys in the series? What's the process behind this? Or does the watch-only copy come with a healthy bulk of pre-generated addresses?

Armory wallets are deterministic. http://bitcoinarmory.com/index.php/using-offline-wallets-in-armory

chrisrico

hero member

Activity: 496

Merit: 500

Quote from: Ploo on December 12, 2012, 04:26:12 PM

Does the watch-only copy derive a new seed that is only capable of generating public keys in the series? What's the process behind this?

That's exactly it. Here is the code that creates a watching-only fork of the wallet.

Basically, the root key has a public and private portion. Given only the public portion, you can derive the subsequent public keys, but not the private keys, for the same reason that you can't derive a private key from a public one.

Ploo

full member

Activity: 125

Merit: 100

Here's something that's on my mind that I hope to get cleared up.

I currently have a cold-storage set up thanks to Armory. Something that only just now came to my attention is the fact that the online watch-only wallet is managing to generate addresses without knowing their private keys. How does it do this? Does the watch-only copy derive a new seed that is only capable of generating public keys in the series? What's the process behind this? Or does the watch-only copy come with a healthy bulk of pre-generated addresses?

BRules

sr. member

Activity: 293

Merit: 250

tried to find the answer but couldn't. Is there a way to verify a message signed in armory with the bitcoinqt and vice versa?

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: chrisrico on December 11, 2012, 07:37:48 PM

In this thread Peter posted a link to some experimental pre-0.8 builds on his server. That's what I'm running. Block data files are now ~128MB, numbered blk000X.dat, and in the .bitcoin/blocks folder. Other than that, I'm not sure what changed.

Yeah, I know what they said it was changing to, that's where this function came from: findLatestBlkFiles(). It should default to looking for blocks/blk0000X.dat, and if those aren't there, switch to blk000X.dat. It's really annoying that the old version is 4 digits and 1-indexed, the new version is 5 digits and zero-indexed (accommodating both versions is ugly, oh well.)

Thanks for the link. I'll dig in when I get some time after RAM reduction (which is going mediocre at the moment, but I may have something worth testing in the next week or two).

chrisrico

hero member

Activity: 496

Merit: 500

Piggybacking off my other posts, here are the changes I made to the Pi:

In /etc/inittab, change the line that looks like this

Code:

1:2345:respawn/bin/agetty ...

to this

Code:

1:2345:respawn:/bin/login -f pi tty1 /dev/tty1 2>&1

This will automatically log in as the pi user to the first tty device upon boot.

Also, remove or comment out the line that looks like this

Code:

T0:23:respawn:/sbin/getty -L ttyAMA0 115200 vt100

That disables logging in over the serial port.

In /boot/cmdline.txt, remove console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 from

Code:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1

This disables outputting boot and debug information over the serial port.

Modify /etc/sudoers (with visudo) to include this line

Code:

pi ALL=NOPASSWD: /sbin/poweroff

This lets you you power off the pi without entering your sudo password.

Finally, in /home/pi/.bashrc, add the following at the bottom

Code:

python $armory_directory/offline_serial_server.py /dev/ttyAMA0 9600
sudo poweroff

Voila. Your Pi will now boot into the offline serial server, allowing you to communicate with it from Armory using the patch I posted at the top of the page. When you're finished, hit CTRL-C and the Pi will cleanly power down.

chrisrico

hero member

Activity: 496

Merit: 500

In this thread Peter posted a link to some experimental pre-0.8 builds on his server. That's what I'm running. Block data files are now ~128MB, numbered blk000X.dat, and in the .bitcoin/blocks folder. Other than that, I'm not sure what changed.

Oh, and it looks like there is a problem where Armory doesn't see any new blocks coming in after it's caught up, when connected to a 0.8 client. I thought it was a permissions issue, but after fixing that I'm still seeing it. Nothing interesting in the log, either.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: chrisrico on December 11, 2012, 06:55:53 PM

Quote from: etotheipi on December 11, 2012, 06:49:12 PM

SOB! I even moved my blk files around to test this before committing the upgrade to beta. If this truly is broken, I need to fix that, because I was under the assumption that this should coast users through the 0.8 upgrade.

And given those errors you report, looks like it may be more than renaming block files. Perhaps they changed the way data is written, too...? Gah! I guess I need to just download 0.8 and test it myself...

Sorry, I might not have been clear. It reads the files, but when it gets to the "end" of one, it throws this error a bunch (random amount?), before it starts reading the next. However, it does seem like maybe once it catches up, Armory doesn't update when new blocks come in when connected to a 0.8 node.

Is there a compiled version of 0.8, or do I have to build it myself? All my updates for 0.8 were based on the assumption that nothing is changing except for the locations and names of the block files. At least, that's all the core developers told me about

chrisrico

hero member

Activity: 496

Merit: 500

Quote from: etotheipi on December 11, 2012, 06:49:12 PM

SOB! I even moved my blk files around to test this before committing the upgrade to beta. If this truly is broken, I need to fix that, because I was under the assumption that this should coast users through the 0.8 upgrade.

And given those errors you report, looks like it may be more than renaming block files. Perhaps they changed the way data is written, too...? Gah! I guess I need to just download 0.8 and test it myself...

Sorry, I might not have been clear. It reads the files, but when it gets to the "end" of one, it throws this error a bunch (random amount?), before it starts reading the next.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: chrisrico on December 11, 2012, 01:37:16 AM

edit... I found a bug (or two). If you're running the LevelDB version of Bitcoin, Armory won't go online if the blk00*.dat files don't exist, even if the blocks/blk000*.dat files do. Also, I saw a lot of this

Code:

***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...

at what I am assuming is the end of a block data file, before the next one begins scanning. It can last for quite a few seconds sometimes, churning those out 10 per seconds or so.

SOB! I even moved my blk files around to test this before committing the upgrade to beta. If this truly is broken, I need to fix that, because I was under the assumption that this should coast users through the 0.8 upgrade.

And given those errors you report, looks like it may be more than renaming block files. Perhaps they changed the way data is written, too...? Gah! I guess I need to just download 0.8 and test it myself...

marcus_of_augustus

legendary

Activity: 3920

Merit: 2349

Eadem mutata resurgo

Quote

After considering the criticism of my Armory Pi setup, I spent the day learning serial ports, (more) python, and Armory. I'll release the codes tomorrow, but my solution is pretty sweet. I disabled the serial terminal and replaced it with a program that listens for unsigned transactions. When it sees one, it unlocks the wallet with the passphrase entered on the Pi's own keyboard, signs the transaction, and sends it back over the serial port. The other modification is to Armory, specifically the first offline transaction screen. Along with "Save as file" and "Copy to clipboard" options, there is now a "Transmit over serial" button, which does just what it seems. When it receives the signed transaction back, it automatically advances to and fills the text box on the transaction broadcast screen.

Okay, this raises the bar significantly. Basically, you have built yourself a RasbPi minimally-connected 'hardware' wallet for Armory ... good stuff.

chrisrico

hero member

Activity: 496

Merit: 500

Quote from: etotheipi on December 11, 2012, 02:47:42 PM

Yes! I agree this is an excellent solution -- one that I very much wanted to implement myself, but decided could not be promoted as a general solution -- due to the risk of users not disabling terminal logins via serial (thus, actually decreasing security for a number of impatient/inattentive users).

But, I always wanted such a solution that could be documented on the website as "Use at your own risk, but it's a pretty sweet solution if you do it right...". I would personally use this, I just never got around to implementing a serial protocol. I suppose, alternatively, I could find a way to make an Armory offline bundle DVD -- with OS and everything preconfigured correctly for this purpose. Then the serial solution would be perfect

Until then, I will graciously accept the modifications to Armory as a side-branch, or maybe a diff file or something that can be downloaded/distributed separately. Put instructions on the website with appropriate warnings.

Thanks! Yeah, I don't think this needs to be a core part of Armory, especially since there are so few changes necessary, and Armory is very easy to build (at least on Linux).

I've run into one strange issue though... the extra scripts work just fine on my online machine, but on the Pi they stopped working. It's not giving me an "unrecognized symbol: armoryengine error", it just doesn't recognize CLI_ARGS (or any other "constant" from armoryengine), even though I'm doing "from armoryengine import *". Oddly, running "python armoryengine.py" works just fine, and starting up a REPL and doing "from armoryengine import *" lets me "print CLI_ARGS" without issue. Any ideas?

edit... Hmm. It must be a path issue of some kind, because when I move the script out of the extras directory and remove the "sys.path.append('..')", it works.

etotheipi

legendary

Activity: 1428

Merit: 1093

Core Armory Developer

Quote from: chrisrico on December 11, 2012, 01:37:16 AM

I'm really excited!

After considering the criticism of my Armory Pi setup, I spent the day learning serial ports, (more) python, and Armory. I'll release the codes tomorrow, but my solution is pretty sweet. I disabled the serial terminal and replaced it with a program that listens for unsigned transactions. When it sees one, it unlocks the wallet with the passphrase entered on the Pi's own keyboard, signs the transaction, and sends it back over the serial port. The other modification is to Armory, specifically the first offline transaction screen. Along with "Save as file" and "Copy to clipboard" options, there is now a "Transmit over serial" button, which does just what it seems. When it receives the signed transaction back, it automatically advances to and fills the text box on the transaction broadcast screen.

All this required surprisingly little code, but I need some sleep right now!

Yes! I agree this is an excellent solution -- one that I very much wanted to implement myself, but decided could not be promoted as a general solution -- due to the risk of users not disabling terminal logins via serial (thus, actually decreasing security for a number of impatient/inattentive users).

But, I always wanted such a solution that could be documented on the website as "Use at your own risk, but it's a pretty sweet solution if you do it right...". I would personally use this, I just never got around to implementing a serial protocol. I suppose, alternatively, I could find a way to make an Armory offline bundle DVD -- with OS and everything preconfigured correctly for this purpose. Then the serial solution would be perfect

Until then, I will graciously accept the modifications to Armory as a side-branch, or maybe a diff file or something that can be downloaded/distributed separately. Put instructions on the website with appropriate warnings.

chrisrico

hero member

Activity: 496

Merit: 500

Here is a diff of my changes: http://pastebin.com/A9x0grtz

There's probably a better way to do some of the threaded stuff for the listener, and I'll probably clone the Github project and submit a push request to make it easier for either etotheipi or others to fix it up, but right now it works pretty well!

prezbo

sr. member

Activity: 430

Merit: 250

Quote from: picobit on December 11, 2012, 02:07:02 AM

Quote from: prezbo on December 10, 2012, 11:30:43 AM

I do not own a safe, so if someone were to break in my house and get to the paper backup, the funds would probably be gone before I would realized it had been stolen. So my idea is to have hdd backup of the wallet with a strong passphrase, and a single copy of a paper wallet with a weak enough passphrase that I could bruteforce it in a decent amount of time in case I would forget it, but would give me enough time to clear the funds out if someone were to steal it. I hope that makes sense.

That would be a most unusual burglar. 99% of burglars would say "crap, a piece of paper, not money, not drugs ... worthless". The last percent would actually look on the paper before saying "what a nerd" and throw it away.

I would be more inclined to think like this if the paper backup didn't have "Armory Wallet" written in bold on it

If I were a burglar and saw a piece of paper with the word wallet on it, I would surely look into it.

Ente

legendary

Activity: 2126

Merit: 1001

Quote from: chrisrico on December 11, 2012, 01:37:16 AM

I'm really excited!

After considering the criticism of my Armory Pi setup, I spent the day learning serial ports, (more) python, and Armory. I'll release the codes tomorrow, but my solution is pretty sweet. I disabled the serial terminal and replaced it with a program that listens for unsigned transactions. When it sees one, it unlocks the wallet with the passphrase entered on the Pi's own keyboard, signs the transaction, and sends it back over the serial port. The other modification is to Armory, specifically the first offline transaction screen. Along with "Save as file" and "Copy to clipboard" options, there is now a "Transmit over serial" button, which does just what it seems. When it receives the signed transaction back, it automatically advances to and fills the text box on the transaction broadcast screen.

All this required surprisingly little code, but I need some sleep right now!

Awesome! That's the way! :-)
I was thinking about how to automatize the whole external signing, but didn't find such a nice solution like you!
That way it would be almost-fully-automatized, with only one click extra and one passphrase on pi's keyboard!
I like it!
So, is the Pi available again? :-)

Ente

picobit

hero member

Activity: 547

Merit: 500

Decor in numeris

Quote from: prezbo on December 10, 2012, 11:30:43 AM

I do not own a safe, so if someone were to break in my house and get to the paper backup, the funds would probably be gone before I would realized it had been stolen. So my idea is to have hdd backup of the wallet with a strong passphrase, and a single copy of a paper wallet with a weak enough passphrase that I could bruteforce it in a decent amount of time in case I would forget it, but would give me enough time to clear the funds out if someone were to steal it. I hope that makes sense.

That would be a most unusual burglar. 99% of burglars would say "crap, a piece of paper, not money, not drugs ... worthless". The last percent would actually look on the paper before saying "what a nerd" and throw it away.

chrisrico

hero member

Activity: 496

Merit: 500

I'm really excited!

After considering the criticism of my Armory Pi setup, I spent the day learning serial ports, (more) python, and Armory. I'll release the codes tomorrow, but my solution is pretty sweet. I disabled the serial terminal and replaced it with a program that listens for unsigned transactions. When it sees one, it unlocks the wallet with the passphrase entered on the Pi's own keyboard, signs the transaction, and sends it back over the serial port. The other modification is to Armory, specifically the first offline transaction screen. Along with "Save as file" and "Copy to clipboard" options, there is now a "Transmit over serial" button, which does just what it seems. When it receives the signed transaction back, it automatically advances to and fills the text box on the transaction broadcast screen.

All this required surprisingly little code, but I need some sleep right now!

edit... I found a bug (or two). If you're running the LevelDB version of Bitcoin, Armory won't go online if the blk00*.dat files don't exist, even if the blocks/blk000*.dat files do. Also, I saw a lot of this

Code:

***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...

at what I am assuming is the end of a block data file, before the next one begins scanning. It can last for quite a few seconds sometimes, churning those out 10 per seconds or so.

Topic: Armory - Discussion Thread - page 151. (Read 521829 times)