Author

Topic: How to convert the backup phrases to a standard HD extended private key? (Read 749 times)

legendary
Activity: 3794
Merit: 1375
Armory Developer
In 1.35c, etotheipi figured he could generate the chaincode from a HMAC of the root, which led to the current 2 strings backups (32 bytes of data). I have no recollection of ever using wallets with a chaincode generated from the bad HMAC (I've been actively contributing the Armory since September 2014). I'm guessing there is a narrow window during which users have generated wallets with the bad HMAC?

In my opinion all wallets are generated with the bad HMAC.

https://github.com/goatpig/BitcoinArmory/blob/dev/armoryengine/ArmoryUtils.py#L1941

Ah I see my mistake now. I didn't think etotheipi's home baked HMAC was still in use. My code relied on cryptopp and now libbtc, which have correct hmac implementations. etotheipi one's botches the pad xoring indeed.

Quote
Because you use the same wrong size (32 instead of 64):

My code uses a proper HMAC implementation (cryptopp/libbtc). I've never ran into this issue because my code rips the chaincode along with the root key from the python wallets in order to convert them to the new format. I went ahead and assumed it generated the chaincode from the root instead, which is why I mistakenly concluded the old python code was using a proper HMAC.

Turns out the new code only generates chaincodes at wallet creation (wouldn't affect preexisting wallets) and when restoring from the root. I have yet to restore python wallets with the new code (they're converted on the fly atm), at which point I'll face this bug directly.
legendary
Activity: 1948
Merit: 2097
In 1.35c, etotheipi figured he could generate the chaincode from a HMAC of the root, which led to the current 2 strings backups (32 bytes of data). I have no recollection of ever using wallets with a chaincode generated from the bad HMAC (I've been actively contributing the Armory since September 2014). I'm guessing there is a narrow window during which users have generated wallets with the bad HMAC?

In my opinion all wallets are generated with the bad HMAC.

This is how armory works:
Code:
bin_root_key = 0xed095dd690c8975209cc820cdda5b71c01ba866623c85396f81ff0eda8416960cffd777c
hash256(bin_root_key) = ddbed0169f589ff48ec32cb448dd16f5288a0719bb2454a15036b8575903ffda
bin_chaincode = HMAC256(hash256(bin_rootkey), 'Derive Chaincode from Root Key')
chaincode = int(codecs.encode(bin_chaincode, 'hex'), 16)
print (hex(chaincode))
84444f1c8c83c9523f120fbae08fa47cb602342ce26b03a31d01dcf343e2e13e

Code:
def HMAC(key, msg, hashfunc=sha512, hashsz=None):
   """ This is intended to be simple, not fast.  For speed, use HDWalletCrypto() """
   hashsz = len(hashfunc('')) if hashsz==None else hashsz
   key = (hashfunc(key) if len(key)>hashsz else key)
   if bytes == str:  # python2
   key = key.ljust(hashsz, '\x00')
   okey = ''.join([chr(ord('\x5c')^ord(c)) for c in key])
   ikey = ''.join([chr(ord('\x36')^ord(c)) for c in key])
   return hashfunc( okey + hashfunc(ikey + msg) )
   else:  #python3  
   key = key.ljust(hashsz, b'\x00')
   okey = bytes(a ^ c for c, a in zip(b'\x5c'*hashsz, key))
   ikey = bytes(a ^ c for a, c in zip(b'\x36'*hashsz, key))
   return hashfunc( okey + hashfunc(ikey + msg.encode('utf-8')) )
  
HMAC256 = lambda key,msg: HMAC(key, msg, sha256, 32)

If instead of 32 I use 64

HMAC256 = lambda key,msg: HMAC(key, msg, sha256, 32) -> HMAC256 = lambda key,msg: HMAC(key, msg, sha256, 64)

I get:

Code:
bin_root_key = 0xed095dd690c8975209cc820cdda5b71c01ba866623c85396f81ff0eda8416960cffd777c
hash256(bin_root_key) = ddbed0169f589ff48ec32cb448dd16f5288a0719bb2454a15036b8575903ffda
bin_chaincode = HMAC256(hash256(bin_rootkey), 'Derive Chaincode from Root Key')
chaincode = int(codecs.encode(bin_chaincode, 'hex'), 16)
print (hex(chaincode))
66fb31cdb3015355c3e2d65bf53f55d4dd45da43d1138a10a887d0e3c7bed51b

that is equal to the result of the correct HMAC:

Code:
import hmac
import hashlib
k = bytearray([0xdd,0xbe,0xd0,0x16,0x9f,0x58,0x9f,0xf4,0x8e,0xc3,0x2c,0xb4,0x48,0xdd,0x16,0xf5,0x28,0x8a,0x07,0x19,0xbb,0x24,0x54,0xa1,0x50,0x36,0xb8,0x57,0x59,0x03,0xff,0xda]);
x = hmac.new(k, "Derive Chaincode from Root Key", hashlib.sha256);
print x.hexdigest()
66fb31cdb3015355c3e2d65bf53f55d4dd45da43d1138a10a887d0e3c7bed51b

then Armory uses a different HMAC function.

Armory in my dev branch is running off of libbtc instead of cryptopp, and it manages to recover older wallets with the expected data, so I wouldn't be too worried about implementing support for the botched HMAC.

Because you use the same wrong size (32 instead of 64):

https://github.com/goatpig/BitcoinArmory/blob/dev/armoryengine/ArmoryUtils.py#L1941
legendary
Activity: 3794
Merit: 1375
Armory Developer
I'm not too sure what's going on here, this was prior to me era of involvement. Armory went through a change in its chaincode generation from v1.35 to v1.35c. Prior to 1.35c, the chaincode was generated as its own PRNG value, with no relation to the root value. As a result, paper backups had to carry 4 strings (64 bytes worth of data).

In 1.35c, etotheipi figured he could generate the chaincode from a HMAC of the root, which led to the current 2 strings backups (32 bytes of data). I have no recollection of ever using wallets with a chaincode generated from the bad HMAC (I've been actively contributing the Armory since September 2014). I'm guessing there is a narrow window during which users have generated wallets with the bad HMAC?

Armory in my dev branch is running off of libbtc instead of cryptopp, and it manages to recover older wallets with the expected data, so I wouldn't be too worried about implementing support for the botched HMAC. I'd just leave that as an unsupported edge case and deal with it when I'd actually run into it, if ever. Keep in mind that the current state of master most likely can't deal with these "bad" chaincodes.
legendary
Activity: 1948
Merit: 2097
Hi,

etotheipi was talking about a problem in the computation of the chaincode:

https://bitcointalksearch.org/topic/m.8890498

How does it work now?

Old wallets and new wallets are using different HMAC functions?

legendary
Activity: 3794
Merit: 1375
Armory Developer
Armory is set to never grab the same address twice. Generating a change address means to increment the use counter and grab the underlying address. You'd need user action to break this behavior, like using the wallet across several processes or restoring the wallet from a backup.
legendary
Activity: 1948
Merit: 2097
Quote
1) how does Armory generate the change addresses?

Grab the next unused address in the chain


The 'next unused' means the next address in the chain or the first address without tx?

Example: I press "Receive Bitcoins" and I get the address A, then I press it again and I get the address B and someone sends bitcoins to the address B. If I spend bitcoin from B, A is still available as change address or Armory grabs another address C?
legendary
Activity: 3794
Merit: 1375
Armory Developer
Quote
1) how does Armory generate the change addresses?

Grab the next unused address in the chain

Quote
2) when you restore a wallet from its Root Key, how many addresses are checked to retrieve the total balance? Maybe it tries until it finds at least 100 consecutive addresses without tx, or more?  I did a test, it stops after 1000 consecutive addresses.

Scan all 1k addresses in the lookup.
legendary
Activity: 1948
Merit: 2097
I'm trying to understand in more detail how it works the Armory's derivation scheme.

Let  
Code:
oiae hiij eauw ekhd aeuu wdau iirh tksu astr
wjjj dfuw hfej nwsn naoi rwgs jeja unni kkku

be the Root Key printed on a paper backup (obviously I use it only to understand the mechanism ...)

'astr' and 'kkku' should be only to check errors at the end of each line, then I remove them and convert from the easy16 to hex format:

Code:
EASY16CHARS  = 'asdf ghjk wert uion'.replace(' ','')
plainRootKeyEasy : oiaehiijeauwekhdaeuuwdauiirhtksuwjjjdfuwhfejnwsnnaoirwgsjejaunni

NORMALCHARS  = '0123 4567 89ab cdef'.replace(' ','')
plainRootKey : ed095dd690c8975209cc820cdda5b71c866623c85396f81ff0eda8416960cffd  (32 bytes)

I derive the chaincode from the Root Key:
Code:
bin_root_key = plainRootKey.to_bytes(32, byteorder='big')
hash256(bin_root_key) = ddbed0169f589ff48ec32cb448dd16f5288a0719bb2454a15036b8575903ffda
chaincode = HMAC256(hash256(bin_root_key), 'Derive Chaincode from Root Key')
chaincode :  84444f1c8c83c9523f120fbae08fa47cb602342ce26b03a31d01dcf343e2e13e  (32 bytes) (it is constant for the entire chain)


To pass from a private key (index N) to the next one (index N+1) I have to do a multiplication a * b mod n, where:

Code:
n = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141
a = chaincode xor hash256(pubkey)
b = privkey (index N)

nextprivkey = a*b % n
nextpubkey = (a*b % n)*G  where G is the generator of the secp256k1 curve
nextpubkey = a*(bG) = a*P where P is the public key (index N)

The parameter 'a' (together the related address) is stored in the multipliers.txt file in .armory directory.
 
'a' is like a sort of private key of the address N+1 computed respect to the previous public key N instead of the generator G.
In this way the wallet offline can generate these particular private keys (multipliers) only from the first (root) public key and the chaincode. We get all the public keys (as multiples of the root public key) and then all the addresses without using the real private keys (because to get an address we need only a public key, the private key is not necessary).

public keys   chain:  root public key P1 -> a1*P1 = P2 -> a2*P2 = P3 -> a3*P3 = P4 -> ...
private keys chain: root private key a  -> a*a1  % n  ->  a*a1*a2 % n -> a*a1*a2*a3 % n --> ...

If only one private key N and the chaincode are revelead, then
Private key N with the chaincode will let you compute all private keys beyond N. To compute private keys prior to N, you need all public keys preceding N as well. Both chaincode and public keys are considered known data (as they lay on online computers), hence the generalization that revealing a private key within the chain is as good as revealing all private keys in the wallet.
 
Returning to the computations, in my particular case:

Code:
chaincode :  84444f1c8c83c9523f120fbae08fa47cb602342ce26b03a31d01dcf343e2e13e
pubkey : 0445563be187fb75440607375558bcc0676e84b30040a2068747f94f82e78dfeb911299531610d0ec3a5602350a643dadd9af51b21f2c33ad446ea9c9f099c61e1
hash256(pubkey) : 12f1ed57097159dbb709229e48aafb2e080f6f0dfe8213b7fa836942e4aba08
a = 856b51c9fc14dccf84629d9304050bce5682c2dc3d83229862a9ea676da85b36
b = ed095dd690c8975209cc820cdda5b71c866623c85396f81ff0eda8416960cffd

nextprivkey = 27dcda057ab1f2c114da68f3fb5d1c42456ccf76b606064fdef4ca38167fb508


Then the first private key (and address) of this wallet is:
Code:
privkey : 27dcda057ab1f2c114da68f3fb5d1c42456ccf76b606064fdef4ca38167fb508
address : 1AyEzG2REn9szCTimJZjZx9X2TJ25DfAqW (base58)
address : 6d5c1a1398b2a869535afda83c3ca62e6537fb6f --> first 5 bytes: \x6d \x5c \x1a \x13 \x98

Id wallet = binary_to_base58(ADDRBYTE + firstAddr.getAddr160()[:5])[::-1]
Id wallet = binary_to_base58(b'\x98\x13\x1a\x5c\x6d\x00')
Id wallet : 2JjGQxFtK

If I apply again the same scheme, I get the other addresses:
Code:
#1  private key  : 27dcda057ab1f2c114da68f3fb5d1c42456ccf76b606064fdef4ca38167fb508	address: 1AyEzG2REn9szCTimJZjZx9X2TJ25DfAqW
#2  private key  : 94094ed331ca8fba87e9f223e71df6945aac5bde1b8523584d828b596f3bd167 address: 1BU1WGcrkNTRSzLVQnEYeoWHXecGYK9YBv
#3  private key  : 80aced7c27ac69b5f359572773c45be903d90012f96b8ae7da398242c2019c47 address: 12JSLGgYVY8EdeF1dfZWpSspLVE2A4Sms8
#4  private key  : db5099b13a7a0309e705e2d6166349ff049cfb95203ddc66b652ae61bd5aac84 address: 1CZqeKPka88ARov1CGXKugU7WEmgENs3My
#5  private key  : f79d421ee19f9278cb802db35bf4df9ec14b57d4aef3ea70d8a659e4e6e8bd3e address: 1K85SqJVxJio5awb3kaCe1VN6vpsfXEXFV
...
#101  private key  : f6c25c16b2bce422fd43c95cfafebc517def32ddbc71ec53259fe3670e863dac address: 18gK7fuU6ymxEdH1ngieiehK4GyzLXBAzy

I noted that Armory at the start generates a pool of 100 addresses. Each time I press the button "Receive Bitcoins", it generates another address (then there are always at least 100 empty addresses in the Address Pool).


2 questions:

1) how does Armory generate the change addresses?

2) when you restore a wallet from its Root Key, how many addresses are checked to retrieve the total balance? Maybe it tries until it finds at least 100 consecutive addresses without tx, or more?  I did a test, it stops after 1000 consecutive addresses.
legendary
Activity: 1948
Merit: 2097
Keep in mind that you shouldn't treat soft derived BIP32 private keys with any less care, as the same property applies:

Quote
One weakness that may not be immediately obvious, is that knowledge of a parent extended public key plus any non-hardened private key descending from it is equivalent to knowing the parent extended private key (and thus every private and public key descending from it). This means that extended public keys must be treated more carefully than regular public keys.

It is good practice to not reveal your wallet's private keys, regardless of the derivation scheme. Consider the wallet compromised if you did, and move the funds out.

Thank you for the clarification.
legendary
Activity: 3794
Merit: 1375
Armory Developer
Those are pointing to the dev branch, which is in constant flux. You can look at the code in the master branch:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/EncryptionUtils.cpp#L749

Quote
You confirm that if a single private key was revelead the entire wallet is compromised?

Private key N with the chaincode will let you compute all private keys beyond N. To compute private keys prior to N, you need all public keys preceding N as well. Both chaincode and public keys are considered known data (as they lay on online computers), hence the generalization that revealing a private key within the chain is as good as revealing all private keys in the wallet.

Keep in mind that you shouldn't treat soft derived BIP32 private keys with any less care, as the same property applies:

Quote
One weakness that may not be immediately obvious, is that knowledge of a parent extended public key plus any non-hardened private key descending from it is equivalent to knowing the parent extended private key (and thus every private and public key descending from it). This means that extended public keys must be treated more carefully than regular public keys.

It is good practice to not reveal your wallet's private keys, regardless of the derivation scheme. Consider the wallet compromised if you did, and move the funds out.
legendary
Activity: 3794
Merit: 1375
Armory Developer
That quote gives you links directly to the code.
legendary
Activity: 1948
Merit: 2097
Armory HD wallets predate BIP32. As a matter of fact, the design in Armory was part of the inspiration for BIP32. Now, I am in the process of adding BIP32 support to Armory, and that is scheduled for the next major release. For now though, there is no such functionality available in Armory.

To compute chained addresses, Armory first derives what is called the chaincode from the wallet's root private key. You can see the Python code here:

https://github.com/goatpig/BitcoinArmory/blob/dev/armoryengine/ArmoryUtils.py#L3489

Using the chain code, Armory can then derive the address chain sequentially, each new key pair N+1 resulting from the exponentiation of the key pair N and the chaincode. The exact procedure can be seen in the code here:

https://github.com/goatpig/BitcoinArmory/blob/dev/cppForSwig/EncryptionUtils.cpp#L825
https://github.com/goatpig/BitcoinArmory/blob/dev/cppForSwig/EncryptionUtils.cpp#L749

Where is the exact procedure now to derive the private key chain from the wallet's root private key? I use a wallet created in 2013.
I'd like to see an article that explains how it works.

I read this answer too:

https://bitcointalksearch.org/topic/m.8570465

and I found out that a wallet is compromised if I reveal a single private key!
member
Activity: 80
Merit: 14
Cool, thank you both.

I'll take a look into the code, goat.
legendary
Activity: 3794
Merit: 1375
Armory Developer
Hi Goat, thanks for the response.
So, I have 2 lines with 36 hexadecimal characters each after I convert them.

What is that exactly? I'm learning everything related to Bitcoin coding using NBitcoin, so I kinda grasp the most important concepts but not all.

Thank you!

It is the wallet's root private key, a 32 bytes binary string, from which Armory wallets are derived.

Hi! Armory doesn't use the standard derivation path or it uses a different algorithm altogether?

Armory HD wallets predate BIP32. As a matter of fact, the design in Armory was part of the inspiration for BIP32. Now, I am in the process of adding BIP32 support to Armory, and that is scheduled for the next major release. For now though, there is no such functionality available in Armory.

To compute chained addresses, Armory first derives what is called the chaincode from the wallet's root private key. You can see the Python code here:

https://github.com/goatpig/BitcoinArmory/blob/dev/armoryengine/ArmoryUtils.py#L3489

Using the chain code, Armory can then derive the address chain sequentially, each new key pair N+1 resulting from the exponentiation of the key pair N and the chaincode. The exact procedure can be seen in the code here:

https://github.com/goatpig/BitcoinArmory/blob/dev/cppForSwig/EncryptionUtils.cpp#L825
https://github.com/goatpig/BitcoinArmory/blob/dev/cppForSwig/EncryptionUtils.cpp#L749



staff
Activity: 3458
Merit: 6793
Just writing some code
You won't get the same addresses since Armory does not use BIP 32.

Hi! Armory doesn't use the standard derivation path or it uses a different algorithm altogether?
It uses an entirely different algorithm that was developed before BIP 32.
member
Activity: 80
Merit: 14
You won't get the same addresses since Armory does not use BIP 32.

Hi! Armory doesn't use the standard derivation path or it uses a different algorithm altogether?
member
Activity: 80
Merit: 14
They're encoded in base16 with a different alphabet.

You can see most of the code here:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/ArmoryUtils.py#L2244
They're encoded in base16 with a different alphabet.

You can see most of the code here:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/ArmoryUtils.py#L2244

Hi Goat, thanks for the response.
So, I have 2 lines with 36 hexadecimal characters each after I convert them.

What is that exactly? I'm learning everything related to Bitcoin coding using NBitcoin, so I kinda grasp the most important concepts but not all.

Thank you!
staff
Activity: 3458
Merit: 6793
Just writing some code
You won't get the same addresses since Armory does not use BIP 32.
legendary
Activity: 3794
Merit: 1375
Armory Developer
They're encoded in base16 with a different alphabet.

You can see most of the code here:

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/ArmoryUtils.py#L2244
member
Activity: 80
Merit: 14
Title, basically.
Is there a way to derive from the two phrases composed of words of 4 letters each?
Jump to: