Pages:
Author

Topic: Using Armory on the BCH chain - page 2. (Read 46034 times)

legendary
Activity: 3794
Merit: 1375
Armory Developer
November 22, 2018, 09:17:24 AM
Hah, you learn something new everyday.

In c++, variables can be changed by passing them to functions, thanks to the reference notation.

This is something you need to pay attention to when reading the method declaration. const& is used to avoid copies when passing arguments, and means the arg is an "in". Plain & implies the method is supposed to modify this reference, in which case it's considered an "out" argument.

This is common practice when you want to use the return value of the method as a status report. Typically your return an int, with 0 for success and negative values for error codes. You can also use this approach if the method has to provide several results, but that style is mostly for C. In C++ you tend to report errors by throwing, and write a class/struct for return types that require several members.

As seen in the code you linked to, the style is actually confusing: the semantics do not outright suggest ReadLastBlockFile takes a reference, as it is not being checked for a status return. This kind of semantics suggest you are setting some internal value for object pblocktree instead. The following snippet would be a little easier to read:

Code:
int result = pblocktree->ReadLastBlockFile(nLastBlockFile)
if(result != 0)
{
   //error handling
}

//do something with nLastBlockFile


With that in mind, the change in BU could very well be a mistake that led to a benign change in behavior, assuming the author of the change did not bother to acquire a broad understanding of this code before modifying it. He possibly saw a global variable changed by a local object and figured that's bad style, correcting for style without caring for the substance.
legendary
Activity: 1232
Merit: 1094
November 21, 2018, 06:54:17 PM
Hah, you learn something new everyday.

In c++, variables can be changed by passing them to functions, thanks to the reference notation.

I checked the validation.cpp file and could only see one place where there is a change to the variable.

It is also changed using this code.

Code:
pblocktree->ReadLastBlockFile(nLastBlockFile);

ReadLastBlockFile takes int& nFile as its input.

I thought references like that were inherently constants.

I think this effect is due to the changes to blockstorage.cpp (including some refactoring away from validation.cpp).  The call to ReadLastBlockFile doesn't change the nLastBlockFile global variable.  It just targets a local variable in the function.

This means the global variable stays at its default of zero.

This is what allows the node to write to previously completed blk*.dat files rather than only looking at new blk files.

In the Bitcoin Core client, the call to ReadLastBlockFile has a side effect that initializes the nLastBlockFile variable.

This is a Bitcoin Unlimited only behavior, I think, since Bitcoin Core and Bitcoin ABC haven't made the changes.
legendary
Activity: 1232
Merit: 1094
November 20, 2018, 06:40:06 PM
I initially wanted this change for security reasons: Core needs to be exposed to the WAN but Armory does not need the WAN

Seems reasonable.  Even if Armory accidentally connected to a remote node and received a false block hash, the local node's filesystem wouldn't have that block.  In that case, Armory stalls rather than trusting the block data from the remote host.
legendary
Activity: 3794
Merit: 1375
Armory Developer
November 20, 2018, 05:44:25 PM
Blocks over P2P would actually fix this, because it would get the longest chain from your node, then grab block data by hash from that chain of headers, all over the P2P socket, completely ignoring the on disk file structure. I initially wanted this change for security reasons: Core needs to be exposed to the WAN but Armory does not need the WAN unless you want to connect to the DB remotely, and even then you could tunnel to it or put it behind a daemon. Limiting ArmoryDB's interactions with Core to just a couple sockets (P2P and RPC) would make way for more secure setups. Dodging this edge case is an added bonus now.
legendary
Activity: 1232
Merit: 1094
November 20, 2018, 03:58:58 PM
Quote
Once it writes to a blk file, it never checks files lower than that anymore.  That is a RAM variable though, so it gets set back to zero each time a node is restarted.

Why would it need to check for previous files appends on new blocks? If the issue only happens on fresh node start (when free space in older block files is eligible), as long as Armory sees that one new block, it will work.

I am not sure if they fully thought out the logic.  I may have missed something when looking at the code on core that protects against the issue.

I think it is a pretty bad policy no matter what.  It's not like keeping all the blk files at the same size actually matters, and keeping nearby blocks in the same file is likely a help.

They want a way to fill in spaces in the old block files but once a file overflows since last restart, they just don't bother with that one anymore.

The sequence is

- on startup set nLastBlockFile to 0
- when a new block arrives scan for a blk file with space starting at index = nLastBlockFile
- Once it jumps to a new file, it sets nLastBlockFile to that value

This means that nLastBlockFile never decreases, so once the node is running for a while it only writes to the latest blk file.

On the next reboot, different spaces could end up being filled.

Quote
Armory does not expect blocks to appear in order in files. It will check block data on disk from the last known block to have extended the chain. Therefor if you have up to blk01000.dat and the last top is in blk00997.dat, it will check #977 to #1000 on every new block notifications until it finds a block that extends the chain again.

Cool, so re-scanning problematic blk files wouldn't be an issue.  The only question is detecting them. 

It looks like c++ doesn't have a portable way to get the last modified time.

The assumption that later numbered blk files contain later data is violated by the code.  In that situation, blk #800 could be updated and no amount of scanning after 977 will fix things.

Quote
The ultimate solution would be blocks over P2P, but idk what kind of mess BCH will be by the time I'm done with that. Chances are Armory won't be compatible anymore at that point.

I don't think it is only restricted to Bitcoin Cash.  The problem is much more likely to be exposed by the variable block sizes, but I think it is possible even if they are mostly stable blocks.  You could get a 950kB space that doesn't get filled for a while and then someone is unlucky that they start their node just before a block that can fill that space arrives.

It might actually "settle down" once it has synced and then been restarted a few times.  This would "fill in" many of the spaces.  As long as the first block found after a restart is bigger than any of the spaces, then it will write to the file with the highest number and that jumps nLastBlockFile to the max value.

I think the workaround of wiping the armory database is probably acceptable, since the problem is rare and doesn't seem to affect the main chain.
legendary
Activity: 3794
Merit: 1375
Armory Developer
November 20, 2018, 01:30:42 PM
Quote
A solution would be to keep a record of "last-checked time" and then re-scan any old blk files that have a modified time greater or equal to the last-checked time.  This check could be run any time a new block notification is received.

Quote
Once it writes to a blk file, it never checks files lower than that anymore.  That is a RAM variable though, so it gets set back to zero each time a node is restarted.

Why would it need to check for previous files appends on new blocks? If the issue only happens on fresh node start (when free space in older block files is eligible), as long as Armory sees that one new block, it will work.

Armory does not expect blocks to appear in order in files. It will check block data on disk from the last known block to have extended the chain. Therefor if you have up to blk01000.dat and the last top is in blk00997.dat, it will check #977 to #1000 on every new block notifications until it finds a block that extends the chain again.

Quote
Would re-scanning already scanned blk files cause problems for Armory?

DB is designed to whistand being written over. Rescanning is fine. As a matter of fact, on each start Armory rewinds back 100 blocks from the last known top to account for long reorgs occuring while it wasn't running. You can see it here: https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/DatabaseBuilder.cpp#L55

Code:
      //rewind the top block offset to catch on missed blocks for db init
      auto topBlock = blockchain_->top();
      auto rewindHeight = topBlock->getBlockHeight();
      if (rewindHeight > 100)
         rewindHeight -= 100;
      else
         rewindHeight = 1;

      auto rewindBlock = blockchain_->getHeaderByHeight(rewindHeight);
      topBlockOffset_.fileID_ = rewindBlock->getBlockFileNum();
      topBlockOffset_.offset_ = rewindBlock->getOffset();

      LOGINFO << "Rewinding 100 blocks";

You can increase this value to try and confirm your theory. On Bitcoin a blk file is ~100 blocks, on BCH idk, try much larger stuff.

The ultimate solution would be blocks over P2P, but idk what kind of mess BCH will be by the time I'm done with that. Chances are Armory won't be compatible anymore at that point.
legendary
Activity: 1232
Merit: 1094
November 20, 2018, 10:43:01 AM
This might be a Bitcoin Cash only issue (or even Bitcoin Unlimited only issue).

On the other hand, it might be a source of intermittent/hard to track down blockchain stalls with all the other clients.

It looks like the Bitcoin (Cash) Unlimited client manages the block files in an unexpected way.  When I look at the block files, some earlier block files have modified times that are much more recent than they should be.

The equivalent Bitcoin Core code seems to work the same way, but I haven't actually checked.

I did a full resync (deleted all the blk files and chainstate) and left the node running in the background.  Once I noticed that it resynced, I restarted the node (before the 15:57 timestamps) and then started Armory after wiping the Armory databases.

Armory resynced and then stalled.  It said connected to the node.

This is the end of "ls -lrt blk0*" and I assume the node completed resyncing around 6:45am.

Code:
-rw------- 1 tiern tiern 120188494 Nov 19 06:41 blk01037.dat
-rw------- 1 tiern tiern 134182052 Nov 19 06:41 blk01038.dat
-rw------- 1 tiern tiern 134024033 Nov 19 06:42 blk01039.dat
-rw------- 1 tiern tiern 127342200 Nov 19 06:44 blk01040.dat
-rw------- 1 tiern tiern 131259910 Nov 19 06:45 blk01041.dat
-rw------- 1 tiern tiern  83886080 Nov 19 13:14 blk01042.dat
-rw------- 1 tiern tiern 134215061 Nov 19 15:57 blk00001.dat
-rw------- 1 tiern tiern 134215184 Nov 19 15:57 blk00004.dat
-rw------- 1 tiern tiern 134216866 Nov 19 15:57 blk00006.dat
-rw------- 1 tiern tiern 134213458 Nov 19 15:57 blk00007.dat
-rw------- 1 tiern tiern 134198809 Nov 19 15:57 blk00008.dat
-rw------- 1 tiern tiern 134207767 Nov 19 15:57 blk00009.dat

Bitcoin node code was changed to this method pretty early on (in this commit from Aug 2012).

When a new block is received from the network, it scans all the blk files and finds the first one that has enough space to store the new block.

This means that the file systems doesn't act as "append-only" overall.  It is append-only for each file though.

It has a variable called nLastBlockFile.  This variable acts as a lower limit on the scan.  Once it writes to a blk file, it never checks files lower than that anymore.  That is a RAM variable though, so it gets set back to zero each time a node is restarted.

I think the problem with Bitcoin Cash is that the blocks are highly variable in size.  If the current blk file doesn't have enough to store a large block, the client will move on to the next blk index, leaving a large amount of free space.  

The next time the node is started, a few 50kB blocks will probably fit in spaces like that.  It highlights the issue, but I think it could happen on the Bitcoin Core client too.

I think this is not compatible with Armory?  It assumes that the blk files are filled in order and there are no movements backwards.

A solution would be to keep a record of "last-checked time" and then re-scan any old blk files that have a modified time greater or equal to the last-checked time.  This check could be run any time a new block notification is received.  It should only be a few blk files at any one time.

Even better would be to record how much of each file was already processed, but that is probably to much hassle.  

Would re-scanning already scanned blk files cause problems for Armory?

The workaround is to wipe the Armory database and let it re-scan everything if the block stalls but that is a sledge hammer.
legendary
Activity: 1232
Merit: 1094
November 13, 2018, 05:00:46 PM
Honestly I'm not sure who came up with the idea of HD wallets first.

It looks like the original idea came from Greg Maxwell, but Armory was the first implementation to run with it.

It is also possible that it was simultaneously invented / repeatedly invented by multiple people.
legendary
Activity: 3430
Merit: 3080
November 10, 2018, 08:51:15 AM
I see, "nothing new under the sun"
legendary
Activity: 3794
Merit: 1375
Armory Developer
November 10, 2018, 07:13:20 AM
Right, in those days Alan Reiner/etotheipi was pretty active on bitcoin dev mailing lists and places like that, and I'm pretty sure he discussed the ideas with everyone. Was it actually Alan's idea to begin with? I can't remember that exactly.

Alan definitely just went ahead and implemented his design, whereas other wallets took a long time to follow.

Honestly I'm not sure who came up with the idea of HD wallets first. In other ecosystems that require cryptographic material management (think CAs, DNS providers), it is common practice to create one root piece of secure material offline and derive lower schedule keys from that. With this in mind, you can't argue the use of HD wallets in the Bitcoin space is actually a novel invention, as there is prior art in other spheres, even though there was the need for some engineering to suit our use case.

Therefor, with the previous paragraph in mind, while I cannot tell you with certitude who brought the idea of HD wallets to Bitcoin first, I suspect it was floating in many devs mind if not yet in the mailing list, since the practice has been common place in other industries.

The only thing I know for sure is that Armory was the first implementation of this idea in the Bitcoin ecosystem. From what I know, it was ultimately Alan's solo take on the underlying principle. Bitcoin devs with sipa in the lead improved on it to birth BIP32 about a year later (give or take).
legendary
Activity: 3430
Merit: 3080
November 10, 2018, 06:48:19 AM
Armory would probably have used BIP-32 as well, but was developed before BIP-32 was created.

Armory's wallet design was based on BIP32 in an early stage of it's evolution, I believe. But BIP32 changed after the fact, and then took a while to get adopted by wallet software.

AFAIK, Armory introduced hierarchical deterministic wallets to the ecosystem. This in turn inspired BIP32 which improved the design while standardizing the feature.

Right, in those days Alan Reiner/etotheipi was pretty active on bitcoin dev mailing lists and places like that, and I'm pretty sure he discussed the ideas with everyone. Was it actually Alan's idea to begin with? I can't remember that exactly.

Alan definitely just went ahead and implemented his design, whereas other wallets took a long time to follow.
legendary
Activity: 3794
Merit: 1375
Armory Developer
November 09, 2018, 10:05:08 PM
Armory would probably have used BIP-32 as well, but was developed before BIP-32 was created.

Armory's wallet design was based on BIP32 in an early stage of it's evolution, I believe. But BIP32 changed after the fact, and then took a while to get adopted by wallet software.

AFAIK, Armory introduced hierarchical deterministic wallets to the ecosystem. This in turn inspired BIP32 which improved the design while standardizing the feature.
legendary
Activity: 3430
Merit: 3080
November 09, 2018, 02:06:02 PM
Armory would probably have used BIP-32 as well, but was developed before BIP-32 was created.

Armory's wallet design was based on BIP32 in an early stage of it's evolution, I believe. But BIP32 changed after the fact, and then took a while to get adopted by wallet software.
legendary
Activity: 1232
Merit: 1094
November 09, 2018, 10:04:53 AM
Armory "Root Keys" are proprietary

I don't think the method is actually 'proprietary'.  That would mean that nobody else was allowed to use their method.

Quote
and, as far as I am aware, only work with Armory...

Right, but that is only because nobody uses the Armory method.  They could if they wanted.

Instead they use the standard (BIP-32) hierarchical key generation method.

Armory would probably have used BIP-32 as well, but was developed before BIP-32 was created.
HCP
legendary
Activity: 2086
Merit: 4363
November 06, 2018, 04:17:48 PM
so my question is, since I want to get my BCH from that prefork address, and I dont want to be forced to synchronize bitcoin core, -
- can I use the Root Key (from my Paper Backup for Armory Wallet that I printed out initially)
in some application/program to recover my private key?

Or I do HAVE to install armory wallet and recover it there?
You HAVE to install Armory.

Armory "Root Keys" are proprietary and, as far as I am aware, only work with Armory... there are no other applications that use or can read this format to regenerate a wallet/keys/addresses. However, as mentioned above, you can install Armory (without Bitcoin Core installed and/or synced) and export the private keys... refer here, where the answer to your previous question regarding this was answered Wink : https://bitcointalksearch.org/topic/m.34442580

jr. member
Activity: 33
Merit: 1
November 06, 2018, 09:43:25 AM
Question:
- can I use the Root Key (from my Paper Backup for Armory Wallet that I printed out initially)
in some application/program to recover my private key?

You can export keys in offline mode, no need to sync with the network for that.
newbie
Activity: 10
Merit: 0
October 25, 2018, 12:11:20 PM
Question:

I have a back up seed for my armory wallet, there is no BTC in it, but there is some Bitcoin Cash there. (prefork wallet, and I moved my BTC from there last december).
I have not updated bitcoin core nor my armory wallet since there.
Now armory doesnt event start.

so my question is, since I want to get my BCH from that prefork address, and I dont want to be forced to synchronize bitcoin core, -
- can I use the Root Key (from my Paper Backup for Armory Wallet that I printed out initially)
in some application/program to recover my private key?

Or I do HAVE to install armory wallet and recover it there?

thanks in advance.
legendary
Activity: 3794
Merit: 1375
Armory Developer
June 26, 2018, 10:25:04 PM
Thanks, good to know.

Quote
It looks like Bitcoin Cash clients have 2 magic patterns

What an ugly hack. I don't think wiping the DB is necessary in that light, but I'd say it's preferable (unless you don't have the BCH chain and are just using the BTC one). I'm considering adding an option to stop DB scanning at a user defined height (to scan up to pre fork height without the need to mess with blockchain data).
legendary
Activity: 1232
Merit: 1094
June 26, 2018, 07:45:08 PM
Edit, that change alone isn't enough, armory won't connect to bitcoinABC.
Check your node log, should tell you why Armory is getting rejected.

This is the error message.

Code:
2018-06-26 23:51:31 PROCESSMESSAGE: INVALID MESSAGESTART version peer=127.0.0.1:56606 (53)

The endianess needs to be swapped.  When expressed as an uint32_t, the magic value for Bitcoin Cash is 0xe8f3e1e3 rather than 0xe3e1f3e8.

The updated version of your suggestion is as follows.

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BlockUtils.cpp#L886

Code:
         networkNode_ = make_shared("127.0.0.1", config_.btcPort_,
            *(uint32_t*)config_.magicBytes_.getPtr());

Should be changed to:

Code:
         networkNode_ = make_shared("127.0.0.1", config_.btcPort_,
            0xe8f3e1e3);

This connects properly which I tried it.  I also wiped the contents of the Armory db.  I don't know if that was required.

It looks like Bitcoin Cash clients have 2 magic patterns, one for the network and one for disk storage.  This means that even without any changes, Armory will sync with the blocks stored on the disk.  It just doesn't do online updates (or broadcasts), since it can still read the files on the disk.
jr. member
Activity: 33
Merit: 1
June 04, 2018, 06:49:45 PM

So if I understand correctly, starting may 15th, you can't use armory on the BCH chain anymore?



Entirely depends on the span of the new HF. If it's just a block size increase and new opcodes, Armory should still "work" provided the network magic word part is fixed.

Honestly, it would probably take me a single day to deal with this stuff assuming all they did was to increase the block size. I don't do it because I refuse to give up development time for BCH.

Got it, thank you for your hard work! Smiley

It works if you just remove some checks for the network magic, those was only added in the last update, peers for other chains will still be banned so it could only lead initial to some more wrong connections from and to your node.

See:
https://bitcointalksearch.org/topic/m.37065819

For now I think its the best solution, its not the optimal solution but that way you could still use armory and also update armory without needing to edit or add own code.
You just need to make sure you remove the checks also in future updates on Bitcoin Cash. ( I assume the checks wont be updated much so should be easy to find and remove it in future versions as well ).

Thanks, the problem is, as stupid as it may sound, that I don't really know how to download the code and build it with the changes, it will probably take me 2-3 days to do just that.
I would have done that don't get me wrong but I think that this is a bad practice.
goatpig don't want to support BCH which is his right and I respect it.
So now for each new version of both armory and BCH you will have to find more and more workarounds to make it work which is really a bad practice.
Armory is for BTC.
I think it is best to just find another wallet for BCH. Any Ideas?


I don't have much time atm else I would create a patch for either BitcoinABC or Armory.
I'm pretty sure that the BitcoinABC devs would include the patch if it only allows Armory to connect without the magic check.
Best practice would be probably if someone with time would fork armory and create and maintain a BCH version.
I don't expect or demand that Goatpig keeps supporting it.
It's also unlikely that the code for the magic checks will be changed in the future as it does what it should do, so on updates you probably only need to update the same file and comment out the same checks.

I know its not the best practice but its pretty quick and works for now.
Pages:
Jump to: