Author

Topic: Armory closing at 99% scanning transaction history (Read 7574 times)

legendary
Activity: 3430
Merit: 3080
Our priority at Armory is security. We do understand the need for better user experience but not at the cost of our utmost priority. If you reread my post you'll see we have a long term solution, but it will come in due time, because other features have to get there first to allow this solution to implemented. It isn't stand alone solution that we could implement over night or we would have done so obviously.

You're talking about using paper wallets, but they too are inconvenient, until the point when BIP32 is widely implemented. There are also a lot more clients out there that choose user experience over security. We have a role to fulfill before we can get the user experience up to acceptable levels. By far Armory is not considered the easiest nor the most welcoming wallet, but some people think this is an acceptable trade off for the added features, and here we are today. What you are trully experiencing is a software niche that hasn't just matured yet.

Obviously we would like to get rid of all these quirks. As a matter of fact my work for the past 4 months has been towards implementing long term scalability and stability fixes. The problem you describe cannot be dubbed as a "leveldb" error. This is a symptom of your local copy of the blockchain. Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

Let's put aside the consequence that has on the network, and focus on Armory. Armory requires a full node to function. Implementing a long term solution to this issue would require a shift in paradigm. That can't be addressed lightly nor quickly.

Also consider that most of the codebase is still etotheipi's work: about 100k lines of code over 2 years. He had to make some choices to ensure Armory functions overall. You can't just spend 6 month to fix scanning issues that can be effectively decentralized. Consider that the earlier versions of Armory didn't even have a DB: you had to fully scan the blockchain at every load. Now that 5 developers work full time on Armory, we have the resources and talent required to implement long lasting solutions to all the bugs and quirks, while adding new features.

Quote
Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory


You can copy the signed Tx and broadcast it with another service, sure.

sorry to revive an old post but iv'e been using armory for two years now and goatpig your reply to the same bug im having now in 08/2015 proves to me this program is unusable, unstable and not worth the headache anymore.  i know its free to use and probably bleeding edge technology...i understand this but jesus man...this program makes me want to slash my wrists in frustration.

i'm sitting here trying to come to grips with the inevitable reality that i'm probably going to have to re-download the entire blockchain for the fifth time since iv'e had armory installed just to verify my cold storage funds even still exist.

Iv'e mainly used armory for cold storage but i would estimate...in the few dozen or so times iv'e actually had to use armory over the past two years it has failed to work for probably 1/3 to 1/2 of those times and has resulted in probably a couple hundred hours of my time trying to get your product to work properly. i cant even imagine people who try and use it on a day to day basis.

there are very...very few products i truly hate and i'm sorry but my frustration has reached such a high level i honestly and sadly admit i truly hate armory. i just cant wait to get it working one final time just so i can get my coins out of cold storage.

goatpig you mention security as such a priority and i agree it is....but my user experience has been so annoying with armory i realy dont care anymore. i would rather risk putting my coins in a less secure form of cold storage having quick access to them, even at the risk of losing them, rather than go through one more attempt at having to download 50 gigs of blockchain.

im sorry i dont mean to outright bash armory without being constructive but my patience (and i am a very patient person) is completely gone. i wish you luck in the future but i will leave that future to newer users or the few lucky ones who have managed to dodge the worst of armory's bugs...


Are you using a Mac or Windows?
newbie
Activity: 24
Merit: 0
Our priority at Armory is security. We do understand the need for better user experience but not at the cost of our utmost priority. If you reread my post you'll see we have a long term solution, but it will come in due time, because other features have to get there first to allow this solution to implemented. It isn't stand alone solution that we could implement over night or we would have done so obviously.

You're talking about using paper wallets, but they too are inconvenient, until the point when BIP32 is widely implemented. There are also a lot more clients out there that choose user experience over security. We have a role to fulfill before we can get the user experience up to acceptable levels. By far Armory is not considered the easiest nor the most welcoming wallet, but some people think this is an acceptable trade off for the added features, and here we are today. What you are trully experiencing is a software niche that hasn't just matured yet.

Obviously we would like to get rid of all these quirks. As a matter of fact my work for the past 4 months has been towards implementing long term scalability and stability fixes. The problem you describe cannot be dubbed as a "leveldb" error. This is a symptom of your local copy of the blockchain. Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

Let's put aside the consequence that has on the network, and focus on Armory. Armory requires a full node to function. Implementing a long term solution to this issue would require a shift in paradigm. That can't be addressed lightly nor quickly.

Also consider that most of the codebase is still etotheipi's work: about 100k lines of code over 2 years. He had to make some choices to ensure Armory functions overall. You can't just spend 6 month to fix scanning issues that can be effectively decentralized. Consider that the earlier versions of Armory didn't even have a DB: you had to fully scan the blockchain at every load. Now that 5 developers work full time on Armory, we have the resources and talent required to implement long lasting solutions to all the bugs and quirks, while adding new features.

Quote
Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory


You can copy the signed Tx and broadcast it with another service, sure.

sorry to revive an old post but iv'e been using armory for two years now and goatpig your reply to the same bug im having now in 08/2015 proves to me this program is unusable, unstable and not worth the headache anymore.  i know its free to use and probably bleeding edge technology...i understand this but jesus man...this program makes me want to slash my wrists in frustration.

i'm sitting here trying to come to grips with the inevitable reality that i'm probably going to have to re-download the entire blockchain for the fifth time since iv'e had armory installed just to verify my cold storage funds even still exist.

Iv'e mainly used armory for cold storage but i would estimate...in the few dozen or so times iv'e actually had to use armory over the past two years it has failed to work for probably 1/3 to 1/2 of those times and has resulted in probably a couple hundred hours of my time trying to get your product to work properly. i cant even imagine people who try and use it on a day to day basis.

there are very...very few products i truly hate and i'm sorry but my frustration has reached such a high level i honestly and sadly admit i truly hate armory. i just cant wait to get it working one final time just so i can get my coins out of cold storage.

goatpig you mention security as such a priority and i agree it is....but my user experience has been so annoying with armory i realy dont care anymore. i would rather risk putting my coins in a less secure form of cold storage having quick access to them, even at the risk of losing them, rather than go through one more attempt at having to download 50 gigs of blockchain.

im sorry i dont mean to outright bash armory without being constructive but my patience (and i am a very patient person) is completely gone. i wish you luck in the future but i will leave that future to newer users or the few lucky ones who have managed to dodge the worst of armory's bugs...
hero member
Activity: 938
Merit: 502
Hiya, I'm having the same issue with 0.92 trying to build the DBs and ending up with the C++ Runtime process crapping out at ever-increasing-but-never-quite-completing percents of the blockchain data.  Is there any process you could run that could verify the bootstrap chain against a p2p-downloaded version of the blockset via the core client?  I don't have a data limit from my ISP and this is kind of a side endeavor to try out multisig, so I'll try and fiddle around with my blockchain copy and see what I can manage.

And also, any updates on 0.93?  How close in the implementation and release process are you guys to being out of beta?  Keep up the good work - from what I've heard, Armory is awesome, so keep up the good work.
legendary
Activity: 3766
Merit: 1364
Armory Developer
Our priority at Armory is security

Security is not a product of just having no known exploits. Sledgehammer has no exploits, but it's ridiculously cumbersome to store bitcoins in it. If your product is as good at being a wallet as sledgehammer, you have a problem. Usability has as much to do with security as access control. When using your product ends in denial of service, it's security failure.

My quote also implies that we can't just implement a random fix and call it a day. It needs to integrate with the model properly and be thoroughly tested. There has been quite an effort deployed to overhaul the backend, which should hit the next release. It will offer a lot more stability, speed and scalability. Scalability we can leverage to implement other sources for the blockchain data and hybrid modes, in a way we think is robust.

The DB version of Armory was introduced in 0.90, and it should be overhauled for 0.93. You are using a proof of concept version and will get to enjoy the rock solid implementation in 0.93. This is the speed at which we deploy releases, and keep in mind 0.90 was essentially a single man effort. Generally the full blockchain approach was the cheapest solution to implement, and  these kind of choices are easy to make when you are putting a proof of concept together. 0.91 was mainly a 3 job, and the whole team of 5 worked on 0.92. Now that we are functioning at full capacity, a lot is getting improved, and much faster.

Also, before we go after this particular issue you suffered from, we had to address the scalability issue that affected a lot more of our users. In contrast, your issue is actually fixable at the cost of a fresh blockchain download, which we did speed up with an integrated bootstrap downloader in 0.91. Keep in mind we identified this issue while deploying 0.90 test releases. This is isn't ideal of course, just side stepping the issue, but cheap enough that we offered this solution while working on something a lot stronger.
newbie
Activity: 3
Merit: 0
Our priority at Armory is security

Security is not a product of just having no known exploits. Sledgehammer has no exploits, but it's ridiculously cumbersome to store bitcoins in it. If your product is as good at being a wallet as sledgehammer, you have a problem. Usability has as much to do with security as access control. When using your product ends in denial of service, it's security failure.
legendary
Activity: 3766
Merit: 1364
Armory Developer
We ignore the bad data. The issue occurs when the header is present, but the block data is missing or damaged. By this I mean there is a single instance of the incomplete block data, or that it is outright missing. There are a few instances of damaged blocks that have a second, proper copy available further down the blk file, but there are also cases where the only instance of a given block data is damaged.

I dont think Bitcoin Core verifies more than the headers.
hero member
Activity: 547
Merit: 500
Decor in numeris
Quote
Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

That does not sound right.  Bitcoin-QT / bitcoind should be verifying its own data, so if block data is missing that verification should fail.  Is the problem not rather that there is extra (defective) data in the local copy of the blockchain which is being ignored by bitcoin-QT / bitcoind but not by Armory?
legendary
Activity: 3766
Merit: 1364
Armory Developer
Our priority at Armory is security. We do understand the need for better user experience but not at the cost of our utmost priority. If you reread my post you'll see we have a long term solution, but it will come in due time, because other features have to get there first to allow this solution to implemented. It isn't stand alone solution that we could implement over night or we would have done so obviously.

You're talking about using paper wallets, but they too are inconvenient, until the point when BIP32 is widely implemented. There are also a lot more clients out there that choose user experience over security. We have a role to fulfill before we can get the user experience up to acceptable levels. By far Armory is not considered the easiest nor the most welcoming wallet, but some people think this is an acceptable trade off for the added features, and here we are today. What you are trully experiencing is a software niche that hasn't just matured yet.

Obviously we would like to get rid of all these quirks. As a matter of fact my work for the past 4 months has been towards implementing long term scalability and stability fixes. The problem you describe cannot be dubbed as a "leveldb" error. This is a symptom of your local copy of the blockchain. Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

Let's put aside the consequence that has on the network, and focus on Armory. Armory requires a full node to function. Implementing a long term solution to this issue would require a shift in paradigm. That can't be addressed lightly nor quickly.

Also consider that most of the codebase is still etotheipi's work: about 100k lines of code over 2 years. He had to make some choices to ensure Armory functions overall. You can't just spend 6 month to fix scanning issues that can be effectively decentralized. Consider that the earlier versions of Armory didn't even have a DB: you had to fully scan the blockchain at every load. Now that 5 developers work full time on Armory, we have the resources and talent required to implement long lasting solutions to all the bugs and quirks, while adding new features.

Quote
Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory


You can copy the signed Tx and broadcast it with another service, sure.
newbie
Activity: 3
Merit: 0
Bitcoin Core checks for blocks THEN writes them to disk, and doesn't bother to verify they were written correctly, rather relying on its LevelDB database. On some occasions, that results in botched blocks. And yes, Bitcoin Core will fail to read these botched blocks if asked to.

Armory depends on the raw files for its blockchain data, this is when those botched blocks get in the way. You also can't access Bitcoin Core's DB since LevelDB is set to lock it at process level.

We could hack around that but that's too demanding, will impact stability, and reflects an underlying issues with your copy of the blockchain, which frankly you should fix. The long term solution which to be introduce after supernode and the new wallet format will be blocks over P2P.

See, this is your problem right here. You see it as a reason, but it's not - it is an excuse. The botched blocks are a problem, but you are not looking for solution, so you are not finding one. Core client works with these blocks just fine. I don't hear core devs saying "oh, we got a leveldb problem, here's a great idea - lets make you reload your whole blockchain"

Think of it in perspective: resyncing blockchain requires several days. Many of us have lives to live, and not babysit applications. Also, for some of us bitcoin is not an academic proof-of-concept, but a tool, meaning: sometimes you need to pay someone, not in several days when you get a chance to fix leveldb, but right now. Is it usable? How about that it also will result in 30-50gb of internet traffic? You have your funds stored in armory wallet, and this problem happens every couple months. Is that usable? Not on day-to-day basis. Many cable internet customers in US are limited to 200-400gb of traffic per month. So, for some of us, fixing this re-occurring problem requires spending a quarter of monthly internet limit. Is it usable? Maybe for those of us who were not thinking and put their funds in armory hoping this someday gets fixed. What is the value of armory after all these issues, why would anybody be even using it? You still need to deal with maintaining core, this doesn't change. But in addition to this pin in the ass, now you also must maintain it in a way that allows armory to swallow its data. Security? I would rather be using paper wallet, at least I can import it into another client when I need access to funds. Offline transactions? I concur, this is a unique feature, but is it worth going through all this hassle to just use this one feature? I strongly doubt so. Don't get me wrong - i tried using armory, I tried hard, and in the end it's just not worth it.

The bottom line here is: if your application is unusable, then it will not be used. You need to find a way to make it working without regular glitches, and preferably without requirements for major downloads, or it will wither.

Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory
legendary
Activity: 3766
Merit: 1364
Armory Developer
Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it

Why doesn't this worst case scenario excite me at all? Thing is like 30 gig.

I have to say that this bug has been there from the very beginning and somehow it's still not mitigated in any way. Reference bitcoin client has no issues with it, but somehow armory cannot handle it. Why? I call it bs.

It's very sad how otherwise functional client suffers from such usability problems.

Bitcoin Core checks for blocks THEN writes them to disk, and doesn't bother to verify they were written correctly, rather relying on its LevelDB database. On some occasions, that results in botched blocks. And yes, Bitcoin Core will fail to read these botched blocks if asked to.

Armory depends on the raw files for its blockchain data, this is when those botched blocks get in the way. You also can't access Bitcoin Core's DB since LevelDB is set to lock it at process level.

We could hack around that but that's too demanding, will impact stability, and reflects an underlying issues with your copy of the blockchain, which frankly you should fix. The long term solution which to be introduce after supernode and the new wallet format will be blocks over P2P.

A big thanks to Armory for helping to find this bad RAM! If it weren't for all of the double and triple checking in the Armory code, this could have gone undetected for months, and who knows what I could have lost!! Thanks!!!

Well, there are redundancy checks in place for the crypto data, and more on their way with the new wallet format, so while unstable hardware will give you a bad experience, it won't outright lose you coins.
hero member
Activity: 672
Merit: 504
a.k.a. gurnec on GitHub
TL;DR: Please try running Memtest86+, this worked for me!

About two or three weeks ago, I was having the worst problems getting past the Sync step. I spent probably about a week or two re-downloading the Bitcoin blockchain (more than once), rebuilding the Armory DB (several times), and rescanning, and it would always fail during the scanning phase with that same "Cannot get tx copy, because don't have full StoredTx!" error often at different points (at different %'s completed).

I became convinced there must be a bug somewhere in Armory. On one occasion in the debug log, it mentioned an additional error message "Invalid txIndex at height # index #". This gave me a place to start looking in the LevelDB database to see if there actually was a corruption. After quite a bit of digging, I did indeed find the issue: in the DB, each transaction in a block is given an ID starting at 0 (that's not the blockchain TxID). There was a transaction in the block whose sequential ID number was greater than the total number of transactions in that block (which is recorded separately in the DB). But how did this happen??

Looking at the transactions that should have been in this block, there was also one missing, so it looked like the corrupted transaction matched up with the missing transaction except for it's sequential ID. Looking closer, the ID was almost correct: it had just a single-bit error in its high nibble, flipping a 0 to a 1 and making it too large.

Well that's strange... maybe my hard drive was failing? Usually HD failures doen't cause single-bit errors, but rather entirely unreadable sectors or relatively large corruptions though.... Maybe a failing RAM stick?

Well, very long story short, I did indeed have some bad RAM which Memtest86+ found pretty quickly. After replacement, I had no trouble rebuilding and rescanning.

A big thanks to Armory for helping to find this bad RAM! If it weren't for all of the double and triple checking in the Armory code, this could have gone undetected for months, and who knows what I could have lost!! Thanks!!!
newbie
Activity: 3
Merit: 0
Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it

Why doesn't this worst case scenario excite me at all? Thing is like 30 gig.

I have to say that this bug has been there from the very beginning and somehow it's still not mitigated in any way. Reference bitcoin client has no issues with it, but somehow armory cannot handle it. Why? I call it bs.

It's very sad how otherwise functional client suffers from such usability problems.
hero member
Activity: 547
Merit: 500
Decor in numeris
I just had this issue, I went to Help -> Rebuild and rescan databases, so far its working. Ill update once it's complete

Edit: Crashed again, ignore my previous advice
It may be due to a bad block in the Bitcoin-Qt/bitcoind database - the reference client seems to handle that better than Armory.  It should of course not place malformed blocks in its database, but it often does, and then ignores it.  But Armory may barf at them.

Try deleting the blocks of the original client.  If you don't want to wait ages downloading it again, google for torrents containing a recent snapshot of the blockchain, it is far faster to download.  If you do use a snapshot, you first need to start bitcoin-qt/bitcoind with a specific command line option (Google again, I cannot remember it), and let it rebuild its own database.  After this is done, you can delete the huge file you got with torrent, and then start Armory.  Delete the torrent before starting Armory, that way you never have three copies of the blockchain on the disk simultaneously, two is bad enough Smiley
legendary
Activity: 1260
Merit: 1000
World Class Cryptonaire
I just had this issue, I went to Help -> Rebuild and rescan databases, so far its working. Ill update once it's complete

Edit: Crashed again, ignore my previous advice
legendary
Activity: 1890
Merit: 1537
try deleting the DB folder in roaming/armory

worked for me
newbie
Activity: 6
Merit: 0
Google is my friend.  Thanks for the pointer in the right direction...
full member
Activity: 238
Merit: 109
Don't mind doing that.  My question, is , HOW?  I can't find any commands to do that....

%appdata%\Armory
%appdata%\Bitcoin
newbie
Activity: 6
Merit: 0
Don't mind doing that.  My question, is , HOW?  I can't find any commands to do that....
legendary
Activity: 3766
Merit: 1364
Armory Developer
-INFO  - 1392041012: (..\BlockUtils.cpp:3690) Reading all headers and building chain...
-INFO  - 1392041032: (..\BlockUtils.cpp:3695) Total number of blk*.dat files: 115
-INFO  - 1392041032: (..\BlockUtils.cpp:3696) Total number of blocks found:   285111
-INFO  - 1392041032: (..\BlockUtils.cpp:3708) Getting latest blocks from blk*.dat files
-INFO  - 1392041032: (..\BlockUtils.cpp:3709) Total blockchain bytes: 15,322,661,233
-INFO  - 1392041032: (..\BlockUtils.cpp:3715) Parsing blockchain file: C:\Bitcoin\blocks/blk00114.dat
-INFO  - 1392041032: (..\BlockUtils.cpp:3812) C:\Bitcoin\blocks/blk00114.dat is 33,554,432 bytes
-INFO  - 1392041096: (..\BlockUtils.cpp:3729) Processed 6 raw blocks DB (64 seconds)
-INFO  - 1392041099: (..\BlockUtils.cpp:3758) Starting scan from block height: 0
-ERROR - 1392042812: (..\StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!

Anyone know how to reload the "full StoredTx" ?

Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it
newbie
Activity: 6
Merit: 0
-INFO  - 1392041012: (..\BlockUtils.cpp:3690) Reading all headers and building chain...
-INFO  - 1392041032: (..\BlockUtils.cpp:3695) Total number of blk*.dat files: 115
-INFO  - 1392041032: (..\BlockUtils.cpp:3696) Total number of blocks found:   285111
-INFO  - 1392041032: (..\BlockUtils.cpp:3708) Getting latest blocks from blk*.dat files
-INFO  - 1392041032: (..\BlockUtils.cpp:3709) Total blockchain bytes: 15,322,661,233
-INFO  - 1392041032: (..\BlockUtils.cpp:3715) Parsing blockchain file: C:\Bitcoin\blocks/blk00114.dat
-INFO  - 1392041032: (..\BlockUtils.cpp:3812) C:\Bitcoin\blocks/blk00114.dat is 33,554,432 bytes
-INFO  - 1392041096: (..\BlockUtils.cpp:3729) Processed 6 raw blocks DB (64 seconds)
-INFO  - 1392041099: (..\BlockUtils.cpp:3758) Starting scan from block height: 0
-ERROR - 1392042812: (..\StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!

Anyone know how to reload the "full StoredTx" ?
member
Activity: 84
Merit: 10
.
Problem signature:
  Problem Event Name:   APPCRASH
  Application Name:   ArmoryQt.exe
  Application Version:   0.0.0.0
  Application Timestamp:   49180193
  Fault Module Name:   _CppBlockUtils.pyd
  Fault Module Version:   0.0.0.0
  Fault Module Timestamp:   5294de6a
  Exception Code:   c0000005
  Exception Offset:   00001a11
  OS Version:   6.1.7600.2.0.0.768.3
  Locale ID:   1033
  Additional Information 1:   0a9e
  Additional Information 2:   0a9e372d3b4ad19135b953a78882e789
  Additional Information 3:   0a9e
  Additional Information 4:   0a9e372d3b4ad19135b953a78882e789

member
Activity: 84
Merit: 10
.
Hey guys,

I'm having an issue I was hoping someone could help with.

For some reason... today when I try to load Armory it hits 99% scanning transaction history and then crashes.
I've tried about 10 times now and realize that I need some extra help.

Can someone please point me in the right direction?

Thanks
Jump to: