Topic: Can't dl blockchain - 'fatal internal error' then segfault (Read 2862 times)

Depends on which "ratio" are you asking. You may be right about ratio of people with problems to people scamming. But if you measure ratio by posts not by people, then the scammers are majority, especially in the mining forums.

Some of them possibly even start without evil intentions. But with time they learn that scamming here is easier than shooting fish in the barrel, and it becomes too hard to resist.

I'm kinda disappointed that you've regained your senses and decided to test hardware. Just less than a week ago I had interesting experience of watching similar scam (but one the corporate contract level and not related to Bitcoin) unfold personally in front of my eyes. They key hook was to convince person sitting in front of the computer with 3" floppy drive that there is no working drive there and they need to download the test program instead running one off of the 3" floppy. I learned a lot watching and listening that unfold.

Could be hard drive fail. Get a program to test it and also find out waht mode the hard drive is in sounds like read error. D oyou get any clicking from it? Or find that when you loading wallet everything slows down to a point cant use anything else?

2 ideas you could try to resolve this to keep using on current hardware if nothing else is is broke only bitcoin.

1 Delete everything in app data folder but backup and keep your wallet safe.  Now download blockchain using torrent and then once it is downloaded drop te .dat file into bitcoin folder in app data and allow it to index all of it and see how or where it gets to. If you get an error doing this could more likely be part read wright error no the drive that is the cause to it.

2 if you needing access to your wallets you could if it will allow you put wallet.dat back into bitcoin folder and dump prive keys on addresses you need access to and use bither I have started to use this wallet since it is lightweight and fast and simple to import wallets directly in from pri key. Am slowly moving away from qt due to the fact of amount of data it uses and should just be like bither light weight effective and does not kill drive space and takes time to load up. bitcoin wallet need to get their act together if they want to go global. Think of it like this if  it went mainstream and a trillion transactions where done in a day or more. How big would the wallet get to use on system and peoples devices. They need to adapt features like in bither or using blockchain wallets.

Hope you manage to resolve your problem or have some usability in accessing your addresses and wallet

2112: Now that's a MUCH better response!

Yes, it looks like hardware faults is where it's at. I will try a new hard drive (actually have one lying around) and test the memory.

Re scams: I'm sure you're right about their existence. I don't read that many threads. But their frequency? Surely the ratio of genuine tech problems to mutual-scammer-backscratching must be quite high here? We're not talking about my granny trying to read MS Word documents in Firefox here.

And I may not be that tech savvy, but TeamViewer... come on, dude.  

Knightdk: will do.

Thanks all.

It is a completely independent process to connect to nodes that begins before the whole blockchain verification at startup. This runs in another thread independent of the state of the rest of the program.

Looked at the code. It looks like it is hard coded to start at the beginning whenever it is reindexing other than when it is reindexing while syncing.

Before you do that, as 2112 said, run some diagnostic checks on your memory and hard drive. There should be an option to do this when your computer first starts up.

Look, this situation is really easy to read, it happened multiple times on this forum.

1) If you are really an inexperienced, frustrated computer user: just run the hardware tests for your machine, memory and disk faults are the most common root source of the problems you're seeing.

2) If your are just another gang of scammers trying to build mutual reputation of fixing the tough computer problems so they can then offer remote assistance using Teamviewer to the fresh marks: this isn't new either, it has been done multiple times already.

This is what makes bitcointalk.org such an anthropologically fascinating place: trying to figure out if this is a new scam or new convolution of old scams.

Knightdk: my surprise at seeing that it still cares about 'misbehaving nodes' is due to the fact that it ISN'T downloading any new blocks at all, it has only been reindexing the ones it already has. THAT behaviour makes perfect sense to me - presumably it would have no way to validate newly downloaded blocks until it has validated, reindexed and built a chainstate for all the existing blocks. So I can't blame it for not downloading new blocks, I don't see why it *should*. But then, *given* that it's not doing that, I cannot see what use there would be in connecting to peers at all.

However I don't suppose that's all terribly important. The crucial issue is the apparently corrupted block 246159. I'd just delete the last blk00xxx.dat file, but it seems that rather than reindexing from somewhere fairly recent, it always starts reindexing from the very beginning and ignoring all those other files (even when I *don't* delete them - I've tried it that way, too).

I've also tried removing everything and redownloading the blockchain from scratch...

...AND removing and reinstalling bitcoin-qt. I think that, notwithstanding how I stuck up for my computer hardware, I may have to suck it up and get a new hard drive and put it on there.

For some reason, this block, block 246159 (according to blockchain.info) is failing validation. Can't really say why. My guess would be that somehow it got corrupted in that file and it is missing some part of the block so the validation fails. This is probably where it gets hung.

Bitcoin Core is still connected to its peers. Each peer receives a banscore based on whether the node thinks it is misbehaving. After the score reaches a certain threshold (default 100), the node should disconncect from the node. It cares whether the node is misbehaving since the misbehaviour involves spam and can eat up your bandwidth.

It should still be downloading blocks, just not ones that it already has. The node is constantly receiving and sending data, but it should not be redownloading those blocks, only downloading blocks that it still does not have.

It should, but I don't know if it actually does (probably not).

Your best bet to fix this would be to just let it completely redownload and reindex the entire blockchain. I know that can be frustrating and time consuming, but it should fix the issues. If that doesn't work, try reinstalling Bitcoin Core.

At least I'm being useful to someone, unlike most people in sig campaigns.

What is at work here is an attempt to learn why my bitcoin-qt client appears not to function correctly, and in doing so, learn more about what exactly it is doing under the hood. If I snark a little sometimes it is because I am frustrated. I'm sure you can figure out who in this thread is helping and who is... well, not.

But please *do* tell me more about my "hidden ulterior motive". I just love hearing strangers explaining mine to me. You can also explain how I am "the perfect mark for an enterprising scammer". While you're at it, have another swipe at knightdk for no discernible reason.

knightdk is just working on his required activity for the signature campaign.

What you are doing is just confirming for everyone that you have no idea what you are doing with your machine and you are the perfect mark for an enterprising scammer.

What I'm doing here is trying to discern if this is a natural, genuine stupidity at work or if there was some better hidden ulterior motive. We'll find out in due time.

I doubt my hardware is broken if absolutely every program on it apart from bitcoin-qt works fine.

My "I know what computers can and can't do" remark was a flippant way of acknowledging that if I see a computer going through an endless loop it is pointless to expect it to change.

I am torturing myself, but no one else, you don't have to read this! Well, maybe I'm torturing knightdk but it seems to be consensual! (And I must repeat I am grateful to knightdk for their help so far)

Your hardware is broken. Get another, correctly working, machine and all your problems will disappear. Why you keep torturing yourself and us if you know how the computers work?

Thanks for your reply, obviously I've just been spending a bit too much time seething and updating my previous post! I will check out the developer-docs link you gave, and in the meantime delete debug.log, and restart bitcoin-qt, probably I'll point it at grue's node.

I suppose if you really want to get technical and if you can read code, you could check out the developer docs here: https://dev.visucore.com/bitcoin/doxygen/. These are useful as you can see paths of which method calls what. Then, if you can read code, you can look at the source code provided on the site and try to find how everything works. There are some explanations as these are developer docs, but not very many.

I'm not sure why this would happen. Try looking in the debug.log and see if it is giving you any error messages. Also, if you hover the mouse over the progress bar, it should say on what block it is on. If you do it again a minute later and the block number hasn't changed, it likely has hung on something, and you should restart it.

I already read that wiki entry and didn't find it too enlightening. But I shall keep looking for the grittier technical details.

Anyway on to the next question in my series, "How to Eventually Reverse Engineer the Complete Bitcoin-Qt Code by Asking About Every Damn Thing that is Bugging Me"...

Up until blk00070.dat, it has been rebuilding the database, and timestamping the blockfiles, at a rate of about one blockfile every two or three minutes. BUT, for the last 40 minutes or so, nothing. Why not? The blockfiles up to 00145 are all sitting there, waiting hungrily to be reindexed. What's the holdup? The "Synchronising with the network" progress bar has also halted.

I'll give it another half hour and then close and restart bitcoin-qt.

ETA: EEEEEK! I've just looked at debug.log to see what it's doing. So... 70MB of cruft in there. Now some of it is, I understand, perfectly OK. Fr'instance stuff like this:

2015-06-06 21:06:52 UpdateTip: new best=0000000000000103b325ddcc16512a6754e710c387d069a17e217169f293447c  height=233167  log2_work=69.868797  tx=16765534  date=2013-04-26 01:02:19 progress=0.102582  cache=109732

And this:

2015-06-06 21:06:52 Pre-allocating up to position 0xa00000 in rev00056.dat

... though even then I must question why it's storing so damn MUCH of it. From block 226924 to block 246158! I know debug.log is supposed to be 'flushed' at intervals but couldn't we make those intervals a little more... timely?

Anyway I don't mind, but THEN I see this:

2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (0 -> 100) BAN THRESHOLD EXCEEDED
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (100 -> 200)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch
2015-06-06 21:38:50 Misbehaving: 71.93.44.219:8333 (200 -> 300)
2015-06-06 21:38:50 ERROR: ConnectTip() : ConnectBlock 000000000000004bbd569ecddf773e8f2d92345fb7027ead5c89221196cf9ec3 failed
2015-06-06 21:38:50 ERROR: CheckBlock() : hashMerkleRoot mismatch

... and so on, for the vast bulk of the file, with the timestamp up front increasing by the seconds... for about 40 minutes until I stop it!

Okay... WHY???

1) Why is there a hashMerkleRoot mismatch?
2) Why is 71.93.44.219 misbehaving?
3) Why should bitcoin-qt CARE about some node misbehaving when it is allegedly NOT downloading blocks, but simply rebuilding the database on the blocks it has?
4) Obvious follow-on... is it, in fact, despite what you said, in *fact* redownloading those blocks after all?
5) After the first 10000 times* of printing this little set of error messages, should not the client somehow grasp that what it is doing is NOT WORKING and, just perhaps, try another node? Or even give up and say "There is something wrong, I'm stopping now."?

*obvious snark - I know what computers can and cannot do. I suppose my snark is really directed at all these coders who must somehow have never seen this behaviour before and never thought to put a little 'break' in that loop.

It should say "reindexing blocks on disk..." when it is reindexing. However, yours doesn't. Perhaps it is because you haven't downloaded to entire blockchain yet. Since it indexes the blocks as it downloads, I think that since you don't have the full blockchain on the disk yet, it will both reindex and download the blocks, and say "Synchronizing with network..."

You can read some more about the data directory on the Bitcoin Wiki here: https://en.bitcoin.it/wiki/Data_directory

Different issue, bc.i is a service that currently has problems, OP is talking about THE blockchain.

I have no idea about it . I am using blockchain.info as my wallet and now its saying "Chain Head Not Found" . Can anyone tell me about it please ?

Okay thanks that makes it somewhat clearer. In that case I shall just let it rescan to its heart's content.

ETA: actually I'll ask while I'm here: what's the difference, if any, between the process you just described, and the process of 'reindexing' that you can force it to do with the '--reindex' option? I ask because I've just started it up again and the status bar at the bottom says "Synchronizing with network..." which is what made me worry it's redownloading blocks.

I shall scoot along to the wiki to see if I can get to grips with what all the different files are trying to do... I understand the blk00xxx.dat files themselves, but I don't yet understand the rev00xxx.dat files, the index/000xxx.ldb files (and a .log file in with them too), and the stuff in the chainstate dir. Anyway ta for the pointers.

Therein lies the problem. the blk files are just the raw data from the network dumped to the disk. What Bitcoin Core actually uses for everything are database files. These are found in blocks/index which contains data on where to find blocks, and the chainstate directory in the datadir. The chainstate directory is the most important regarding the blockchain. From the Bitcoin Wiki:
By deleting this directory, Bitcoin Core must rescan the entire blockchain from the blk files in order to rebuild these databases. It is not redownloading them either. The different timestamps that you see is because Bitcoin Core is reading through every single blk file and rescanning them to rebuild the database.

Okay I'm at my wit's end here.

Please, if somebody could talk me through this I'd be grateful. Apologies for the vitriol in this post. Please be assured it is directed at the general nature of computers, not at you.

Let's be clear, I'm NOT expecting anybody to get me to a fully synced up chain here. My goals are far, FAR more modest.

First off: suppose I have deleted EVERYTHING in the .bitcoin folder with the SOLE exception of the blocks/blk00xxx.dat files, of which I have the first 146. That's some 16GB of data that I'm really, really hoping bitcoin-qt isn't TOO STUPID TO SEE. No, Mr. Bitcoin, you do NOT need to start downloading the blocks all over again, they are RIGHT FUCKING HERE.
How do I translate that previous sentence into something Mr. Bitcoin can understand?

The problem is I started it again and thought well I'll let it dl them for a bit, I now have blk00000 and 00001.dat that are timestamped AFTER the blk00002-00146.dat.
Surely that can't be a problem? Surely Mr. Bitcoin can't decide he's going to ignore the subsequent block files because of their timestamp? Please tell me the code can't be designed THAT badly?