Topic: Problems syncing this morning : invalid block, please help (Read 571 times)

Just wondering, do you ever perform stability/stress testing (e.g. with Prime95 on Windows) when you undervold your CPU?


Bitcoin Core use your hardware intensely compared with most application, so it's not surprising it only happen with Bitcoin Core. IIRC Bitcoin Core has such warning message when you run it for first time.

Love the mystery stuff, oh well - that's not the initial sync tho, it's the weekly catch-ups that had those issues. i don't use it 24/7, weekly sync usually.

Anyhow just wanted to share in case others also get stuck, disable the undervolting - and yes improving cooling is also great, in fact i have a very good Klim Coolpad when necessary now.

No, i am using Intel Extreme Tuning Utility, so software, nothing hardware at all. Don't care as i only use it when needing extra juice and on BTC core i know what to do.

The weird part is that only BTC core is affected. This laptop has been a workhorse since 5 years already, poor thing hehe.

It sounds like you've created a hardware problem by undervolting. You may not notice the problems in other applications, but they could still occur.
I'd say improve your cooling instead of using hardware outside it's specifications. For your next computer, consider a processor with lower TDP if possible.

---

---

So that's why Bitcoin Core gives a warning when started for the first time:

Are you doing the undervolt though software (e.g. cpupower) or are you setting a hardware setting?

Maybe you're using a hardware undervolt and so CPU cycles are being thrown away as a result without Core realizing it's being slown down. I never had any problems with underclocking with cpupower, you should look into that (but its underclock is system-wide).

Glad that you found the source of the issue.

AFAIK, processors don't have consistent quality. Some are better some are worse, it's what they call "silicon lottery".
You may have picked the one which performs bad with the same underclock/undervolt setting as the others.

hi guys,

I wanted to give an update on this issue...since this was last updated last year in Aug 2021.

So, this invalid block thing was happening quite often until i discovered something: I was always running my CPU with an undervolt flag for better temperature under heavy loads.

For some reason, at some point i disabled that flag while running Core, and again and again and i noticed that the issue with the invalid block NEVER happened again. I didn't change anything, i didn't change SSD drive, didn't format, didn't re-download the entire blockchain, nothing.

So i am quite happy i managed to find a solution that works. I still run the undervolt for other things and some other wallets but not for BTC core.

Still a mystery the reason why though ;-)

Best,
Gabrio

hi,

Well, it's very hard to say if it's the drive because, i ONLY use this ssd for crypto really, i have nothing else on it and it's also a very fast Samsung T7 SSD (1 gb/sec write and read) with built in encryption etc (which i am not using though).

Now, yes this happened many times now and i am quite clueless on why, I have never lost data on this ssd on other occasions. I am not mining at the moment but was helping the network with open port, etc. I have 2 others privacy coins and one of those also experienced same thing, reindex then now fine, the other one instead never had this issue,

I am gonna live with this, not ideal and again, the reason i am having this on external SSD is that i wanted a drive that does not stay on continuously like my "main" normal SSD where i do other stuff and my work, etc. As we all know these drives have an X lifespan and if they're on 24/7 that will get shortened every day.

Still, that's way bizarre so i am gonna send Samsung a message, maybe they can help or maybe this type of SSD is not suitable for crypto in general, hence the errors.

thanks for ideas and feedback anyway

Well it keeps sounding like a bad disk, and no comment in the first post about that

He did mention CHKDSK 7 posts later,
but CHKDSK isn't by default a full check of any disk, and on a SSD even less so.

Make sure to backup the wallet (or that should have been done anyway) then,
at the very least run smartmon on the disk and check it's history and run a 'long' smartmon check of the disk.

Standard I/O to any disk wont cause errors no matter what you do, unless there's something wrong with the disk or the interface.
With an SSD, give it a few years at least before smartmon should start mentioning issues.

Though there are some altcoins that are VERY bad for any computer drives, so if he's 'mining' any of them, then the whole issue is simply to expect failure and don't dare run something important like a Bitcoin wallet on the same drive.

.. bad disk?

A "reindex" won't prevent this from happening... something in your setup is causing data corruption and that corrupt data is being written on your disk.

It shouldn't be an invalid block being received from another node, that should be automatically invalidated and discarded by your node when it is received... so it is most likely something at your end (bad drive cable, bad hardware, bad power unit, faulty memory, faulty disk, bad chipset etc).

I would wager that this will continue to happen from time to time unless you substantially change your setup and replace hardware (the difficult part is that it will be very difficult to track down which specific bit of hardware is causing the error )...

Until then, you're going to need to continue using reconsiderblock whenever this happens.

it happened again, used reconsiderblock and that's that - it's really strange though. oh well moving on - thought the reindex corrected the issue but apparently didn't. i cannot keep this plugged in all the time so that'll do.

in the end it works, i can operate when i need to. thanks all

In any case, this really should not be happening. It's not "normal".

I have both a desktop with a SATA HDD running Win10+Bitcoin Core, and a Pi with a USB3.0 external HDD running Umbrel, and neither exhibit this behaviour.

There seems to be something peculiar to your specific setup that is causing this, and I fear that you are going to continue to get these invalid block errors unless something in your setup (ie. hardware config) changes.

ok thanks but i figured it out myself, i did a -reindex and it finished, took 17 hours, let's see if it happens again now.

Exact same thing happened?

Yes? Then redo post number two but check your "debug.log" for the latest lines with:
ERROR: AcceptBlockHeader: block xxxxxxxxxxxxxxxxxxBLOCK_HASHxxxxxxxxxxxxxxxxxx is marked invalid
And use that block hash to the command: reconsiderblock

ok, this happened again - i will take the plunge and do a rescan / reindex most likely taking a week...... something similar happened on a smaller chain and it fixed the issue

what's the command? is that - reindex something via command like?

can anybody help on this please?

thank you,
Gabrio

I guess it is possible that one of the peers you're getting data from may, for whatever reason (unknowingly or maliciously), be sending your node bad data... You could try "banning" all the current peers and see if it continues to happen.

Otherwise, it's likely some transient hardware issue with your setup... and those are an absolute MoFo to try and diagnose/rectify

ha! i talked too soon.... while doing a sync just now, again i got the same thing, this time with only BTC core running, i guess i need to rescan o re-do the whole blockchain at some point soon.... that will need 1 week or so though... arghhh again invalid block.

this time was a bit weird, it started doing the sync, was a good point and only 369 blocks left

any tips to do a full rescan or complete blockchain sync again? of course i just ignored the block again and it worked again.

it's actually not bad. on average if you do the BTC core sync once a week, it takes only 10-15 mins to sync with my system and this SSD. 

Interesting... but I'm not surprised that Bitcoin has the highest usage. I would assume that the amount of data (note: not just the number of blocks) being processed on the Bitcoin blockchain is quite a bit larger than the other private chains.

Still, running on a Samsung SSD... even an external one, I would have thought that it should be easy enough for the system to keep up, given that I have run multiple chains on an HDD.

I'm sure that running it one at a time will stop any issues, but could be annoying depending on how long it's been since that chain was synced and how long you have to wait for it to sync up.

yeah that's right, and they all have full blockchains running - i am gonna run only one at a time, finish what i have to do, close then run the other one, etc.

The private ones actually are much more intense but bitcoin always triggers the highest CPU usage and I/O on the hard disk....this is a fast Samsung SSD T7 but then at the same time, maybe that's too much.

anyway time will tell

Sorry... just to clarify... you're running 3x cryptocoin Core applications at the same time (eg. Bitcoin Core + Litecoin Core + Dogecoin Core) with all the blockdata stored on the same external drive?

Outside of the initial block download, I wouldn't have thought that running them all side by side would stress things too much... once nodes are synced up, the processing requirements aren't super high.

I've had multiple Core applications running before... I think 5 was my "personal best" (Bitcoin mainnet, Bitcoin testnet, BCH, Litecoin and Dogecoin) and never seemed to have any issues... but that was on a desktop with all the blockdata on the same (internal) SATA HDD.

thanks guys - on the hard disk i don't completely agree with that, as in the USB port of the laptop, i think is something else, read below:

Now, what i think it could be is the fact that i am running simultaneously also 2 other crypto CORES which will be remain unknown for now so it's very well possible that sometimes slips out.

Power delivery issue i don't think so, it's plugged straight to the usb-c to the laptop and always with AC on.

I will try running only BTC core alone when i do the sync and not the 2 other chains as well - they never had this issue, though.

with that exclude command executed i can then transact, everything works normally, and if i need to do it ever 2-3 months is not a huge deal.

As i said, i don't think that's an issue with the USB port of the laptop but due to the setup i have - where i basically do my weekly sync for 3 CORE blockchains together (not always but quite often because the machine and the SSD is capable enough), that i think could lead to some minimal data corruption with the amount of I/O happening at the same time. And given that BTC is the "biggest" whale, i will let it go on his own only now.

time will tell

thanks again,
Gabrio

They aren't. If anyone were to send you an invalid block, your client would just reject it and follow the longest chain. This isn't the case because the block is valid and it is also in the longest chain. That is why you're always stuck at synchronization.

It shouldn't happen at all. This means that Core wasn't able to validate the block when it first saw it, perhaps due to it being unable to validate certain data within the block. But yeah, as I said might be your USB controller which can't be fixed unless you swap it out or use the internal drive.

The only way to know what goes wrong that rejected this block is to have the full block with 0000...09e7f7 hash as it was the first time your node saw it and see why it was rejected in first place.
The error message saying "is marked invalid" is not enough information since the hash belongs to a real bitcoin block.

There aren't rogue blocks as such... so the issue is likely data corruption due to an issue with hardware.

Given the length of time between issues... it seems like it is a transient issue, which can be a real pain in the arse to diagnose From your previous post, you're using a laptop, with an external drive is being used to store the block data. Is it a USB powered drive, or does it have a separate power adapter?

If USB powered, it's possible that you might be experiencing some sort of intermittent power delivery issue with the laptop's USB port that is causing the data corruption. Again, given that this isn't a constantly occurring issue, it will likely be very very difficult to reliably replicate this issue to try and figure out the exact conditions that cause it to happen.

For instance, it's possible that adding/removing other USB devices might do it... or the system trying to put devices to sleep or low power mode etc.

Have you tried a different USB port on your laptop? If that doesn't solve it and you continues to get these invalid blocks on a relatively regular basis, you might need to try and power the device with an external power adapter using a Y-Cable etc and see if that helps.

hello,

resurrecting a thread of a little while ago, it happened again i had to launch a "reconsiderblock" was wasn't syncing.. are these rogue blocks or something?

i mean i don't mind doing this but ideally shouldn't happen right? then picked up again and works now again. just like last time.

didnt have any unclean shutdowns, etc

thanks

nope, i am using an asus S15 Vivobook slightly pimped

Do you happen to be using an AMD ryzen 3000 or 5000 series CPU on one of the 4xx or 5xx series chipset motherboards (B450/X470/B550/X570)? There have been issues surrounding USB and AMD Ryzen 400/500 series motherboards causing random dropouts of USB connectivity...

Refer: https://www.tomshardware.com/news/amd-suggest-possible-fixes-for-usb-connectivity-issues

Yes indeed but i find it better and safer having the data stored on an external hard disk, although the occasional problem like this may occur.

Without any information about what went wrong when Bitcoin Core invalidated it, it'll be hard to pinpoint the problem especially if your hardware is doing fine.

However, I'll consider USB external drives far less reliable than SATA ones. The controllers that I've had for my drives were not exactly the best though they were fairly expensive and well known. Bitcoin Core tends to magnify the problems associated with them due to the nature of the program and how it interacts with the disk. Assuming that your data directory is on the external SSD.

i just did extended mem testing all night over 8 hours, plus basic and standard testing, all no errors.

and frankly, i never never get any blue screen here which is usually RAM, since i installed this and i have a high performance Ram.

yeah you are right, you are like Sherlock Holmes hehe looks like i had that problem before.

and no, i am on Win10 64x.

only thing i can think of is that i have BTC core on ext usb 3.1 SSD drive (at that time it was not SSD tho). this one is faster now.

I saw that you've already encountered the same issue before: Core 0.20.0 sync stuck on a block, please help, has there been a fork?
On August 2 last year, experiencing the same issue twice is a bit concerning.

Are you perhaps using Debian OS? Because there was an old closed issue, but not closed because of a PR that fixed it, but for using par=1 to use only one thread for script verification that seems to effectively fix the issue.
Issue link: /bitcoin/bitcoin/issues/2726

Note that if you want to test your RAM you would download memtest86 and make a ISO image of it to boot from.

On Windows it is even easier to test your memory, you just click on the Search magnifying glass at the bottom left and type "Windows Memory Diagnostic" or "mdsched.exe" and launch the resulting program that appears. Then you can either tell it to restart immediately or wait until the next restart to run the memory test.

Some Linux live DVDs  already have memtest86 and display a Test Memory entry in the boot menu so you can just use that instead if you have one lying around.

it's fully synced now. closed BTC.

checked disk, ran a CHKDSK no errors, ran a disk check from win10 also all good.

launched BTC core again, fully working this time instantly.

well all i can think of is that i keep BTC core on an external fast ssd, i will check that one for corruption, etc once finished

thanks again

Yup. Great that it is working for you!

Nonetheless, if you have the time, you can try some utility to test your memory, CPU and your disk and see if there are any defects in them. Blocks shouldn't be invalidated without the user specifically marking it or if there is a block corruption. It has worked once previously but it didn't for another and I didn't manage to get that user to run any tests.

more connections now - 15 and i see the 4 little arrows filled... in log i have now:

2021-04-06T10:46:15Z UpdateTip: new best=0000000000000000000b818d87315f69398aa6ba886d869f8c403a41f509e7f7 height=676881 version=0x20000000 log2_work=92.771642 tx=629219354 date='2021-03-29T18:50:42Z' progress=0.996321 cache=19.4MiB(144694txo)
2021-04-06T10:46:17Z New outbound peer connected: version: 70015, blocks=677998, peer=789 (full-relay)
2021-04-06T10:46:18Z New outbound peer connected: version: 70015, blocks=677998, peer=791 (full-relay)
2021-04-06T10:46:30Z New outbound peer connected: version: 70015, blocks=677998, peer=794 (full-relay)
2021-04-06T10:47:09Z New outbound peer connected: version: 70015, blocks=677998, peer=800 (full-relay)
2021-04-06T10:47:11Z New outbound peer connected: version: 70015, blocks=677998, peer=801 (full-relay)
2021-04-06T10:47:38Z New outbound peer connected: version: 70015, blocks=677998, peer=805 (full-relay)
2021-04-06T10:47:45Z New outbound peer connected: version: 70015, blocks=677998, peer=806 (full-relay)
2021-04-06T10:47:45Z New outbound peer connected: version: 70016, blocks=677998, peer=807 (full-relay)
2021-04-06T10:47:47Z New outbound peer connected: version: 70015, blocks=677998, peer=808 (full-relay)
2021-04-06T10:47:47Z New outbound peer connected: version: 70016, blocks=677998, peer=809 (full-relay)

no more notes about the block being invalid, let's give it some time. thanks for now....

YEEEEEEEEEEE it picked up blocks now!!! CPU is flying again thank youuuuuuuu

argh ((( everything is perfect here and last sync was without errors, so that's odd.

something similar happened other times when say i unplugged the hard disk without disconnecting or pulling the cable by mistake etc.

this time not. everything was always done proper.

thanks, this is what he spit back to me:

12:46:15

null

is that the intended output?

best,
Gab

Go to Window>Console and type this:


If it doesn't solve the problem, then your database is corrupted and you need to reindex. It can happen due to hardware failures; would be good if you could test your RAM for any inconsistencies.

hi there guys,

I just launched Core earlier this morning - usually things are very fast on my SSD but for some reason it's stuck, connections with the little 4 dots come and go and it's still stuck on "Unknown, syncing headers 676880 99.8%"

I didn't have any crashes or unclean shutdown, all perfectly normal, no odd restarts or cable unplugs on the hdd.

I tried to shut it down cleanly once then relaunch and same thing. it's about 30 minutes there now - usually very fast to sync like 10-15 mins max.

Is there something going on the network?

I am syncing monero core and that one is going well.

On the log i can see several of these:

2021-04-06T09:50:14Z Disconnecting and discouraging peer 163!
2021-04-06T09:50:23Z New outbound peer connected: version: 70016, blocks=677993, peer=181 (full-relay)
2021-04-06T09:50:35Z New outbound peer connected: version: 70015, blocks=677993, peer=182 (full-relay)
2021-04-06T09:50:44Z ERROR: AcceptBlockHeader: block 0000000000000000000b818d87315f69398aa6ba886d869f8c403a41f509e7f7 is marked invalid
2021-04-06T09:50:44Z Disconnecting and discouraging peer 181!
2021-04-06T09:50:45Z ERROR: AcceptBlockHeader: block 0000000000000000000b818d87315f69398aa6ba886d869f8c403a41f509e7f7 is marked invalid
2021-04-06T09:50:45Z Disconnecting and discouraging peer 182!
2021-04-06T09:50:46Z New outbound peer connected: version: 70015, blocks=677993, peer=184 (full-relay)
2021-04-06T09:50:58Z New outbound peer connected: version: 70015, blocks=677993, peer=188 (full-relay)
2021-04-06T09:51:43Z New outbound peer connected: version: 70016, blocks=677993, peer=193 (full-relay)
2021-04-06T09:51:50Z ERROR: AcceptBlockHeader: block 0000000000000000000b818d87315f69398aa6ba886d869f8c403a41f509e7f7 is marked invalid
2021-04-06T09:51:50Z Disconnecting and discouraging peer 188!

Apparently this block is marked invalid:

2021-04-06T09:29:32Z ERROR: AcceptBlockHeader: block 0000000000000000000b818d87315f69398aa6ba886d869f8c403a41f509e7f7 is marked invalid throughout

please let me know

thanks,
Gabrio