Topic: why bitcoind started download bloks from the very begining? (Read 424 times)

reindex process finished, I am pretty sure, I found a specific note in the log file.
I am not really sure either it downloading blocks or just checking it, but definetly it move forward too slow, fraction of percents per day.

so, at the end of the day I decided to start from the very beginning



well, I am pretty sure with datadir, because debug.log created exactly in those dir, and I were observing debug.log....

external drive, well, in the article about building full node with raspberry nothing was told about potential problems. in the fact, this is only way to have a pretty big storage.



oh, didn't know this fact, so thank you.


---
Sergey

Are you monitoring network traffic and sure that it is actually downloading the blockchain?

During a reindex, it will appear to be downloading the blockchain because it looks like it's syncing from genesis. But it isn't actually downloading anything, just loading blocks from disk and updating the chainstate.

Are you sure that it is using the correct datadir? Also, external hard drives are known to occasionally have issues with Bitcoin Core. Usually something gets corrupted.

That's expected. Blocks are downloaded out of order, but stored in the order downloaded. Different nodes will typically end up downloading blocks in different orders depending on their peers and internet connection.

it is wired
I thought that blk000XX.dat files are similar in each node, but it is not!

this is old one

$ ll blocks/blk00000.dat
-rw------- 1 srg srg 134214783 Apr  2 05:45 blocks/blk00000.dat
$ sha256sum blocks/blk00000.dat
a49e3196378bcd5fa59f53a7b568684fcdd4c626445deda127e5933122008765  blocks/blk00000.dat

and I restarted process from the scratch

$ ll blocks/blk00000.dat
-rw------- 1 srg srg 134215966 Apr  5 16:03 blocks/blk00000.dat
$ sha256sum blocks/blk00000.dat
2b41358fac62f7bf20a29dfa5761bbd39fa4f5e4ca8874cd15982abf1b7d0f42  blocks/blk00000.dat
any ideas why?

---
Sergey

thank you very much!

it was a Raspberry pie. OS was on microSD card and blockchain data on the attached USB external HDD.

so, one day it rejected my ssh connection and after hard reset it died.

I don't know either it was corrupted SD card or whole computer.

But I managed to get to blockchain data I downloaded so far. and I am trying to use it, but I think it wouldn't work well.

--
Sergey

A bit here:
https://en.bitcoin.it/wiki/Data_directory

Some more info here:
https://bitcoin.org/en/developer-reference#block-chain

You said that the machine you had it on before is no longer working. How did it die?
In theory, once a block is written it should be good forever, but if there were hardware / OS issues and one or more of the block files got corrupted, even one from 6 years ago got corrupted it's going to re-download the entire chain.

-Dave

I copied everything

can you suggest, where I can read about how bitcoins works?
everything I found were about general principles, but I interested to learn how file bitcoins organised....

actually I copied everything

reindex process finished but it didn't help
the whole process started from the beginning...

...and it's downloading the whole blockchain again Time-travel back to 2014.

Oh, good to know... Shall learn how it actually works.

I have moved everything. Just tried re-indexing again while watching the log and it is stuck on the following line:


Does not move any further. Slowly progressing... Will update the topic once re-indexing is over.

Not likely, as individual block files are not interchangeable... my "blk01881.dat" file will not be identical to yours.

As above, did you move just the "blocks" or did you move everything including "chainstate"?

I have just registered on this forum to ask about the same issue

I have previously downloaded full blockchain (~247Gb of ./blocks/) and since moving to the new computer can't make Bitcoin Core to pick it up.

Tried however the client still downloads from approximately 2012.

Is there a way to debug what exactly is wrong with the cache and fix it in a targeted manner instead of re-downloading >200Gb of data?

Did you just copy the "blocks" folder? or the entire Bitcoin Core "datadir"? If you didn't take the "chainstate" folder as well, then you'd need to do the reindex, otherwise you'd likely get the full download and sync.

I did this way

$bitcoind -reindex -datadir=/bitcoin >> /bitcoin/reindex.log &


will returns later to check if everything all right

Yes.

-Dave

h, thanks

should I just run
$bitcoind -reindex -daemon?

You can try to stop and reindex it now but it may or may not work since you already a chunk of the blockcain downloaded.
Unless it's a time or cost of bandwidth issue I would just let it run at this point. Depending on the performance of the VPS it should not be more then a couple of days for a complete sync.

-Dave

I copied from node that I've been running for quite some time at raspberry. but those raspberry is broken for now. So I hired VPS, installed fresh node and copied whole blocks directory

I didn't do reindex, unfortunately, just make a copy and started bitcoind

I have found this does happen when there is something corrupted.
Was the node you copied the data from fully synced? Do you still have the files?

If so you can remove the block files and copy them back. You can then start bitcoin with reindex and it should scan through everything and get you back to where you were.

Depending on the speed of your drives & PC it can take a while.

If it has been downloading for a while a reindex is probably not going to work.

-Dave

right now it says:

2020-04-04T13:24:29Z UpdateTip: new best=0000000000000000103c37b02ede2b9975be2d341bae4e89fba9e498fa56f0c9 height=369252 version=0x00000003 log2_work=83.193593 tx=79442049 date='2015-08-10T13:22:14Z' progress=0.152376 cache=11.3MiB(85281txo)
2020-04-04T13:24:55Z UpdateTip: new best=000000000000000005881a03b55aa56e0eb35d88aa68ec20a6b076ccced5a77b height=369253 version=0x00000003 log2_work=83.193623 tx=79442269 date='2015-08-10T13:30:40Z' progress=0.152377 cache=12.1MiB(91873txo)
2020-04-04T13:25:11Z UpdateTip: new best=00000000000000000a8e80d38404a7b086f087e31cf54801ecd02999ea430b7c height=369254 version=0x00000003 log2_work=83.193652 tx=79443428 date='2015-08-10T13:35:51Z' progress=0.152379 cache=12.6MiB(96583txo)
2020-04-04T13:25:11Z UpdateTip: new best=0000000000000000090ecfb2131b0a56ff725571f9f9bdade558b2821cb6d0be height=369255 version=0x00000003 log2_work=83.193682 tx=79443429 date='2015-08-10T13:36:52Z' progress=0.152379 cache=12.6MiB(96584txo)

I think my -datadir is correct, because debit.log sit inside correct directory.

Here is the first 100 rows from my debug.log

-----
$ head -n 100 debug.log.1





2020-04-02T18:19:50Z Bitcoin Core version v0.19.1.0-05f5dd96c (release build)
2020-04-02T18:19:50Z Assuming ancestors of block 00000000000000000005f8920febd3925f8272a6a71237563d78c2edfdd09ddf have valid signatures.
2020-04-02T18:19:50Z Setting nMinimumChainWork=000000000000000000000000000000000000000008ea3cf107ae0dec57f03fe8
2020-04-02T18:19:50Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2020-04-02T18:19:50Z Using RdRand as an additional entropy source
2020-04-02T18:19:50Z Default data directory /home/srg/.bitcoin
2020-04-02T18:19:50Z Using data directory /bitcoin
2020-04-02T18:19:50Z Config file: /bitcoin/bitcoin.conf
2020-04-02T18:19:50Z Using at most 125 automatic connections (1024 file descriptors available)
2020-04-02T18:19:50Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2020-04-02T18:19:50Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2020-04-02T18:19:50Z Using 0 threads for script verification
2020-04-02T18:19:50Z HTTP: creating work queue of depth 16
2020-04-02T18:19:50Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2020-04-02T18:19:50Z HTTP: starting 4 worker threads
2020-04-02T18:19:50Z Using wallet directory /bitcoin
2020-04-02T18:19:50Z init message: Verifying wallet(s)...
2020-04-02T18:19:50Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2020-04-02T18:19:50Z Using wallet /bitcoin
2020-04-02T18:19:50Z scheduler thread start
2020-04-02T18:19:50Z BerkeleyEnvironment::Open: LogDir=/bitcoin/database ErrorFile=/bitcoin/db.log
2020-04-02T18:19:50Z init message: Loading banlist...
2020-04-02T18:19:50Z ERROR: DeserializeFileDB: Failed to open file /bitcoin/banlist.dat
2020-04-02T18:19:50Z Invalid or missing banlist.dat; recreating
2020-04-02T18:19:50Z Cache configuration:
2020-04-02T18:19:50Z * Using 2.0 MiB for block index database
2020-04-02T18:19:50Z * Using 8.0 MiB for chain state database
2020-04-02T18:19:50Z * Using 440.0 MiB for in-memory UTXO set (plus up to 286.1 MiB of unused mempool space)
2020-04-02T18:19:50Z init message: Loading block index...
2020-04-02T18:19:50Z Opening LevelDB in /bitcoin/blocks/index
2020-04-02T18:19:51Z Opened LevelDB successfully
2020-04-02T18:19:51Z Using obfuscation key for /bitcoin/blocks/index: 0000000000000000
2020-04-02T18:20:07Z LoadBlockIndexDB: last block file = 1999
2020-04-02T18:20:07Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=32, size=45428898, heights=621477...621508, time=2020-03-13...2020-03-13)
2020-04-02T18:20:07Z Checking all blk files are present...
2020-04-02T18:20:07Z Opening LevelDB in /bitcoin/chainstate
2020-04-02T18:20:07Z Opened LevelDB successfully
2020-04-02T18:20:07Z Wrote new obfuscate key for /bitcoin/chainstate: b5d8231d442982ad
2020-04-02T18:20:07Z Using obfuscation key for /bitcoin/chainstate: b5d8231d442982ad
2020-04-02T18:20:07Z init message: Rewinding blocks...
2020-04-02T18:20:07Z  block index           16927ms
2020-04-02T18:20:07Z init message: Loading wallet...
2020-04-02T18:20:07Z BerkeleyEnvironment::Open: LogDir=/bitcoin/database ErrorFile=/bitcoin/db.log
2020-04-02T18:20:07Z [default wallet] Wallet File Version = 10500
2020-04-02T18:20:07Z [default wallet] Keys: 0 plaintext, 0 encrypted, 0 w/ metadata, 0 total. Unknown wallet records: 0
2020-04-02T18:20:07Z [default wallet] Performing wallet upgrade to 169900
2020-04-02T18:20:11Z [default wallet] keypool added 2000 keys (1000 internal), size=2000 (1000 internal)
2020-04-02T18:20:11Z [default wallet] Wallet completed loading in            3758ms
2020-04-02T18:20:11Z [default wallet] setKeyPool.size() = 2000
2020-04-02T18:20:11Z [default wallet] mapWallet.size() = 0
2020-04-02T18:20:11Z [default wallet] mapAddressBook.size() = 0
2020-04-02T18:20:11Z UpdateTip: new best=000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f height=0 version=0x00000001 log2_work=32.000022 tx=1 date='2009-01-03T18:15:05Z' progress=0.000000 cache=0.0MiB(0txo)
2020-04-02T18:20:11Z block tree size = 621509
2020-04-02T18:20:11Z nBestHeight = 0
2020-04-02T18:20:11Z Bound to [::]:8333
2020-04-02T18:20:11Z Bound to 0.0.0.0:8333
2020-04-02T18:20:11Z init message: Loading P2P addresses...
2020-04-02T18:20:11Z ERROR: DeserializeFileDB: Failed to open file /bitcoin/peers.dat
2020-04-02T18:20:11Z Invalid or missing peers.dat; recreating
2020-04-02T18:20:11Z UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 version=0x00000001 log2_work=33.000022 tx=2 date='2009-01-09T02:54:25Z' progress=0.000000 cache=0.0MiB(1txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd height=2 version=0x00000001 log2_work=33.584985 tx=3 date='2009-01-09T02:55:44Z' progress=0.000000 cache=0.0MiB(2txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000082b5015589a3fdf2d4baff403e6f0be035a5d9742c1cae6295464449 height=3 version=0x00000001 log2_work=34.000022 tx=4 date='2009-01-09T03:02:53Z' progress=0.000000 cache=0.0MiB(3txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000004ebadb55ee9096c9a2f8880e09da59c0d68b1c228da88e48844a1485 height=4 version=0x00000001 log2_work=34.32195 tx=5 date='2009-01-09T03:16:28Z' progress=0.000000 cache=0.0MiB(4txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000009b7262315dbf071787ad3656097b892abffd1f95a1a022f896f533fc height=5 version=0x00000001 log2_work=34.584985 tx=6 date='2009-01-09T03:23:48Z' progress=0.000000 cache=0.0MiB(5txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000003031a0e73735690c5a1ff2a4be82553b2a12b776fbd3a215dc8f778d height=6 version=0x00000001 log2_work=34.807377 tx=7 date='2009-01-09T03:29:49Z' progress=0.000000 cache=0.0MiB(6txo)
2020-04-02T18:20:11Z upnp thread start
2020-04-02T18:20:11Z UpdateTip: new best=0000000071966c2b1d065fd446b1e485b2c9d9594acd2007ccbd5441cfc89444 height=7 version=0x00000001 log2_work=35.000022 tx=8 date='2009-01-09T03:39:29Z' progress=0.000000 cache=0.0MiB(7txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000408c48f847aa786c2268fc3e6ec2af68e8468a34a28c61b7f1de0dc6 height=8 version=0x00000001 log2_work=35.169947 tx=9 date='2009-01-09T03:45:43Z' progress=0.000000 cache=0.0MiB(8txo)
2020-04-02T18:20:11Z torcontrol thread start
2020-04-02T18:20:11Z UpdateTip: new best=000000008d9dc510f23c2657fc4f67bea30078cc05a90eb89e84cc475c080805 height=9 version=0x00000001 log2_work=35.32195 tx=10 date='2009-01-09T03:54:39Z' progress=0.000000 cache=0.0MiB(9txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000002c05cc2e78923c34df87fd108b22221ac6076c18f3ade378a4d915e9 height=10 version=0x00000001 log2_work=35.459454 tx=11 date='2009-01-09T04:05:52Z' progress=0.000000 cache=0.0MiB(10txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000097be56d606cdd9c54b04d4747e957d3608abe69198c661f2add73073 height=11 version=0x00000001 log2_work=35.584985 tx=12 date='2009-01-09T04:12:40Z' progress=0.000000 cache=0.0MiB(11txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000027c2488e2510d1acf4369787784fa20ee084c258b58d9fbd43802b5e height=12 version=0x00000001 log2_work=35.700462 tx=13 date='2009-01-09T04:21:28Z' progress=0.000000 cache=0.0MiB(12txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000005c51de2031a895adc145ee2242e919a01c6d61fb222a54a54b4d3089 height=13 version=0x00000001 log2_work=35.807377 tx=14 date='2009-01-09T04:23:40Z' progress=0.000000 cache=0.0MiB(13txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000080f17a0c5a67f663a9bc9969eb37e81666d9321125f0e293656f8a37 height=14 version=0x00000001 log2_work=35.906913 tx=15 date='2009-01-09T04:33:09Z' progress=0.000000 cache=0.0MiB(14txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000b3322c8c3ef7d2cf6da009a776e6a99ee65ec5a32f3f345712238473 height=15 version=0x00000001 log2_work=36.000022 tx=16 date='2009-01-10T04:45:46Z' progress=0.000000 cache=0.0MiB(15txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000174a25bb399b009cc8deff1c4b3ea84df7e93affaaf60dc3416cc4f5 height=16 version=0x00000001 log2_work=36.087485 tx=17 date='2009-01-10T04:45:58Z' progress=0.000000 cache=0.0MiB(16txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000003ff1d0d70147acfbef5d6a87460ff5bcfce807c2d5b6f0a66bfdf809 height=17 version=0x00000001 log2_work=36.169947 tx=18 date='2009-01-10T05:03:11Z' progress=0.000000 cache=0.0MiB(17txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000008693e98cf893e4c85a446b410bb4dfa129bd1be582c09ed3f0261116 height=18 version=0x00000001 log2_work=36.24795 tx=19 date='2009-01-10T05:12:14Z' progress=0.000000 cache=0.0MiB(18txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000841cb802ca97cf20fb9470480cae9e5daa5d06b4a18ae2d5dd7f186f height=19 version=0x00000001 log2_work=36.32195 tx=20 date='2009-01-10T05:22:54Z' progress=0.000000 cache=0.0MiB(19txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000067a97a2a37b8f190a17f0221e9c3f4fa824ddffdc2e205eae834c8d7 height=20 version=0x00000001 log2_work=36.392339 tx=21 date='2009-01-10T05:39:55Z' progress=0.000000 cache=0.0MiB(20txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000006f016342d1275be946166cff975c8b27542de70a7113ac6d1ef3294f height=21 version=0x00000001 log2_work=36.459454 tx=22 date='2009-01-10T05:49:13Z' progress=0.000000 cache=0.0MiB(21txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000098b58d427a10c860335a21c1a9a7639e96c3d6f1a03d8c8c885b5e3b height=22 version=0x00000001 log2_work=36.523584 tx=23 date='2009-01-10T06:04:27Z' progress=0.000000 cache=0.0MiB(22txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000000cd339982e556dfffa9de94744a4135c53eeef15b7bcc9bdeb9c2182 height=23 version=0x00000001 log2_work=36.584985 tx=24 date='2009-01-10T06:06:51Z' progress=0.000000 cache=0.0MiB(23txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000fc051fbbce89a487e811a5d4319d209785ea4f4b27fc83770d1e415f height=24 version=0x00000001 log2_work=36.643878 tx=25 date='2009-01-10T06:17:57Z' progress=0.000000 cache=0.0MiB(24txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000008e35a1d59ea1be8d76683662f47fd13c62a9e347ad5845a26f762026 height=25 version=0x00000001 log2_work=36.700462 tx=26 date='2009-01-10T06:24:06Z' progress=0.000000 cache=0.0MiB(25txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000041438e52d25bccab8798a92cabafdaad69a071d3d2a41718faf01098 height=26 version=0x00000001 log2_work=36.75491 tx=27 date='2009-01-10T06:54:10Z' progress=0.000000 cache=0.0MiB(26txo)
2020-04-02T18:20:11Z UpdateTip: new best=0000000071350772f98f84babf35502b33d42ee8466d3dde0f376c4120352081 height=27 version=0x00000001 log2_work=36.807377 tx=28 date='2009-01-10T06:56:13Z' progress=0.000000 cache=0.0MiB(27txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000bb0d9430d3d1bab474be5050342161efcca9f7e45b151bff9a700944 height=28 version=0x00000001 log2_work=36.858003 tx=29 date='2009-01-10T15:30:57Z' progress=0.000000 cache=0.0MiB(28txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000c57a1b6351208c592eef8eff015d93c899a047fe35b35252a4a59bcb height=29 version=0x00000001 log2_work=36.906913 tx=30 date='2009-01-10T15:31:43Z' progress=0.000000 cache=0.0MiB(29txo)
2020-04-02T18:20:11Z init message: Starting network threads...
2020-04-02T18:20:11Z UpdateTip: new best=00000000bc919cfb64f62de736d55cf79e3d535b474ace256b4fbb56073f64db height=30 version=0x00000001 log2_work=36.954218 tx=31 date='2009-01-10T15:42:02Z' progress=0.000000 cache=0.0MiB(30txo)
2020-04-02T18:20:11Z UpdateTip: new best=000000009700ff3494f215c412cd8c0ceabf1deb0df03ce39bcfc223b769d3c4 height=31 version=0x00000001 log2_work=37.000022 tx=32 date='2009-01-10T15:52:16Z' progress=0.000000 cache=0.0MiB(31txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000e5cb7c6c273547b0c9421b01e23310ed83f934b96270f35a4d66f6e3 height=32 version=0x00000001 log2_work=37.044416 tx=33 date='2009-01-10T15:59:31Z' progress=0.000000 cache=0.0MiB(32txo)
2020-04-02T18:20:11Z UpdateTip: new best=00000000a87073ea3d7af299e02a434598b9c92094afa552e0711afcc0857962 height=33 version=0x00000001 log2_work=37.087485 tx=34 date='2009-01-10T16:12:19Z' progress=0.000000 cache=0.0MiB(33txo)