what is -checkblocks for, and why does it default so high?

mmortal03

legendary

Activity: 1762

Merit: 1010

Quote from: Pieter Wuille on February 05, 2013, 05:05:16 AM

3: In addition, check that the current chainstate matches can reasonably be the result of the last N blocks, where N is limited by both -checkblocks and the amount of in-memory cache (-dbcache); typically it's around 150.

So, if you raise the value of dbcache, does it lengthen the checkblocks process at startup, assuming that the value of checkblocks stays constant? If it does, this would seem to be a counter-intuitive result.

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: gmaxwell on February 07, 2013, 01:49:01 AM

Quote from: K1773R on February 05, 2013, 05:21:58 AM

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.

That doesn't make you sure it won't be corrupted. No widely used filesystems give you atomic operations across multiple files (and even if any did— we're not invoking them). Especially with 0.8 introducing a brand new potentially warty storage engine it's good to have the checking though turning it down is way better than turning it off— most of the benefit comes from checking the most recent blocks.

i got my checkblocks set @ 100, 2.5k is imho overkill since its such a long timerange.

gmaxwell

staff

Activity: 4172

Merit: 8419

Quote from: K1773R on February 05, 2013, 05:21:58 AM

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.

That doesn't make you sure it won't be corrupted. No widely used filesystems give you atomic operations across multiple files (and even if any did— we're not invoking them). Especially with 0.8 introducing a brand new potentially warty storage engine it's good to have the checking though turning it down is way better than turning it off— most of the benefit comes from checking the most recent blocks.

dooglus

legendary

Activity: 2940

Merit: 1330

Quote from: Pieter Wuille on February 06, 2013, 11:25:26 AM

Yes, it does. It uses the same mechanism the block synchronization mechanism uses to find forks: the wallet stores a series of hashes: the last one, the 9 direct ancestors of the last one, and then the hashes of blocks with exponentially increasing distance back.

I was trying to get my head around that code yesterday after I asked. Thanks for explaining it.

Pieter Wuille

legendary

Activity: 1072

Merit: 1174

Quote from: dooglus on February 05, 2013, 02:37:55 PM

Does this feature know about blockchain reorgs?

Yes, it does. It uses the same mechanism the block synchronization mechanism uses to find forks: the wallet stores a series of hashes: the last one, the 9 direct ancestors of the last one, and then the hashes of blocks with exponentially increasing distance back.

deepceleron

legendary

Activity: 1512

Merit: 1028

Quote from: dooglus on February 05, 2013, 02:37:55 PM

or does it remember the actual block hash that it last saw?

The data looks like this:
{
"bestblock": "00000000000004cce890839ecc7a680b095dd28cce99f964bcfadcd43e4be389",

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: dooglus on February 05, 2013, 04:34:53 PM

Quote from: K1773R on February 05, 2013, 03:31:47 PM

it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

I'm talking about the case where I have two different wallet.dat files, but only one copy of the blockchain. I rename the wallet.dat files while bitcoin-qt is shut down.

When the fork happened, I had the other wallet loaded. When I switch to the first wallet, will it notice that a fork has occurred and act accordingly?

if its later than 2500 blocks, then i dont think it will. but you can fix this by using -rescan

dooglus

legendary

Activity: 2940

Merit: 1330

Quote from: K1773R on February 05, 2013, 03:31:47 PM

it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

I'm talking about the case where I have two different wallet.dat files, but only one copy of the blockchain. I rename the wallet.dat files while bitcoin-qt is shut down.

When the fork happened, I had the other wallet loaded. When I switch to the first wallet, will it notice that a fork has occurred and act accordingly?

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: dooglus on February 05, 2013, 02:37:55 PM

Quote from: Pieter Wuille on February 05, 2013, 05:05:16 AM

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

Does this feature know about blockchain reorgs?

For example, the wallet knows it has seen up to block 10000, but since I last loaded the wallet up the blockchain was reorganised, with 'better' blocks 9999 and 10000 replacing the ones my wallet scanned. Does the wallet just remember "10000" as its bestblock, or does it remember the actual block hash that it last saw?

it will first load, afterwards it will see the blockchain "fork" and remove the orphaned blocks. in this case it will check from block 10000-2500 to 10000 and validate each of em, if invalid -> redownload, invalid -> just start as usual.

dooglus

legendary

Activity: 2940

Merit: 1330

Quote from: Pieter Wuille on February 05, 2013, 05:05:16 AM

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

Does this feature know about blockchain reorgs?

For example, the wallet knows it has seen up to block 10000, but since I last loaded the wallet up the blockchain was reorganised, with 'better' blocks 9999 and 10000 replacing the ones my wallet scanned. Does the wallet just remember "10000" as its bestblock, or does it remember the actual block hash that it last saw?

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: Pieter Wuille on February 05, 2013, 05:32:55 AM

Quote from: K1773R on February 05, 2013, 05:21:58 AM

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 0 without any problems.

-checkblocks=0 actually means "check all blocks" (because of compatibility with the pre-0.6 -checkblocks (without parameter) that checked everything instead of just the last 2500 blocks).

ouch, edited my post! ty

Pieter Wuille

legendary

Activity: 1072

Merit: 1174

Quote from: K1773R on February 05, 2013, 05:21:58 AM

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 0 without any problems.

-checkblocks=0 actually means "check all blocks" (because of compatibility with the pre-0.6 -checkblocks (without parameter) that checked everything instead of just the last 2500 blocks).

K1773R

legendary

Activity: 1792

Merit: 1008

/dev/null

Quote from: dooglus on February 04, 2013, 08:48:52 PM

Quote from: DeathAndTaxes on February 04, 2013, 08:16:03 PM

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make? The blockchain is simply an unprotected file on the file system.

So are and ~/.bitcoin/bitcoin.conf. Once the attacker is has the ability to write to the local hard drive there's not a whole lot we can do to protect ourselves. If I'm malware rewriting the blockchain, don't I just add a "checkblocks=1" to bitcoin.conf so my change isn't detected?

if your sure your blockchain wont get corrupted (RAID with brtfs for example) u can sest it to 1 without any problems.

Pieter Wuille

legendary

Activity: 1072

Merit: 1174

Quote from: dooglus on February 04, 2013, 08:10:16 PM

I notice in the current git version of bitcoin-qt it sets -checkblocks to 2500 by default, which results in:

"Verifying last 2500 blocks at level 1"

each time I start bitcoin-qt.

What is that for? Why do blocks need verifying? Nothing writes blocks to disk except bitcoin-qt, so what's this extra verification step for? Is it perhaps like a 'rescan', looking for new transactions which may not be in my wallet yet?

-rescan is a wallet option, which rescans the block chain for transactions missing from the wallet. It shouldn't be necessary in normal operation since 0.3.21 (since then, wallet auto rescan from the last block they 'saw').

The primary reason for -checkblocks (a feature that has existed for as long as I remember, but has only been made configurable in 0.6) is preventing accidental disk corruption of the block chain file, which could result in rejecting the best chain. It is certainly not a protection against an attacker that is able to write to your block files (though it does make such an attack harder).

Also note that both the meaning of the check levels as the default number of blocks to scan has been changed in git head (only 288 blocks, but with a far more thorough check). This is safe, as block data corruption is no longer a chain forking risk in the new database layout (only the unspent transaction output database, aka coins database, aka chainstate matters).

Here's an explanation of the new -checklevel meanings (for 0.8 ):

0: Validate all block headers + compare (by hash) to blocks on disk for the last -checkblocks blocks
1: In addition, verify (standalong) validity of those -checkblocks blocks
2: In addition, verify that undo data matches checksums
3: In addition, check that the current chainstate matches can reasonably be the result of the last N blocks, where N is limited by both -checkblocks and the amount of in-memory cache (-dbcache); typically it's around 150.
4: In addition, for the last N blocks (see above), do full validation (including signature checks).

Note that the new default (3) is in fact a stronger check than the previous highest level (6).

deepceleron

legendary

Activity: 1512

Merit: 1028

Quote from: dooglus on February 04, 2013, 10:27:14 PM

Quote from: Peter Todd on February 04, 2013, 09:53:42 PM

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

So is it a security feature primarily?

I was wondering if it might be doing a 'rescan', looking through recent blocks for transactions which aren't in the wallet.dat, but I did a test that seems to suggest rescan is no longer needed. I sent coins from one wallet to another, waited for the transaction to get 2 confirmations, closed bitcoin-qt, switched the wallet.dat out and replaced it with the recipient wallet, restarted bitcoin-qt with -checkblocks=1 and the transaction showed up in the recipient wallet immediately.

Is that a feature of the new leveldb stuff?

The wallet contains a value called "bestblock", which contains the last block seen when the wallet was last used. When starting with an older wallet than blockchain, Bitcoin knows that it needs to look for any wallet transactions after that block height.

Since Bitcoin may previously have crashed or been terminated unceremoniously while it was writing to the blockchain, the startup check ensures that data is in a reasonable state. The default level should really be bumped up to 2:

Quote

-checklevel specifies how thorough the verification must be:
0: only check whether the block exists on disk
1: verify block validity (default)
2: verify transaction index validity
3: check transaction hashes
4: check whether spent txouts were spent within the main chain
5: check whether all prevouts are marked spent
6: check whether spent txouts were spent by a valid transaction that consume them

Code is self-documenting.

dooglus

legendary

Activity: 2940

Merit: 1330

Quote from: Peter Todd on February 04, 2013, 09:53:42 PM

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

So is it a security feature primarily?

I was wondering if it might be doing a 'rescan', looking through recent blocks for transactions which aren't in the wallet.dat, but I did a test that seems to suggest rescan is no longer needed. I sent coins from one wallet to another, waited for the transaction to get 2 confirmations, closed bitcoin-qt, switched the wallet.dat out and replaced it with the recipient wallet, restarted bitcoin-qt with -checkblocks=1 and the transaction showed up in the recipient wallet immediately.

Is that a feature of the new leveldb stuff?

Peter Todd

legendary

Activity: 1120

Merit: 1149

FWIW the default has been reduced to 288 blocks (2 days) for 0.8: https://github.com/bitcoin/bitcoin/pull/2222/files

dooglus

legendary

Activity: 2940

Merit: 1330

Quote from: DeathAndTaxes on February 04, 2013, 08:16:03 PM

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make? The blockchain is simply an unprotected file on the file system.

So are and ~/.bitcoin/bitcoin.conf. Once the attacker is has the ability to write to the local hard drive there's not a whole lot we can do to protect ourselves. If I'm malware rewriting the blockchain, don't I just add a "checkblocks=1" to bitcoin.conf so my change isn't detected?

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

"Nothing writes blocks to disk except bitcoin-qt"

Think from a security point of view that is a good assumption to make? The blockchain is simply an unprotected file on the file system. Would be downright trivial to design a malware which replaces it with bogus transactions.

By going back x blocks it requires the attacker to produce x blocks. 2500 blocks at current difficulty is beyond the ability of all but the most determined attackers. They simply can't replace history.

dooglus

legendary

Activity: 2940

Merit: 1330

I notice in the current git version of bitcoin-qt it sets -checkblocks to 2500 by default, which results in:

"Verifying last 2500 blocks at level 1"

each time I start bitcoin-qt.

What is that for? Why do blocks need verifying? Nothing writes blocks to disk except bitcoin-qt, so what's this extra verification step for? Is it perhaps like a 'rescan', looking for new transactions which may not be in my wallet yet?

Topic: what is -checkblocks for, and why does it default so high? (Read 3278 times)