Topic: What are checkpoints in bitcoin code? (Read 14841 times)

They have positives and negatives.

A checkpoint means that miners won't jump to a secret fork.

Miners who didn't checkpoint will still jump and so will some clients if they were offline when the checkpoint triggered.

This means you split the network into 2 pieces if a major fork happens.

I'm having a hard time figuring out why checkpoints do not provide any security ....

How do you measure current hashing power?

Do you mean difficulty must be at least 50% of what it was 120 blocks ago?  That isn't much protection since difficulty doesn't change very often.


The problem with checkpointing is what happens if there is a reversal.

For example, if I launch a 51% attack starting from behind the checkpoint, then all new nodes will follow my chain.

Nodes that were offline will also follow my chain.

Nodes that were online would follow the other chain.

This forks the network and it can't heal itself.

I think sending the entire header tree would help here.  A new node that connects would see that there was a 120+ block fork.  It could warn the user that the chain is potentially unstable at the moment.

Nodes would be able to see that there was a fork that would have caused a checkpoint.


Including block time is risky, since it creates an incentive to mess with the field.  An attacker could set it to whatever he likes (but not to far in the future).

Faking high hash rate means earlier block times, so the future rule doesn't apply.  If I produce a sequence of blocks with timestamps 1 second apart, it will seem like a massive hash rate.


If they are both on the same chain, then keeping the old one has no effect one way or another.

What do you mean by soft?  The rule could be that a fork 120 blocks longer can displace a checkpoint.  This would at least allow the network to re-sync.

Clients could enter emergency mode using this rule.

- find longest chain (L)
- find any other chain which
-- forked more than 120 blocks ago
-- where the fork is at least 120 blocks long
-- has a weight of at least (L's weight - 120 blocks)

The hard checkpoints also speed up validation.  They could do that without acting as checkpoints.

Validation would start at the last hard checkpoint that is on the longest chain.

If the entire chain was displaced by a higher POW chain, then nodes would accept the new chain.  They would have to verify it in its entirety.

I still consider that since coinbase transactions are agreed by the network to be spendable, that means they are agreed to be existent to foreseeable future and thus should be checkpointed.

How about following rule:
when a block is generated, the block 120 blocks deep the chain is checkpointed, *if*
- current hashing power is more than half of the to-be-checkpointed block

Not sure if this checkpointing even need to be broadcasted; every client should be in full agreement of it.

Then, I also suggest that coinbase transactions *are not spendable* unless they are checkpointed.

Does this guarantee solving problems with forked chains? Only the chain is checkpointed which has majority of the hashing power.
Still thinking good formula to estimate if the hashing power is more than half.. both blocks difficulty and time should be considered. And maybe some safety reserve, so variations on block times does not affect the result.

Does this open any attack vectors?


Edit: when a new soft checkpoint is made, the previous one can be forgotten, ie there need not to be more than one soft checkpoint at a time.

amazing checkpoint,still cant quitely follow

Being peer-to-peer also means that everyone in the network can choose to use whatever client or whatever version of a certain client in order to be part of the Bitcoin network. As of now, we don't have too many options for running a full node, but in the future, more independent implementations will likely to emerge.

In that case, whoever controls the update system controls everything.

That is my point and until 100% of nodes upgrade their software they would be vulnerable to an isolation attack.  It creates a more fragile and exploitable system than users wallets simply being able to switch to the longest chain with no action required on the part of the user.

Do you mean this:
https://bitcoin.org/en/alert/2013-03-11-chain-fork

But it would have been solved simply by bootstrapping the blockchain along with the software downgrade. If it needs software update, you allways have the option to force block reload, maybe from some hardcoded checkpoint.

The main issue is that in the event of a flaw which causes the network to fork you have now hard coded that fork.

For example if the db bug issue which caused a fork in the network, miners on the longer fork modified their blockchains to support the shorter fork.  So some intervention was required but for all 100,000+ other nodes they simply saw a re-org once the the shorter fork became the longer one.  If they used ratcheting checkpoints they would have been permanently locked onto a minority fork and attackers could have exploited that.

There may be some merit in using a ratcheting checkpoint system however it would have to be set much much deeper than 6 blocks.  Even 144 blocks (one day) may be insufficient.  Something like 1,024 blocks (~1 week) is more the range that would be conservative enough.

Also understand a ratcheing checkpoint system only helps existing nodes.  Bootstraping nodes would be remain vulnerable as they haven't seen the longest chain yet.

I'm quite sure this has been discussed before, but why clients does not make automatic checkpoints? I understand that transactions which are at least 6 blocks old, are considered valid. There should be no reason to re-arrange the blockchain after that, for any reason. If 6 blocks sounds too short, maybe 120 blocks then? Coinbase coins should definitely be stoned when they are spendable.

So I'd like to hear what is the reasoning behind the fact that there are no automatic checkpoints?

Let's *try* to imagine situations where automatic checkpoint (here I mean every node considers it chain's blocks from head-6 and earlier to be 'stoned', non-changeable) would cause harm:

1. Branch which will survive 6 blocks. Isn't this practically an error case? If it can happen, that just means 6 block validation time is too short since it cannot guarantee transactions confirmed.
2. Some node mines isolated from the others. This is the case where it would cause minor problem. However the problem is very local, and very similar problem would be regardless of the automatic checkpoint.

Basicly it would just be a non-centralised agreement between nodes after they find a new head block; we all consider blocks backwards from this-6 nodes to be immutable.

It's not my idea, the OP in that thread thought of it.


No, they always points backwards.

To determine where to point the up link.

- count the number of leading zeros on the previous block's hash (i.e. the block you plan to build on top of)
- starting from the block before that, find the first block with more leading zeros
- point to the block one after that

So, if the blocks had leading zeros

A: 11(up = Genesis)
B: 10 (up = Genesis)
C: 20 (up = B)   (Not both prev and up point at B)
D: 10 (up = Genesis)
E: 10 (up = D)
F: 10 (up = D)
G: 10 (up = D)
H: 15 (up = D)
I: 10 (up = D)
J: 10 (up = I)
K: 10 (up = I)

The for block L, the links would be

prev: K
up: I

The reason is that H has more leading zeros than K.  You don't link to C, even though it has more leading zeros.  Once you find that block, the up link points to the one after that.

So, finding the best block

stating at K, follow up link, until up link points to genesis
- follow up link to get to I (prev now point at H, so found block with 15 leading zeros)
- follow up link to get to D (prev now points at C, so found block with 20 leading zeros)
- up for D link points to genesis, so break from loop

follow D's previous link to get C which is the best block

You can also find all blocks with more than n leading zeros by following the previous link any time you find a block with more than that number of leading zeros.

Heh.  This is the sort of nonsense I was talking about.

This is local control.  You, the node operator, have options.  Hardly a new thing.  Since day 1, all node operators have had the ability to fork for whatever reason and through whatever means they desire.

You have a warped view of "decentralized".  It doesn't mean that no one has any authority, it means that no one can stop you.

Who can stop you from turning checkpoints off?  No one.
Who can stop you from stripping them out of the source entirely and running your own pure node?  No one.

Unbeatable arguments, thank you.


Do you understand that this option is protocol specification change? It effectively forks blockchain.

Yes, they are centralization. But as long as they do not adjust blockchain on regular basis (select which chain is good and which is not) - they gain trust of users.


Yes, but the problem is that the client with checkpoints and one without - are not 100% compatible with each other. Checkpoints are not just cosmetic change.


Bitcoin already has mechanism to achieve consensus - voting by computing resources, it should not be substituted by any centralized authority.

Heh, oops, -addtag is a custom mod that I maintain.  It does indeed add tags to the coinbase for mining.

-txindex makes the node keep an index of all transactions.  This is very helpful for RPC services, particularly those using raw transactions.

What do those two do? I'm guessing the tag is meant for miners? Is it useful for regular people who just run a node or wallet?

Since I just run QT like normal, I don't get the full transaction index? Does that take up additional space or isn't everything in the blockchain anyway? Or I am correct to guess that this is better used by large online wallets or something?

Nonsense, every word.

In a way, bitcoin (and almost all other crypto coins) are still some what centralized. The devs are the centralization. There's not much we can do about that.

The redeeming thing is that almost all of them are open source with code available to be self-compiled.

If it were not check points, there is also the bootstrap file uploaded by Jeff Garzik. That is sort of centralized, but distributed by torrent, for all new users who need an initial sync of the correct block chain.

While the protocol itself is decentralized, it is a requirement that honest nodes form a consensus. Remember the hard fork early this year.

Of course hidden fork crunched at high speed in shadow is unlikely, and maybe checkpoints do not harm due to low probability of such fork.
But they harm in other way - they compromise Bitcoin decentralization, which is in my opinion is the biggest attractive feature of Bitcoin.
Today they add checkpoints (this effectively changes protocol specification), tomorrow they could freeze BTC on your address, by adding small tweaks to validation routine (just like checkpoints validation).

Yes - I considered this. I think it is not a problem, because it is very unlikely that someone can prepare 0.01*MaxBlocks different blocks (unchained, as you noted) which have accumulated difficulty more than 1% or so of total accumulated work spent on main chain for whole time of Bitcoin existence.

I should look closer to your proposal. For now I have one question:
Does "up" link points to future? If so - is it included in block hash or can it be falsified?

I think the checkpoints are about at least a month or two back in time. Anyone who has more than 51% of 6.5 Petahash of mining power that can fork their own chain past the latest check point, is better off mining regularly and getting 51% of 3600 coins daily or about 1800 coins, at least for the next 2000 blocks when the difficulty will double.

There would be no way to prove that they are part of the chain.

In the High hash highway, a second pointer is added.  Headers would have a link to the previous block and also an "up" link.

This allows you follow backwards from the leaf of the chain much faster, but still hit all the high blocks.  However, it is a hard fork.

The 2nd link points to the nearest block with more leading zero bits than the previous block.

In fact, I think optimal would be, point to the block after the nearest block with more leading zero bits than the previous block.

So, the blocks had the following leading zero bits

A) 30
B) 31 - up -> genesis (since 30 is the best so far)
C) 29 - up -> genesis (since 31 is the best so far)
D) 33 - up -> C
E) 31 - up -> genesis (since 33 is the best block)
F) 32 - up -> E
G) 30 - up -> E
H) 32 - up -> G

If you follow the up link backwards, you go to a block that is the successor of a block with at least 1 more leading zero than the previous block.

To find the highest block, these are the steps

- Set N to the index of the leaf block
- Set b to the number of leading zeros for block N-1

Loop
- The up pointer for block N points to block M
- Block M-1 has more leading zeros than block N-1 (by definition)
- It also has more leading zeros than any blocks from N-1 to M, inclusive
- This means you can't have skipped a better block than block M-1
- set N = M
- set b = the number of leading zeros for block M-1

Each time through the loop b is guaranteed to increase and also you are guaranteed not to skip the best block.

Therefore, this search will find the best block.

The PoW of the main chain can be estimated just by looking at the work of the best block.

You can also use the system to find all blocks which have difficulty within a number of bits of the best block.

I agree, for now and most likely for future downloading all block headers is not an issue. So don't even need to adjust protocol.

Yes, this is the key for improvement. It does not have to provide strict upper_bound, but something that >= strict_upper_bound.
Plus, if this approach will be accepted, we should add ability to request block headers sorted by difficulty, starting with the most expensive one and continue in descending order. This would allow to detect flood with even less traffic, like few percents of Max_Blocks.

I think the headers first change is a key stepping stone for this.

Downloading 300k blocks is around a 20MB download.

With modern bandwidth, nodes could download that from 8 peers reasonably quickly and then discard any chain that doesn't have sufficient POW.

The more blocks in a blockchain, the higher the total POW.  It would be cool if someone could work out a formula.

Max blocks = f(time since genesis, POW)

Not everything goes to BIP.  This is a purely internal function of the reference client.  Most discussion of changes like this actually happen on github, attached to the pull request that implements them.

I think this idea should be discussed first in separate topic. Then, as I understand it should go to Bitcoin Improvement Proposals.

they're what make Bitcoin not Peer-To-Peer.

Cool.  When can we expect your patch?

Again, it is possible to solve this problem by other approaches.
For example via MAIN_CHAIN_MIN_POW as suggested above - it does not compromises decentralization.

We know when Bitcoin was released, we know current date - we can calculate upper bound for block count in any chain.
When somebody tries to send us old chain - we ask it to sent block headers sorted by difficulty, starting with the most expensive one and continue in descending order. 1% of top block headers should have at least 0.01*MAIN_CHAIN_MIN_POW accumulated difficulty, if it is not - then it is fake, else continue, and so on.

Way back on post #5 gmaxwell explains it pretty well.

Even if it was done for the purposes of optimization - it is still protocol specification change. Clients with disabled "optimization" could accept some branches which would be rejected by client on default settings.

There are other possible approaches to do such kind of optimization, for example MAIN_CHAIN_MIN_POW described above, but it does not compromises decentralization trust as checkpoints do.

It's an optimization that can be disabled with a command line switch.

Checkpoints were put by small group of people, most of people even do not heard about checkpoints - and continue to think about Bitcoin as pure decentralized currency, while it is not.


Protocol rules allows to choose other branch if it has more accumulated difficulty, even if it is rooted in old blocks before checkpoints. Checkpoints override those rules - they essentially change protocol specification. By adding new checkpoint - you are forking protocol specification.
Choice of checkpoints should be done decentralized - by voting, and Bitcoin already has voting mechanism - vote by computing work. But if voting happens by hard checkpoints, selected by dozen of people - then it is centralized, with all bad sides of centralization.

I don't understand how you think that checkpoints mean that a small group of people is deciding which branch is correct.

The protocol rules chose the correct branch.  The checkpoints only document what that historical choice was.

I did read this thread, and several old ones: https://bitcointalksearch.org/topic/bitcoin-032-released-437 , https://bitcointalksearch.org/topic/bitcoin-is-not-as-advertised-1647
Do you have any particular objection to discuss?

You didn't actually read this thread before posting, did you?

Bitcoin is decentralized. And no small group of people should decide which branch is correct and which is not.
Checkpoints hardcoded in code were selected by small group of people.
But if Bitcoin is still claimed to be decentralized, not controlled by government, small group of people (who can be blackmailed, or forced by thermo-rectal cryptanalysis) - then only majority of users should have right to make decision.

Users should vote on which branch is correct and which is not. Bitcoin already has such voting mechanism - vote by computing power.


Of course hidden fork crunched at high speed in shadow is unlikely, and maybe checkpoints do not harm due to low probability of such fork.
But they harm in other way - they compromise Bitcoin decentralization, which is in my opinion is the biggest attractive feature of Bitcoin.
Today they add checkpoints, tomorrow they could freeze BTC on your address, by adding small tweaks to validation routine (just like checkpoints validation).

Checkpoints add almost no value - they do not prevent offensive usage of high computing power, but they trading off fair amount of credibility.
Nobody except majority should have control over blockchain, and Bitcoin's definition of majority is computing power.
Otherwise users must know that blockchain is controlled by small group of human being.

Yes, it is total accumulated work.

Is it clearly total work, rather than block height?

It is not just change of button color or font size, it is protocol specification change.
If one fork has checkpoints and other does not - then they do not have consensus on how to select main blockchain, and this could lead to blockchain fork (at least theoretically).
Bitcoin already has consensus mechanism, and it is the foundation - block-chain with bigger combined work amount is the winner, period.

In a same way as we have checkpoints now, it is possible to freeze BTC on particular address - tune a bit validation rountine (just like checkpoints validation), wait for update of majority of clients - and vu à la. I bet, governments would love this feature, keep going!
Any kind of such validation is poison to trust for the most important of Bitcoin features - decentralization.

I like this idea - it doesn't contradict decentralized principle (as checkpoints do).

Saved

There are two possibilities where this would not happen:

1. Either change enough bits in such a way that the hash stays the same. This is called a "hash collision".

2. Or perhaps insert a block which has the same hash, again a hash collission except it's doubled:  block 1.5 to be inserted has same hash as block 1, block 2 will happily accept block 1.5.

The blockchain is already a chain of hashes.  The hash for block 2 uses the hash of block 1 as (part of) its input and the hash for block 3 uses the hash for block 2 and so on.

If you were to change a single bit for block 10, then the hashes for all blocks from 10 onwards would change.  Ofc, that would mean that blocks 11 onwards would no longer meet the proof of work (which is the whole point).  You can't change anything without destroying the chain entirely.

The SHA-256 function breaks large messages into pieces for processing anyway.  It uses the output from one piece as the input into the next piece.

More like moving each cell to their own universe so if they escape they have nowhere to go.  I think you vastly misunderstand the odds we are talking about.

It's more like placing each criminal in it's own cell. Currently if one wall is blown up many thousands of criminals can escape.

By placing more walls around the criminals less criminals can escape if one wall were to be broken.

No preimage attack against SHA-2 is possible. 

Saying "it's already difficulty enough" is like saying it is already difficulty to walk through a wall due to quantum tunneling but just to be safe we should double up all walls in case someone gets "lucky".

No the opposite ofcourse, if he can find one, he can find two, or three or four, hence my suggestion to add as many hashes as possible.

So my suggestion is:

Check point hash 0 = Hash(block 0, block 1, block 2, block 3, etc up to block 2015)
Check point hash 1 = Hash(block 2016, block 2017, block 2018, etc up to block 2016+2016-1)
Check point hash 2 = Hash(block 2016 * Hash Number to (Block 2016 * (Hash Number+1))-1)
Check point hash 3 = etc

^ This will lead to many more hashes and best part is that all block hashes are included, not just a few, actually the entire block should be hashed, why stop at only block hashes ? Yes it would lead to more computational effort but that's basically the point in general, however perhaps block hashes only would be enough, it's already difficulty enough to fake one, however including all could give a bit more security because it adds dates, times, transactions, etc, but maybe could also lead to less security because of easy adjusting comments or other bogus information, with a hash only that would be a bit more difficult to do it could throw off all input calcultions to try and fake such a hash.Blocks can probably fluctuate in size, while for now the hash is always same size, so gives some constant speed and also easy to implement reliably without all kinds of potential strange block size quircks).

So, you are assuming that an attacker can find one has collision, but not two?  That doesn't make any sense.

The way the chain headers are hashed and linked makes bitcoin insanely strong against preimage attacks.  MD5 is considered to be totally broken, and preimage attacks on MD5 are rampant.  However, if we used MD5 (not even double MD5) as our blockheader hashing mechanism, we would still be totally safe.

P.S.  I didn't read much past the part that I quoted.  I hope you don't take any offense at that.  If you had other important things to say, please try to organize your thoughts better.

Satoshi answers your question in the original document:

"
The proof-of-work also solves the problem of determining representation in majority decision
making. If the majority were based on one-IP-address-one-vote, it could be subverted by anyone
able to allocate many IPs.
"

Currently bitcoin seems to be a system that uses both concepts: "confidence" and "proof of work".

The checkpoints can be considered a dirty hack to make bitcoin more secure, it's an admission that bitcoin in it's original concept/design was flawed and pretty much still is if it werent for this

However perhaps that last part of my sentence is a bit hasty... I do wonder if it's possible to insert a block anywhere in the chain if a hash collission was found.

I am not sure what kind of protective measures bitcoin has to protect against block insertion.

If these checkpoints are truely based on single blocks then it won't protect against insertions.

Thus a smarter hash would be a hash computed over the entire blockchain.

Such a hash could be included to protect against insertions. However if the attacker can calculate a hash collission then this would only double his calculation efforts.

However the point is: the more checkpoints, the better.

I think the solution to make it a little bit more secure is simply calculate a hash over the entire blockchain every 2016 blocks and store it.

At least this will increase computational resources needed to produce any fake chains and also makes insertion problematic.

However I am not so sure how easy it would be to create a "fake chain" that would be accepted by a bitcoin client if it did not have these checkpoints.

Perhaps the bitcoin client also calculates "difficulty" each 2016 blocks and would then detect that the difficulty on the fake block chain was not properly increased... (<- note flaw in logic, it can also decrease, fluctuate wildly, no rules govern it, explained and examined further below).

If the bitcoin client does not yet currently do this then it could start doing that to prevent fake chains in the future... this will make it harder for attackers to properly construct fake chains, since they will also have to increase the difficult and thus consume more hashing power.

However difficulty can be manipulated by simply pretending bitcoin was not popular at the time and there were little transactions and it took days and months to find a block, so ultimately it would probably get accepted unless some measurements based on history would be used but this could hamper future wild changes.

Biggest change in difficult was probably around 10 july 2010 when bitcoin was announced on slashdot... resulting in the slashdot effect, a jump in difficulty of 300%.

However using difficulty as a way to detect fake chains could have merit.

The idea for a fake block chain would be to "out race" the real chain.

This implies either:

1. The fake block chain has more blocks, and thus the blocks were found faster <- indicating something fishy might be going on with the difficulty, this could indicate the difficult was not properly increased each 2016 blocks.

2. The real block chain fell out of use, bitcoin became impopulair... difficulty was too high... not enough hashing power.... fake chain starts to over take it.

Option 2 could imply that bitcoin could be at risk as soon as it becomes impopular... it's difficult will start to drop... but at a slower pace then a fake chain.

A possible attack to create a fake blockchain could be:

1. Determine current ammount of blocks in the real chain.

2. Divide current ammount of blocks by 2016.

3. This indicates how many difficulty changes must be calculated.

4. Perhaps use a rollcoaster tactic:

5. Calculate as many blocks as possible with minimum difficulty.

6. This will cause the difficulty to shoot up to insanity.

7. Then stop calculating blocks for the next 2016 blocks.

^ Problem, cannot stop calculating, thus algorithm must be changed:

Spread difficulty over as many groups of 2016 as possible.

Possible attack vector:

Difficulty at start of chain was low, replace low difficulty with high difficulty to try and produce a constant difficulty.

Having a constant difficulty might be exactly what the attack would need, no more, no less, if this is better than the roller coaster approach is uncertain.

Perhaps the roller coaster approach has merit too I think so:

Roller coaster approach version 2:

Calculate as many blocks as possible without raising the difficulty to an unrecoverable state, keep it sane so that the next 2016 blocks can still be calculated.

Or other attack vector:

Spread the 2016 blocks over an enormous ammount of time... weeks, moths, years or so...

This will drop the difficulty back to minimum.

Then produce an enormous ammount of blocks.

So I think the roller coaster approach might be possible as long as it's not over done too much

So two possible attack vectors:

1. Roller coaster approach within reasonable difficulty recoverment/crash.
2. Constant difficulty approach.

However I assumes the roller coaster approach would be capable of generating limitless ammount of blocks within 2016. This is not the case. After 2016 blocks were produced it would recalculate the difficulty.

So now ask yourself one question: what would happen to the difficulty is 2016 blocks were generated in 1 second ?

What would happen to difficulty if 2016 blocks were generated in one month ?

Perhaps there is a possibility to roller coaster it in such a way that it would end up producing blocks faster than the real block chain... with less computation power needed... not sure...

I guess the answer lies in the "computation power" that is required at different difficulties relating to the hash itself.

If the computation power scales linearly with the difficulty.... then this attack may not work.

However if it's somehow possible to gain an adventage at lower difficulty then perhaps a netto sum effect is achieved, perhaps it will profit from the lower difficulty/lower computation power needed once it crashes.

Also perhaps other security checks are in place, such as checking the date/time for each block. Thus generating 2016 blocks in 1 second might not be possible/accepted by the clients ? Not sure about that...

Theoretically it might be desireable to still allow it, nothing prevents anybody from suddenly adding insane ammounts of computer power to the bitcoin network... if the bitcoin network could not scale up beyond a certain point that would be a bit strange I guess it's all about certain assumptions/sanity checks... difficult to predict the popularity of bitcoin... thus such sanity checks might hamper suddenly super popularity.

Such an event already happened with the slashdot effect: a 300% increased, as if 2 or 3 earths were added ? which could be precisely why these checks were not implace... it would have hampered it...

Time for some fun numbers:

There are currently: 271789 blocks.

Number of difficulty changes:

271789 / 2016 = 134.81597222222222222222222222222

Close to 135 difficulty changes.

That's a surprisingly low number of changes.

Let's assume a 2016 block can be generated in 1 second with minimum difficulty.

After it's generated some unknown ammount of time is needed to drop it back lower.

So 50% we can try and do in 1 second which gives:

134/2 = 67 difficulty changes.

These can be spread out over the entire time that bitcoin was running. Block 0 started at 2009-01-03 18:15:05

It's now 27/11/2013 17:38

That's a whole lot of years to spread 67 over...

Now I need to plugin the difficulty algorithm or code in bitcoin, I am not sure what it is but for now I will take this:

difficulty=((Time for a block to be found in seconds)*(hashes per second))/2^32

difficulty = 1/10.000 * ? / 2^32

not sure what hashes per second is supposed to be, perhaps the total estimated hashes per second. Perhaps 4 gigahash should be taken.

However all of this might not matter.

There seems to be a hard limit of 1/4x or 4x of what it previously took.

If true then we golden.

We might be able to produce a fake blockchain pretty easily.

1 second, 4 seconds, 16 seconds, 64 seconds, and so forth.

Ofcourse this is exponential...

But to reset the algorithm, we simply mine the next 2016 in 4 or 8 seconds to crash it back to 1.

Thus creating a fake blockchain would seem very easy as long as these minimum and maximum multiplication factors are in place.

Hence the need for these checkpoints... it's the only thing prevent a fake block chain from taking over ?

It doesn't happen that often.

However, I meant total PoW for the chain.  This is the sum of the PoW for all blocks from the genesis block to the last block in the chain.  That always increases and even if a new main chain takes over, by definition, it has more PoW.

The spam DoS attack was to create 2015 difficulty 1 blocks that are 1MB each.    You can create as many of those as you want and they are valid forks of the chain, so are stored.

If only forks with lots of total PoW are stored, then that is less of a problem, since those blocks require high PoW.

With a 51% attack, you could drop the difficulty back to 1, so there is still some risk.  However, by dropping checkpoints, you are inherently assuming that is no possible.


That is what I meant by MAIN_CHAIN_MIN_POW, it would be the minimum total work before a chain can be called the main chain. 

I think it would be cool for checkpoints to be distributed periodically in dead tree format.

Fair enough— for whatever it's worth, I'm no fan of them either. If nothing else they really create a hard time for people reasoning about the security model of Bitcoin: There are a lot of people who have a hard time getting how the POW consensus works and then they hear about the checkpoints and say "Ah ha! this is how its secure!", after all people are familiar with centralized security so this is just a lot easier to understand.

The big damage from that comes because we're then subjected to a non-stop stream of IDEAS and IMPROVEMENTS from people who's understanding of the security model was broken by the mere existence of checkpoints, even though checkpoints do nothing relative to consensus. These people propose things which depend on a centralized security model and don't have the benefit of the checkpoints being set thousands of block in the past so they are never a practical influence on the consensus, and don't spend the mental cycles trying to fit their ideas into the real, primarily zero trust, Bitcoin security model. As a cautionary sign of where that could be going: There altcoins which user developer controlled checkpoints broadcast to checkpoint blocks in real time, and more altcoins being added that do this... as if this were an acceptable way to construct a decentralized cryptocurrency!

When headers first syncing is merged, just by adding a "must be this tall" minimum sum difficulty check we'll be able to remove checkpoints for all DOS purposes, and we'll also be able to remove them for syncing acceleration (using random sampling for ECDSA in the deeply burried chain).

Checkpoints as a cognitive security blanket against doomsday reorgs will likely stay, but we could update them less frequently, with a more regimented process, and perhaps change how nodes respond to them (e.g. a conflicting chain becoming longer making all nodes go into a safemode where they regard everything new as unconfirmed, process no transactions, and discourage any new blocks that process transactions allowing for a public resolution of the "impossible event", rather than directly coercing the consensus).

Well he seemed to think the fact that they are included at all = the death of Bitcoin.  I was just providing a way for him to "save" Bitcoin from the ebil developers. 

Yup, I just mentally corrected TierNolan's proposal to match ones made by many people before— have a minimum total chain work checkpoint (e.g. any valid chain will have at least 2^70 work by height XXX). That allows the same protection from spamming as the current stuff.

Uhhhhhh. Or just run with the checkpoints=0 command-line/config option.

You are free to use and distribute a fork of the client without any checkpoints.  Checkpoints aren't required. Assumming an attacker isn't attempting to DDOS your node by filling it with years worth of bogus blocks or try to perform a 51% attack months deep in the chain the lack of checkpoint will have absolutely no effect on your node.

So if you are worried then remove the checkpoints, compile it and use that node.  If you are really worried set up an online mirror and advertise your checkpoint free version.

See peer to peer at work.

Haven't there been periods where the difficulty went down?

Then you pick among them like normal, the point there is to close of the denial of service attack without undermining the consensus algorithm with an element of centralization.

And then what if multiple chains are above this min pow?

Instead of checkpoints, client could hard-code a MAIN_CHAIN_MIN_POW variable.  This would be the PoW of the main chain when that version of the client was released.  All clients wouldn't need to agree on a value.  It is just a spam protection value.

Clients should download headers first and only commit blocks to disk that are on a chain that has PoW higher than the hard coded minimum.

Clients should respond to "inv" messages containing unknown block hashes by sending a "getheaders" message rather than asking for the block.

If the block extends your main chain, you will be sent it.

Peer sends: "inv" containing unknown block
Send to peer: "getheaders" with standard block locator (send at most once per inv message received)
Peer sends: "headers" with new blocks on it's main chain
Add these headers to header tree
If the main chain is extended and the main chain has PoW greater than MAIN_CHAIN_MIN_POW, request those full-blocks


No matter what happens, the longest chain will always have more PoW than the longest one when the client was writen, so the best chain is guaranteed to have more PoW than MAIN_CHAIN_MIN_POW.

The client won't consider itself synced unless it is on a main chain with PoW that exceeds that value.

Well, the genesis hash was coded in from day one.  If you want an alternate chain, you need a new genesis hash, plus you'd have to clear out the checkpoints.  (My apologies if that wasn't what you meant, I'm not entirely sure that I understand you.)

The early blocks had very low difficulty.  If I wanted to, I could generate thousands of low difficulty blocks, with appropriate fake timestamps, and I could set my node to wait for new connections, then feed those bogus blocks to the noob.  They'd figure it out eventually, but I could keep them mired down for hours or days, working on my garbage.  The checkpoints give a handy shortcut to that "eventually".  If my chain doesn't include the checkpoint blocks, the new node won't waste their time on it.

The notion of "the correct chain" gets fuzzy.  If I have a lot of hashing power, like more than the rest of the world, and I start today making a new chain, but I keep my farm isolated from the rest of the network, is it "the correct chain"?  The only really hard rule is that the longer (more work, not necessarily more blocks) chain is correct.  My chain would be (work) longer, so in that sense, it is correct.  But no one knows about it but me, so the world would be working on a chain that they think is correct.

When I release my chain, it will overturn possibly thousands of blocks, possibly millions of transactions.  The hard rule says that it should happen, but almost no one would actually wants that outcome.  A checkpoint could invalidate my whole chain, if a big part of the network upgrades to a client that includes a checkpoint that happens to come after I started.

That situation is unrealistic, intentionally.  But it shows that the simple rule that we can easily write into our software is incomplete.  We humans want the rule to be "the longer chain is correct, except...", and the "except" part is hard to write.  We know that checkpoints aren't the right way to codify the exception.  If you've ever talked to Gavin, or even read his posts on the subject, you'll know that he absolutely hates that he could end up in the position of picking which chain wins.

So, we end up in the strange situation we are in.  The checkpoints are useful in the real world , but we can see that they also hold a power that no one actually wants to wield.  The devs advance them now and then, but always sorta grudgingly, and always months behind.  If someone released a 5000 block attack chain, we'd all be bitching that the checkpoint was too far behind.  Because no one has (and probably never will) hide a fork of that length, we are all bitching that checkpoints are contrary to the bitcoin design.

Also see:
https://en.bitcoin.it/wiki/Vocabulary
http://bitcoin.stackexchange.com/questions/1061/can-a-51-attack-be-detected-and-dealt-with/1065#1065

So you are saying that checkpoints don't actually prevent my node from downloading, storing and routing an alternative block that would refer to the genesis?
Or are you saying that they do prevent it, but the important thing here is that I should care more about those who are just downloading the chain and dont know yet which branch to choose?

Your solution is indeed simple.  But it doesn't do the same job that checkpoints do.  Checkpoints are not about forks in the present or future, they are about helping clients avoid dead end forks in the past, while they are doing their initial sync.  Checkpoints help new nodes catch up to the network with minimal annoyance for the node operator, nothing more*.

The real replacement for checkpoints is a smarter sync system.  Greg is working on a smarter sync (see here), but I don't think it has been implemented or tested yet, so for now, we are stuck with the dumb downloader and checkpoints as a sanity check.

* They can also be used to rectify a fork in the network, but this is an extraordinary usage that requires a lot of manual intervention by a lot of node operators.  When a signed/unsigned bug was found in the value interpretation code and someone used it to give himself billions of bitcoins, a checkpoint was used to help the corrected chain overtake the errant chain that had grown for several hours before the fix could be applied.  It was also used to unroll the more recent BDB/blocksize fork.  But again, these were not checkpoints already built into the code, we just used the same mechanism as a manually override.

if you disagree with the use of checkpoints the just fork 'em.

I wish I did, but as you figured, I didn't.
I'm a man of simple solutions.
Checkpoints that are there now, in the bitcoin client, are stupid.
My idea - may be simple, but at least it would do what it has to do and close the problem once and for all. And it's surely much simpler.
Your idea - I believe it's impressive, but why to complicate things that are simple and do the job?

I'm glad you managed to dig up and read my proposal for exponential difficulty reorgs, and that you find it superior.  Not surprising, considering that yours is a degenerate case of mine where the exponent is infinite.

But it really isn't a checkpoint system.  The checkpoint system is all about the past, while both your proposal and mine are useless in that domain.

Yeah, I'm sure your checkpoints idea is much better.
I'm actually even ashamed mentioning something such stupid

No, that sucks.  Sorry.

I understand that they violate some purpose of bitcoin.
But I am also a pragmatic person and so I prefer to have a system that violates some very unlikely to happen purposes, as long as it is a way to keep my resources away from spammers.

But your concerns are 100% theoretical and it isn't really possible to start a fork like 2000 blocks back and get with it to the top.
And even if it was possible, would you really want it? Because I would not.

Again: the node, and the network, must protect itself from DoS attacks.
What are you even trying to achieve forbidding people doing this?

If you one day manage to create a branch that would beat their chosen checkpoint - then I will agree with you.
But so far, I don't get your problem, though I appreciate your innocent ideas.

Do you understand that checkpoints completely violate the entire purpose of Bitcoin?

What they do is introduce a form of approval(consensus) about the state of a chain.  This form of authority could certainly be abused, and it runs counter to practically every design principle of Bitcoin.

at the most fundamental level, the client code should not have ANYTHING to say about the block chain.  Whether this is required in order to operate is irrelevant.  The reason people use Bitcoin is *because* it's peer-to-peer.

I think you don't understand.
Checkpoints are not there to fight proof of work - they are there to protect your node from attacks.

Do you understand that if there were no checkpoints, anyone with a graphic card could just keep mining blocks that link to the genesis one, and this way fill up your hard disk? Also stuffing the net with useless data.

the fact is, if we've introduced a form of consensus OUTSIDE of proof-of-work, that overrides Proof-Of-Work, and that's precisely what checkpoints are- why do we have proof of work at all?

can we remove proof of work, and have a block chain that is based on consensus between N parties?  Yes!  https://docs.google.com/file/d/0BwUFHE6KYsM0ZkxLVmFwbXQ3ck0/edit?usp=sharing

-bm

I can come out with a really good solution, if you want

Just make a rule that the client would not accept any forks deeper than N blocks back from its currently known top.
Set N to whatever constant you want, but I think more than a week time would be exaggerating.

 What do you think 'Peer To Peer' means?

 as in "A Peer-To-Peer Electronic Cash System"?  http://bitcoin.org/bitcoin.pdf

I'm having a hard time figuring out what you mean by "equality of all nodes".  The equality (or inequality) of nodes is not part of the system.  No one cares about nodes.  We care about the chain.  If a node has the correct chain and can give you a copy, it is "equal" to all of the other good nodes.  If it doesn't, or can't, then it equal at all, isn't even a node.

The checkpoints do not provide any security, nor were they ever intended to.  They just keep "unequal" nodes from wasting your time with bogus chains (whether by accident or malice).

Also, you are free to edit and recompile as needed.  You can strip them out entirely, or add your own, or change them.  Whatever you want.  The reference client is written for (and compiled for) working well for a wide audience.

Technically, a longer chain is the best chain, regardless of the checkpoints.  In practice, the checkpoints are far enough in the past that if a new chain showed up that overturned one, we'd almost universally consider it an attack, and roll back.

P.S.  No one particularly likes the checkpoints, the devs least of all.  There has been much discussion about removing them, or switching to a soft system (config file, etc).  So far, no one has come up with a really good solution.

yeah, I agree with you that it goes against an important principle, but as gmaxwell said, it does have some actual purpose, protecting your storage from being flooded by forks that would suddenly start to appear at low block numbers.
though, it does not make any sense to have multiple checkpoints (instead of i.e. a one that moves by itself), but well... who are we to argue with a solution chosen by the best bitcoin developers that satoshi left behind?

well that's a practical solution, but it violates the notion of 'peer-to-peer' or equality for all nodes in the network.

there is no indication of 'checkpoints' in the satoshi whitepaper.

furthermore, if we rely on such checkpoints for security, why do we even have proof of work at all?  Why don't we just rely on the checkpoints?

not really. they only put checkpoint for blocks that are buried at least few hundreds deep already - unlikely to be reverted by honest means.
but if you disagree with any checkpoint they add, just don't upgrade your node and you will be fine.

so ultimately, the state of the block chain depends entirely on them.

The bitcoin core devs

who makes the checkpoints?

Yeah, with "just mined" I thought you mean "mined the same day" or something like this - usually a block a (few) thousand blocks deep is selected.

This part?

I'm trying to understand the whole file, I'll give it another good read tonight since I'm at work now ;-) but what do u mean by not just mined? I meant there can never be checkpoints for blocks not yet in the blockchain..

Not just mined, please read the comments in that file too. Generally yes though, a new client release usually also includes an updated checkpoint.

So the checkpoints are added afterwards? With every new version of the client a checkpoint of a block that was just mined is added?

Thanks

Thats not really what they're for— though that have the effect too. Most of their usefulness is that they prevent a dos attacker from filling up bitcoin node's disk space with long runs of low difficulty blocks forked off low in the chain.

e.g. you start off with difficulty 1 blocks at block 0, now mine-able by the millions by a single asic— _MAYBE_ a chain that starts off that way could eventually turn out to be the longest so absent the checkpoints a node would happily follow an endlessly long chain of them.

They also make is so that an attacker who has complete control of your network (and thus can prevent you from hearing the longest chain from the honest bitcoin network) from putting you on a fake (low difficulty) isolated chain unless they can also trick you into running replaced software. With the checkpoints such an attacker hast to have a ton of mining power in order to continue the chain.

A powerful quantum computer couldn't automatically change the rules of the network such as 1k-coin blocks.  (If it could out-mine the entire network I guess it might also be capable of cracking everyone's ssh keys and passwords and hacking their system to replace bitcoin, but leaving that out, it can't change the rules.)

So they are added gradually as we reach there, this can only protect tampering with the existing data, there's no way to prevent future data tampering. Am I right? If the quantum computer comes out, then it just need to take the current blockchain, then change code and make new executable, by doing something like each block generate 1k coins, for example, and attach the new ones in the chain. This will overwrite the existing chain, though no previous data (checked by checkpoints) will be changed.

Yes, you are correct.  They are there so some quantum computing farm (doesn't exist, but...) can't come out of nowhere and roll back blocks by mining a long chain from far in the past.

The whole point is to thoroughly check all the hashes from block 0 to today, that is how it's secure.  It only has to check all of them once though, and then trusts that nobody will tamper with the disk data to invalidate that check (safe assumption, I think.)

I am reading bitcoin codes, and I don't quite get what are checkpoints in the bitcoin code:


Are they hashes of the blocks at 11111,33333, etc positions? If so how can we precompute them, as going from 1 to block 225430 would take big amount of time of computations...