Topic: Alert: chain fork caused by pre-0.8 clients dealing badly with large blocks (Read 155551 times)

Yes on all counts
The redundancy parts especially.

Ok, so I think you're agreeing and being nitpicky.  I like that!  So I'll join you:
"not by the government and not via coercion" is redundant.
That being said, redundancy is important, especially in the bitcoin world! 

That is semantics. Centralization explicitly refers to the centralization of power in the hands of governments through legal coercion.
That being said, semantics are important...
That being said, the whole concept of p2pool is to further avoid "centralization" of power done not by the government and not via coercion.

Everyone can be a prophet afterwards

IMHO, I think coercion is too strong a term. BTC Guild had some 25% of the hashing power and 60% recognized the large problem block. So it was up to BTC Guild alone to decide what they wanted to do, and they obviously gave weight to the opinion of core dev. This was an unusual situation resolved by actions which did not strictly violate the principles which support #3.

Ultimately, I think centralization is not the essence of what those of us who want to avoid it actually loathe.  We loathe coercion.  After all, for whatever reason, we have violated directive #3 (I think - right?  That was Stephen's example, right?) because, through some centralization, the community was able to reason with miners and thereby convince enough of them to switch to a broken client in order to save the masses of bitcoiners still on that version, unable to validate the good block that forked them over.  There is a kind of coercion in that, but it depended on the cooperation of the miners and isn't something I would call coercion because no rights of anyone were violated or threatened to be violated.

I suspect for the most part no, if such corporate entities are registered / listed so governments can go knock on their doors.

It is similar to how having a list of people who hold gun licenses impacts decentralisation of guns. First it lets them go knock on doors, later it also can become a moat preventing "undesirables" from even being allowed a license, eventually those who do have a license welcome deeper and deeper moat to keep competitors at bay and so on.

Meanwhile big brother knows where they all live.

Except the "undesirables" who to the extent they continue to exist and use the technology are the real bastion of decentralisation.

-MarkM-

Perhaps the perceived degree of support for #1 is really the question of what defines centralization (conversely decentralization):

Consider 10,000 nodes of which 1,000 are mining, all owned by individuals. Does that have the same decentralization as 10,000 nodes (1,000 mining) owned by companies, organizations and co-operatives?

Love the graphic (enough to break my rule of thumb about not re-posting them through a quote...)  Two things though:

1) I don't sense that everyone agrees with directive #1.  Or at least the degree to which it is important varies widely even among those who can conceptualize high traffic systems and the magnitude of various economies.

2) The elephant in the room would be directive #4.  You know...the 'classified' one.

Theoretically it could have. I don't know how it would work out though. Would the generator have a BDB error when trying to persist his own block and then abandon it, or would it manage to attempt a propagation and then get rejected by everybody else?

That said, the demand for such large blocks is recent. Probably the event of generating such problematic block never happened before simply because there was no demand for it.

What the miners did by switching their hashing power from the 0.8 chain to the 0.7 chain is intentionally abandon the correct chain in order to allow users with version 0.7 to continue using their software without seeing what amounted to invalid confirmations (from other miners who were already on the 0.7 chain, and erroneously ignoring a valid block).

I don't see it as a serious technical flaw.  There is a serious technical flaw in 0.7, and miners now know how to avoid it.  The deeper problem to contend with is that people can use old software that can provide confirmations to possibly invalid transactions if they are added to a block by a miner after there is another hard fork.  So just upgrade to 0.8, or, as you say, baby-sit so you know when every block is getting orphaned.

Yes, baby-sitting.  Setting things up and then ignoring them because you believe they are perfect is dangerous and foolish.  It's the ease with which we fix things that is important.  Convention says that the 0 in 0.7 (and 0.8 ) means that the bitcoin-qt reference client is still in beta.

I have a more administrative idea, which is to piggy-back some kind of notice on the peer discovery work that the client does.  The notice would generate a list of "problems that are less prevalent in the other side's version" so that once a client gets a number of peers telling it about an important problem that will be 'less prevalent" in a different version, it can inform the user.

As  community, we have accepted miners colluding (however openly) to abandon one chain in order to accommodate a flaw in a client in widespread use.  Any powerful interests threatened by bitcoin might start working on exploiting such flaws.  If this bothers any merchants, their software can watch for consecutive orphan blocks as that would be a sign that currently valid blocks might not contain all valid transactions, and pause for user input.  In other words, we can make the software do some babysitting.

Ok, but those last two ideas are just trying to get around the fact that we should be babysitting anyway.  And we are.

Nope.  Those v0.8 clients simply added those transactions from the orphaned block back into the memory pool.  It does not re-broadcast them.  I don't even think the node that sent the transaction will re-broadcast the transaction immediately upon the block being orphaned -- it probably gets treated like normal, at some random point (e.g., over the next half hour) it will re-broadcast it because it has no confirmations.  [Edit: the sending node was a v0.7 node so it had no concept of orphaned v0.8 blocks, so it should've just simply re-broadcast periodically until confirmed.]


Take the 211.9093 BTC (~ $10K) transaction to OKPay (12814b8ad57ce5654ba69eb26a52ddae1bff42093ca20cef3ad96fe7fd85d195).  It had been broadcast and then included in the v0.8 side in block 225,446 (00000000000000757de9173ee2f01c9f957c8aa3b4f71b901b0fa19683ab1fa1) but did not confirm on the v0.7 side, presumably because the transaction had inputs that hadn't even confirmed yet on the v0.7 side.  So on the v0.7 after the transaction had been broadcast and relayed you had new v0.7 mining nodes starting up.   But they don't pull transactions from peers, they only listen for broadcasts.

So that's a normal situation where these new v0.7 mining nodes are ignorant of a double-spending race attack potentially being attempted.  Only BTC Guild knows if and when that node had received (12814b8ad57ce5654ba69eb26a52ddae1bff42093ca20cef3ad96fe7fd85d195) and if it had then why their node (running v0.7 by then) no longer knew of it when it mined the double spend (762443f6373b7c8b3833d4ad23578fc3099cc29b86d1359d0c0565e3c8614f91) more than six and a half hours later.  It could be that the node was just spun up and the memory pool just hadn't received that transaction yet.  I might suspect that the memory pool for that node was filling and as a result was housekeeping by flushing older and lower priority transactions out.  This was an intentional double spend (even if the losses to the merchant were repaid voluntarily by the "attacker" at a later time) and was carried out by running a script to broadcast the double spend continuously every ten seconds.   The initial transaction was submitted through Blockchain.info (though I'm not sure if that was as a raw transaction through their pushtx or if it was through their API (or normal web interface maybe even).  It is possible that it was a fire and forget action, where the transaction wasn't re-broadcasted ever again.

What might have been useful is if there was some hard fork recovery approach in which the memory pool was initialized using the list of transactions from a set of blocks (the ones that would be orphaned on the v0.8 side).  If any blocks arriving had transactions that included a double spend of the recovery transactions those blocks would be ignored.  This mode would continue until all transactions from that initial set were either included in blocks or rejected for being invalid.  Then the client would switch back to normal and bring in the transactions that had queued up while the node was in the recovery mode.  As long as enough hashing capacity followed this "deferential recovery" mode the recovery might have been nearly as quick and no merchants would have had confirmed transactions on the v0.8 side that ended up not confirming on the v0.7 side when the transaction was valid on both.  That's not a violation of the protocol but instead is just a manual override mode that would help protect Bitcoin's prime directive #3 -- transactions having six confirmations or more on the longest chain are never reversed.

So the bitcoin technology still has a possibly serious technical flaw. It cannot cope well with certain software defects.

I don't know the technology very well. I don't know what reorg means, for example. I would have hoped that bitcoin clients can automatically detect a fork and decide for one of the branches. If I understand things correctly, that does not work in this case, because the 0.7.x clients reject and thus do not take into account a branch that begins with a block they consider illegal.

If I see this correctly, then there is no easy and obvious solution to this problem, except perhaps to make the block structure so simple that provably error-free software can be produced to handle it.

The only other solution I can see is baby-sitting by the developer and miner community. It has worked this time; perhaps it will work similarly well for the whole future of bitcoin.

This question fascinates me.  Both chains would continue getting built, and everyone who pays attention would have to be really unobservant in order not to notice that every block eventually has an orphan version.  But let's say we all got stoned and didn't notice.  Then what?

As long as enough miners stayed on the 0.8 chain that accepted the large valid block, that chain would be the valid chain (but only according to the clients that accepted it).  0.7 clients would see the other chain as valid and the network for all clients would seem to be getting spammed by a lot of invalid blocks.

How many orphaned blocks will the reference client remember in case a reorg comes up?  I'm guessing all of them?

If enough miners were on the 0.7 chain, it would eventually catch up and do a reorg (as actually happened).  This would force all the transactions that "used to be good" but no longer are to get re-broadcast to get into the new chain.  There's another question:  Wouldn't most of them be in both chains already?

No such thing exists, and probably never will exist.  BDB doesn't have a reliable formula for calculating how many locks a given transaction takes.  Even BDB isn't consistent with itself..  When it rejects a block at runtime, it is capable of processing it a second time around when the memory state is different.


0.8 is already like this.  By default, neither 0.7 or 0.8 will generate blocks that 0.7 will reject.  Both 0.7 and 0.8 can be configured to make blocks that will break 0.7.  Given the right luck, a block could be generated that broke some 0.7's and not others.

The fork happened because somebody used the -blockmaxsize setting to change its behavior and create valid, full sized blocks.


There is no silver bullet.  The only robust solution is to have people either:
1) update to 0.8+ (except for the large mining pools).
or
2) update to a (yet to be released) fixed 0.6 or 0.7 that increases the lock tables such that this will never happen again
or
3) manually raise the lock table limits, via DB_CONFIG.

Once the merchants and end users have had time to do their updates, then the mining pools can fix their end.

People who are mining as part of a pool should still update.  What matters is what their pool is running, not what you're running.

0.8.1 can't fix this problem. The mining pool hashpower is holding back to buy people time to upgrade.

I can see no other fix than not including these transactions in the good chain. Thus, all related transactions would vanish from their recipient's balance, regardless of the level of taint.

This is quite radical, so I hope it doesn't happen.

...see explanation 2 posts above. Point is, not only the reward itself, but every transaction which is based on this reward can not be re-verified on the other chain. Thus, this can evolve into an avalanche, given enough time passes after the split

Someone correct me if I am wrong, but AFAIK transactions are not lost, they would be duplicated on both sides of the fork.
Its just that whomever mines on the losing fork ends up losing all their mining work.
Transactions could appear delayed for a few hours though.

In retrospect you are right.
The best solution would be to make a new release that does both A and B concurrently.
That is, make a version of 0.8.1 that:
A. Has a block rejection algorithm that will reject non 0.7 compatible blocks - must be programmed first as such a thing does not exist
B. Is configured to not generate non 0.7 compatible blocks - functionality exists, it just needs to be configured.
Both of those can be set to expire after a certain date.

This would make v0.8.1 superior for mining than 0.7 because even though it rejects the same blocks 0.7 does, it has the advantage of not ever generating such blocks (which would be orphaned). Mining on 0.7 is thus more risky as there is a chance that you will generate a block that will be incompatible and orphaned.

Reorg will work itself out, unless a chain has been "pinned" by one of those hard-baked Checkpoints done by the Bitcoin devs, to make the "very ancient" history definite and unchangeable. Regarding what counts as harsh consequences -- well that is debatable.

Whenever someone spends coins on a branch, and that branch gets superseded, these transactions cease to be valid, so the receiver's balance ceases to exist. Now, if both branches are "basically compatible", the now unconfirmed transactions could be replayed in the new and now valid branch, thereby bringing the ceased balance of the receiver back into existence. (The reference client does this re-verification automatically, if I am informed right)

But this re-confirming of obsoleted transactions from the orphaned chain can not be done under certain circumstances:

• when a transaction uses (even indirectly) a TX output which has been spent on another transaction on the new chain
  (this would be a "double spend")
• when a transaction relies (even indirectly) on a TX output which does not or can not exist on the new chain.
  These would be the money base (block reward) transactions on the old chain since the split.

The second point explains the significance of the (roughly) 100 block limit.