Topic: The MAX_BLOCK_SIZE fork - page 2. (Read 35600 times)

Argh... BDB doesn't allow for an unlimited number of locks? (how many your system can handle, that is)
It really needs to specify a limit?

The number of locks required depends not only on the number of transactions in the block (N) but also on the size of the blkindex.dat (S). Even if you have a fixed upper limit on N, the S grows without an upper bound.

The most common requested lock is "read with intent to write". As the process descends down the B-tree it takes one of those locks for each page traversed until it reaches the leaf page. Then it promotes the last lock to the "write lock". So the number of locks required to insert single Bitcoin transaction is O(H) where H is the height of the B-tree. This is turn is O(log_b S) where log_b is number of k,v-pairs per page of the B-tree.

So for a single database transaction which consists of all Bitcoin transactions in the block the number of locks required is O(N * log_b S). This number has no upper bound, it logaritmically increases with the size of blkindex.dat.

I'm sorry I can't give a good reference. It is in some of the Springer's series "Lecture Notes in Computer Science", something related to concurrency and deadlock detection/avoidance in database systems. The only thing I remember now is that was one of the thickest LNiCS volumes on the whole shelf with LNiCS.

Yes, my point was that we definitely cannot wait until the next halving day to kill this beast, or it will kill us instead.

The problem with BDB is actually worse than that... 5k database transactions isn't necessarily 5k bitcoin transactions. BDB is a monster that needs to be killed with fire.

It is not. It's been worked around. But we still have this bug limiting blocks to 5k transactions top, when they should be able to handle more.


I don't see other way around it.


hehe, dude, we'll have to abandon BDB in a few weeks. Or at least abandon this configuration which limits it to 5k transactions. We're already bumping on it. The only reason we had this fork was precisely because 5k tx per block is already not enough sometimes.

This simply shows yet again that, sorry, the implementation is not and cannot be the specification.

That functioning in the heterogeneous wild might be a "hard problem" might suck, but probably is really a stronger call for standards and specifications than functioning in the land of fairies and unicorns might be.

The more sure it is that we'll never never actually be able to be fully up to spec, the more important it probably is to actually have a spec to aspire to.

If we don't even have "second star to the right and straight on 'til morning" things might not bode well, but if we do have it, ahhh, then, the land of never never might not actually be so very bad.

So please, lets just admit we aren't up to spec and have never been up to spec, and focus on getting there instead of pretending that "the world is after all perfect, if requiring a little adjustment" is a bastion of software engineering.

(Just because it is true doesn't mean it is computer science. )

-MarkM-

Per operation is difficult to calculate. I don't know much about how the bitcoin code interacts with the BDB, so much of what I am going to write can be taken with a grain of salt. In the next few weeks I plan on printing out chunks of the Bitcoin source code and trying to understand how it works with the Database as a personal enrichment exercise, and I'll share anything I find that might be useful. I can't promise anything material, but when my plate clears I can try to make an effort (for whatever all of those qualifiers are worth).

Here's Oracle's rather worthless documentation of locking and blocking in BDB. Normally there's one lock per database transaction, but it can get messy because of lock contention. What it sound like you are looking for though is better documentation. Such a thing might exist but Oracle...

A real max block size fix is probably going to have to involve picking a distant future deadline and hardforking things over to LevelDB. This might involve the painful experience of making a LevelDB version of 0.3.x to accompany newer clients, but LevelDB and BDB are different enough in their quirks that leaving BDB for LevelDB will probably mean abandoning BDB wholesale. How many years have gone by without a good WordPress port for PostgreSQL?

The deadline to move to a LevelDB bitcoin client might have to be set in the very distant future, maybe even at the next halving day. BDB isn't sufficiently well documented to the point where its quirks can be accounted for in any other database software for all but simple cases (Oracle probably likes it this way and BDB's is locking contention is probably a selling point for their other database products), and if Bitcoin has demonstrated anything this far into its life, Bitcoin attracts fringe use cases.

I don't believe he does Mr. Andresen.

But now that the bug is resolved, will there still be an update to allow increased block sizes? I'm not understanding how the team intends to allow older versions to accept larger blocks from 0.8+.

Will a full-scale 0.8 update be forced?

If Mr. Augustus found some documentation that I don't know about it, I genuinely want to know about it, because it will save me time. Right now I'm throwing blocks at an instrumented v0.7.2 bitcoind that tells me how many locks are taken, so I can be sure whatever fix we implement will always work.

Now, now, Gavin. Play nice. ;)

Ah, excellent, can you please send me the documentation that says exactly how many locks will be taken by each bdb operation?  I haven't been able to find that.  Thanks!

I'd still call it a bug. Actually, it's still an unsolved bug. We've only worked around it, but it's still there.

It's worse than that. If < 0.8 implementations all behaved the exact same way you could call their behavior a de facto standard, but since their behavior was not consistent between nodes of the exact same version you can't even say that. The behavior of all implementations prior to 0.8 with regards to the protocol is non-deterministic.

A non-understood limitation of an implementation dependency does not define the protocol. To the Bitcoin protocol, blocks up until 1Mb are allowed. That was the consensus, that was what every documentation available said. The <= 0.7 Satoshi implementation wasn't capable of dealing with such blocks. That implementation was bugged, not the 0.8.

Incorrect.  0.8 does not use BDB for blockchain indexing.   Those BDB settings you quote are only relevant to the wallet.dat file in 0.8.

The BDB lock limitation simply does not exist in 0.8, because leveldb is used for blockchain indexing, not BDB.

Weird, 0.8 uses leveldb not BDB doesn't it?

Does leveldb use those same calls to set its configuration?

-MarkM-

Removed incorrect, thnx Jeff.

I was told on #bitcoin-dev that actually the devs have met the BDB configuration numbers before, and to look at db.cpp to see where in the bitcoin code they explicitly set the numbers they want BDB to use.

Also, they they ran into problems with it before.

So supposedly they were not unaware that BDB can be configured. They even confided that the page size BDB uses is by default the block size of the underlying block device (the disk drive, for example).

So from the sound if it they simply had not set the configuration numbers high enough to accomodate all platforms, or maybe all possible sizes of blockchain reorganisation. (During a re-org, apparently, it needs enough locks to deal with two blocks at once in one BDB-transaction?)

-MarkM-

How can it be a bug if it is a clearly defined behaviour in the documentation of the s/ware dependency?

The fact that the devs (or anyone) seems to have never read the documentation of the standard dependencies is more the worry, in my opinion.

Come on, such obscure limit was not known by anyone in Bitcoin-world up until it blew yesterday. You may claim it's not a bug on BDB side*, what's arguable, but it is definitely a bug on bitcoin implementation side.
Everybody should be able to handle blocks up until 1Mb. That was the general agreement, the protocol spec if you will. The particular implementation of Satoshi client <= 0.7 was not capable of following such protocol specification as it should. 0.8 onward was. If anything, 0.8 is the "correct version". Bringing everybody back to 0.7 was an "emergency plan" since pushing everybody to 0.8 was believed to be much harder to accomplish (and likely truly would be).

* And bug or not, the fact that nobody here even knew about it just shows how much we cannot rely on BDB - not a single person among all the brilliant minds on the core dev team understands fully how this thing works (and neither did Satoshi) .
Moving out of BDB is certainly a desirable thing. Now with this even more crippling block size limit, it's pretty much urgent.