Databases | Bitcointalksearch.org

samson

legendary

Activity: 2097

Merit: 1070

Quote from: hack_ on November 24, 2015, 04:29:33 PM

Quote from: domob on November 24, 2015, 04:17:56 PM

Quote from: achow101 on November 24, 2015, 04:14:13 PM

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea. At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

Ok, the backwards compat makes sense up to a point....but i should want to believe that instead of full compat, why not just update/ move to a better version and maintain a function that can import the data from old version?

I know at least one alt that is already testing SQLite in a relational model, but if it really is faster, i'll tip the dev and maybe we can have a case study for the key/data storage model.

What is wrong with SQL ?

It's not fully buzzword compliant if it doesn't use NoSQL. SQL just isn't 'trendy' enough any more.

There's nothing wrong with SQLite for what we need it for.

When I started using databases some 20+ years ago it was all NoSQL.

hack_

hero member

Activity: 501

Merit: 503

Quote from: DumbFruit on November 24, 2015, 05:06:32 PM

A conversation was had last month, about changing from LevelDB, and the gist of it is; it is fast and it works for what Bitcoin needs it to do.

Quote from: GMaxwell

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for. Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data. They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.¹

BerkeleyDB is used to manage the majority of data and log files.² LevelDB is used to store the "chainstate" (Including UTXO) and the "block index".³

As to why one database is used for some things as opposed to others, it's certainly because they are designed for different purposes. BerkeleyDB is slower, but scales well, is robust, and supports read/write concurrency. LevelDB has a very limited feature set but is several times faster than BerkeleyDB⁴.

¹http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011633.html
²https://en.bitcoin.it/wiki/Data_directory
³http://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair
⁴http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html

Edit: Fixed wrongness, I think I have it right now..

Thanks

droark

sr. member

Activity: 525

Merit: 282

Quote from: domob on November 24, 2015, 04:17:56 PM

Quote from: achow101 on November 24, 2015, 04:14:13 PM

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea. At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

This. There's been talk, and I believe Jeff Garzik made an experimental fork with a new DB, but I don't think this is being taken seriously just yet. I'd love to see a switch to LMDB but the lack of real 32-bit support appears to make it a non-starter.

DumbFruit

sr. member

Activity: 433

Merit: 263

A conversation was had last month, about changing from LevelDB, and the gist of it is; it is fast and it works for what Bitcoin needs it to do.

Quote from: GMaxwell

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for. Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data. They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.¹

BerkeleyDB is used to manage the majority of data and log files.² LevelDB is used to store the "chainstate" (Including UTXO) and the "block index".³

As to why one database is used for some things as opposed to others, it's certainly because they are designed for different purposes. BerkeleyDB is slower, but scales well, is robust, and supports read/write concurrency. LevelDB has a very limited feature set but is several times faster than BerkeleyDB⁴.

¹http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011633.html
²https://en.bitcoin.it/wiki/Data_directory
³http://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair
⁴http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html

Edit: Fixed wrongness, I think I have it right now..

hack_

hero member

Activity: 501

Merit: 503

Quote from: domob on November 24, 2015, 04:17:56 PM

Quote from: achow101 on November 24, 2015, 04:14:13 PM

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea. At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

Ok, the backwards compat makes sense up to a point....but i should want to believe that instead of full compat, why not just update/ move to a better version and maintain a function that can import the data from old version?

I know at least one alt that is already testing SQLite in a relational model, but if it really is faster, i'll tip the dev and maybe we can have a case study for the key/data storage model.

What is wrong with SQL ?

domob

legendary

Activity: 1135

Merit: 1166

Quote from: achow101 on November 24, 2015, 04:14:13 PM

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea. At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

achow101

staff

Activity: 3374

Merit: 6530

Just writing some code

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

domob

legendary

Activity: 1135

Merit: 1166

BDB (particularly the old version) is only used for the wallet file since 0.8 (I think), which is done for backwards-compatibility. In other words, the wallet format was never changed so that people could still access their old wallets (which kind of makes sense).

LevelDB has replaced BDB for performance (and maybe other) reasons for everything else, i. e., the actual "blockchain data" relevant for operations (block index and UTXO set).

hack_

hero member

Activity: 501

Merit: 503

Why do we still use an old version of BDB ? Also why do we use both BDB and levelDB?

Topic: Databases (Read 1045 times)