Topic: fuck you wallet format!!!! (RANT) - page 2. (Read 3194 times)

You know what would be really nice down the road is if the database layer was abstracted in a way that allowed someone to use their own SQL server as the data store for the block chain, but defaulted to something statically linked for the sake of slimness for those who won't use this feature.  Likewise, the "in-memory transaction pool" also ought to be maintained as a database table.  This would help with the development of other applications (like payment processing, shopping carts) without requiring them to hack or tangle with bitcoind.  Likewise, I'm sure someone else has thought of this first.

I think you're preaching to the choir. I don't think many people consider bdb for the wallet a good idea. I've already experimented with an own format to function as drop-in replacement for the database file. It doesn't look too hard, but obviously compatibility issues will complicate things.

These benefits have not been brought to Bitcoin by the choice of using a database to hold the wallet.  While I agree that a real database system should be used for a mission critical database, I don't regard a bitcoin wallet as a database.  I see it as a document, like an excel spreadsheet.  Using a database engine for a bitcoin wallet to me is as ridiculous as using a power drill screwdriver to take apart a pocket watch.

In my view, the wallet (containing keypairs) should totally be separate from the file that contains the transaction history.  You don't use your real wallet to keep all of your receipts for everything you bought since you were twelve, neither should Bitcoin do the same.

Atomic transactions: The wallet should be nothing more than a repository of keypairs, so the only transaction that should ever need to take place is the occasional topping up of the keypool, and perhaps saving labels attached to addresses.  (For example, a keypool created with 500 keypairs, and topped back up to 500 when the reserve reaches 100, would make transactions and/or modification of the wallet happen so rarely that it would hardly be inefficient to achieve atomicity simply by writing a brand new file, swapping it with the old one via renaming, and then deleting the old one, as though it were a document.

Read/write verification: Nothing unique to a database - if verification is needed, this is easy to code in the client.  Write file, close file, read file, verify file (or hash).

High level of availability:  How does bdb increase availability as compared to say a flat xml file?  If anything, it has hindered it, judging by the number of people who have gotten "critical DB errors" and lost access to their wallet.

Data loss recovery:  If anything, people have lost more wallets as a result of bdb than have ever gained from it.  At least with XML you can manually hack together in a text editor.  bdb, no way.

Backups: Another place where bdb has offered no benefit, and the unusual placement (especially on Windows clients, as compared to the typical Windows user) has made backups difficult for the average user.  Rather, the client should treat a wallet like a document, the same way Microsoft Word treats a .doc file as a document, where a user can back it up the same way they'd back up a letter, such as by clicking File - Save As - and then choosing removable media as the destination for the file, or by dragging a copy out of their Documents folder and onto their flash drive etc.

You would think that the wallet only stored the public/private key pairs.  In that case it could have been a flat file.  But it seems that all transaction history is stored in there, too, which is what is really complicating things.  The issue I see is that there are so many files in the data directory.  I've read somewhere that this version of bitcoin was not intended to take off, so maybe that's part of the reason.  I'd suggest using an SQL database of the blockchain which would make the job of pruning unneeded data (redeemed transactions) easier, then you could execute some simple SQL statements and get address balances and transaction details.  You could also build in most of blockexplorer.com into the client.

I don't think it being a database is a silly idea (maybe that is because I work w/ db everyday).  More applications that you think use databases internally.  Any mission critical I/O is going to need support for:

atomic transactions
read/write verification
high level of availability
dataloss recovery
backups

Once you implement all that you just built a good chunk of the "plumping" that makes a database and likely it isn't as good as databases which have tens of thousands of hours of development already.

Still the "db" as used by the wallet is horrible.  It really isn't a database.  More like a loosely typed flat file stuffed inside a database.  So you get all the disadvantages of flat files combined with all the complexity and disadvantages of a database.   Then it is compounded with the "weird" choice of Berkeley DB over SQL Lite.  Anyone have insight into that decision or was it simply a case of "using what you know"?

Having everything stored as varchar and recorded as a set of type, key, value is ..... yuck.

@Peter:
I never realized Satoshi didn't appreciate the need for alt clients.  Strange he saw the danger of centralization everywhere else but didn't notice the danger that centralization of development would bring.    Honestly IMHO even the term "alt client" is dangerous.  Hopefully in time Bitcoin will be evolve to a point where there are simply compatible clients and the project (under whichever name is evolves to) which began as the Satoshi client is just one of many peers.

^^ this

I believe the reason for only using one table is the result of the glue layer on top of BDB (which I suppose was designed by Satoshi). This way, only one open database object needs to be kept around per file. It's an ugly system, but it seems reasonable at least. As far as I know, Satoshi didn't really like the idea of alternate clients, so I don't think he designed with ease of interoperability in mind. Once that layer was in place, it was probably all to easy to add fields to (mostly) the wallet for various pieces of data (and I did so as well, later on...).

Regarding the choice for BDB in the first place: it makes sense for the transaction/block index database. These are large, need frequent updates and queries, and need a high degree of transaction atomicity. For wallets and IP addresses, I believe better solutions are possible. Both are only read at startup, and subsequently updated but mostly appended to. I hope we can move to a better format (especially for the wallet) soon. One that isn't prone to corruption under flaky hardware (perhaps by only appending to it, and occassionally rewriting it), isn't linked to a fixed database environment directory, and is backward compatible...

I think the fact that it is a database in the first place is a silly design decision.

it's not the code, its the structure of the wallet database. The code looks fine, I would have wrote it in another way, but that's irrelevant.

Is it insecure, slow, poorly arranged code?
If it's python, can't you modify it and use your own improvements? Submit your improvements to joric?

i have one question in mind right now: "WHY??"
WHY have you designed the wallet format this way it sucks!
you are even using dbname/tables but you only have one table in wallet.dat(the "main" table), BUT you are using it incorrectly.
im right not reading this code:
https://github.com/joric/pywallet/blob/master/pywallet.py#L1283

WHY have you not put the each type, in each separate table? i don't get it! its a mess.
a table for settings, one for keys, one for accounts, ...
and i see you are using public keys, as a database key why not use the hash/address, as a key to both the publickey and privatekey?

what is the benefit, of this madness? the dude that came up this db scheme, is a bad database coder.


/rant over