Docs on the structure and format of the wallet database

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

Quote from: kjj on July 19, 2013, 10:23:04 AM

You really shouldn't mess with the file directly. Treat it like a black box and use the proper tools (bdb).

+1 to anyone making proper documentation of the key/value pairs though. I can think of a number of useful utilities that could be made more easily if people knew how.

Agreed. Maybe "file" was the wrong word I don't really care how bdb is structuring it on disk. I am more interested in the structure and format of the various key value pairs which make up the wallet database. So when bdb returns the next key as "XXXX" one knows what the hell it is and how to parse the stored value.

Updated OP for clarity.

jackjack

legendary

Activity: 1176

Merit: 1233

May Bitcoin be touched by his Noodly Appendage

Sorry for the ugly formatting

Code:

##### Element: key
value1
value2
...

The final value is value1+value2+..., where + means concatenation

##### Transaction: VS(tx)+VS(32bytes(txid[::-1]))
VS(rawtx)

##### Label: VS(name)+VS(address)
VS(label)

##### Version: VS(version)
int32(version)

##### Minversion: VS(minversion)
int32(minversion)


##### Setting: VS(setting)+VS(settingname)
If settingname starts with "f":
	int8(bool)
If settingname starts with "addr"
	int32(nVersion)
	int32(nTime)
	int64(nServices)
	12bytes(pchReserved)
	4bytes(ip)
	int16(port)
If settingname == "nTransactionFee":
	int64(nTransactionFee)
If settingname == "nLimitProcessors":
	int32(nLimitProcessors)

##### Unencrypted private key: VS(key)+VS(public
VS(private key)


##### Wkey(?): VS(wkey)+VS(public
VS(private key)
int64(created)
int64(expires)
VS(comment)

##### Default address: VS(defaultkey)
VS(address)

##### Pool address: VS(pool)+int64(index)
int32(nVersion)
int64(nTime)
VS(public_key)


##### Account: VS(acc)+VS(account)
int32(nVersion)
VS(public_key)

##### Account entry: VS(acentry)+VS(account)+int64(n)
int32(nVersion)
int64(nCreditDebit)
int64(nTime)
VS(otherAccount)
VS(comment)

##### Best block: VS(bestblock)
int32(nVersion)
VI(number of hashes)
for(hash in hashes){
 32bytes(hash)
}


##### Encrypted private key: VS(ckey)+VS(publickey)
VS(encrypted private key)

##### Encryption settings: VS(mkey)+VS(nID)
VS(encrypted key)
VS(salt)
int32(nDerivationMethod)
int32(nDerivationIterations)
VS(otherParams)

Meanings:
VI, VS, intN=(N/8 bytes), [::-1]=ReverseEndianness

jgarzik

legendary

Activity: 1596

Merit: 1091

Quote from: kjj on July 19, 2013, 10:23:04 AM

You really shouldn't mess with the file directly. Treat it like a black box and use the proper tools (bdb).

This is true for most people (but I am not most people )

Quote

+1 to anyone making proper documentation of the key/value pairs though. I can think of a number of useful utilities that could be made more easily if people knew how.

+1

piotr_n

legendary

Activity: 2053

Merit: 1354

aka tonikt

Quote from: DeathAndTaxes on July 19, 2013, 10:12:39 AM

bestblock = highest block height seen/scanned by client

BTW, I don't think this bestblock works.
Unless it got fixed recently.. or unless I don't understand what it should do.

I used to use satoshi client for my cold storage and each time after updating the blocks+chainstate in this cold PC, I always had to do the -rescan.
Without explicit rescan, it was never able to notice the new txs that came to the addresses from my cold wallet - even if there was only one new block added to the chain.

piotr_n

legendary

Activity: 2053

Merit: 1354

aka tonikt

Quote from: kjj on July 19, 2013, 10:23:04 AM

You really shouldn't mess with the file directly. Treat it like a black box and use the proper tools (bdb).

It's a good advise. Though it sucks that the bdb lib is so big - probably even bigger than openssl.

May I ask, do you guys plan to remove bdb dependency from satoshi client one day?
I mean, switching to HD wallets seems like a reasonable moment to drop it, especially since UTXO moved to leveldb.

kjj

legendary

Activity: 1302

Merit: 1025

You really shouldn't mess with the file directly. Treat it like a black box and use the proper tools (bdb).

+1 to anyone making proper documentation of the key/value pairs though. I can think of a number of useful utilities that could be made more easily if people knew how.

jackjack

legendary

Activity: 1176

Merit: 1233

May Bitcoin be touched by his Noodly Appendage

"key" is an uncrypted key
"ckey" is an encrypted key

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

Quote from: jgarzik on July 19, 2013, 09:33:15 AM

Quote from: DeathAndTaxes on July 18, 2013, 05:38:49 PM

Title says it all, any documentation (other than the code) on the format of the wallet.dat file?

No.

The file format is Oracle Berkeley DB. A quick google did not find any file format specification. (help requested!)

The key-value pairs are encoded using standard serialization (binary encoding) found elsewhere in the bitcoin protocol.

So doing some code spelunking in both pywallet and QT client I think I have figured most of it out.

version = version which last updated the wallet.
minversion = minimum version required to read wallet (throws error in older clients).
tx = transaction record involving a key in the wallet.
name = user entered label of a key (possibly a tx/account as well? or just keys?)
acc & acentry = used in QT accounting system (not really interested)
bestblock = highest block height seen/scanned by client
mkey = has nothing to do w/ Bitcoin keys, it contains info on encrypting/decrypting the wallet.
pool = list of public keys in keypool with their order (seems redundant by ok)

so that leaves "key", "wkey", and "ckey" which appear to all be related to bitcoin keypairs (addresses).

Why three different key-types for handling Bitcoin keypairs?
What is the difference?

Then there is "orderpostnext" and "cscript" which I can't make heads or tails of.

jgarzik

legendary

Activity: 1596

Merit: 1091

Quote from: DeathAndTaxes on July 18, 2013, 05:38:49 PM

Title says it all, any documentation (other than the code) on the format of the wallet.dat file?

No.

The file format is Oracle Berkeley DB. A quick google did not find any file format specification. (help requested!)

The key-value pairs are encoded using standard serialization (binary encoding) found elsewhere in the bitcoin protocol.

piotr_n

legendary

Activity: 2053

Merit: 1354

aka tonikt

Quote from: DeathAndTaxes on July 18, 2013, 06:21:54 PM

Still the lack of documentation is kinda sad. It just means countless hours wasted "relearning" the same thing by each developer.

I guess the best entity to address your concerns would be Oracle, since they own the development of this db.
But I believe there might be a doc with the format of these files, somewhere.

Inside the DB itself, there are satoshi specific "key-value" records, which indeed would be nice to have somewhere on the wiki, but AFAIK there is no such page.
And moreover it's getting changed as we speak - a new type of "keymeta" records are coming...

Up to v.0.8.3, the possible key-types were: name, tx, acentry, key, wkey, mkey, ckey, defaultkey, pool, version, cscript, orderposnext
Most of the key-types are followed with additional data (i.e. tx is followed by its hash). The key-type plus this additional data is the actual DB key.
As for the format of the value, you'd need to analyze the source code. Check ReadKeyValue in walletdb.cpp

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

Quote from: jackjack on July 18, 2013, 05:55:13 PM

I don't think so.
It's a berkeley db file.

Look at the pywallet code if you want more than one point of view.
kds is the key and vds is the value.

For example, the (key, value) pair for an unencrypted private key would be:
('\x03key\x21\x03\x01\x01\x01...\x01', '\x20\x54\xfd...\x31')
If the public key is '03010101...01' and the private key is '54fd...31'

Thanks I can follow the python code a lot easier than the reference client.

Still the lack of documentation is kinda sad. It just means countless hours wasted "relearning" the same thing by each developer.

jackjack

legendary

Activity: 1176

Merit: 1233

May Bitcoin be touched by his Noodly Appendage

I don't think so.
It's a berkeley db file.

Look at the pywallet code if you want more than one point of view.
kds is the key and vds is the value.

For example, the (key, value) pair for an unencrypted private key would be:
('\x03key\x21\x03\x01\x01\x01...\x01', '\x20\x54\xfd...\x31')
If the public key is '03010101...01' and the private key is '54fd...31'

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

Title says it all, any documentation (other than the code) on the structure and format of the wallet database?

Topic: Docs on the structure and format of the wallet database (Read 1533 times)