Author

Topic: What are the markers byte in wallet file (Read 267 times)

staff
Activity: 3458
Merit: 6793
Just writing some code
March 22, 2018, 01:43:56 PM
#6
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.
The lengths are there because of Bitcoin Core's serialization framework. It tends to prepend everything with the length and it's kind of annoying to make it not do that. It's easier for it to just do that for everything because some things need to be length prepended and others don't, and it also depends on the use case (e.g. in transactions, length prepending is necessary, but not necessarily for the databse, but the same data ends up in transactions as well as in the database).

Also, Bitcoin Core tends to have multiple things concatenated in each key or value so the length prefix helps to differentiate between the two (e.g. each key in the wallet is some string followed by the actual key like a public key, so the length prefix lets us know where the string ends and the pubkey begins).
member
Activity: 61
Merit: 15
March 22, 2018, 04:20:59 AM
#5
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.

That makes sense, thanks for answering!
full member
Activity: 198
Merit: 130
Some random software engineer
March 22, 2018, 03:04:14 AM
#4
Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.

Having took a look sometimes to dbwrapper in Bitcoin Core, I would say it is not absolutely necessary, but it helps for error checking while reading records, especially badly formatted ones (don't forget some parts of wallet/chainstate/whatever db) can be obfuscated/crypted, and having those kind of metadata helps to know if data is valid or not.
member
Activity: 61
Merit: 15
March 22, 2018, 02:44:59 AM
#3
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.

Hi, is this necessary? my understanding is Berkeley DB already keeps track of data item's length internally, for every db get operation, the Dbt return its data as well as the length.
staff
Activity: 3458
Merit: 6793
Just writing some code
March 21, 2018, 11:51:25 PM
#2
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.
Those aren't marker bytes. They are length prefixes. The specify the length of the following piece of data.
member
Activity: 61
Merit: 15
March 21, 2018, 09:23:01 PM
#1
So I tried to figure out the content of the wallet file, and a dump shows that each key value is prefixed with a non-printable mark, even when the data item is valid printable strings, what's the purpose of these markers bytes, like
Code:
\07, \09, \0a
etc.

$ db_dump -p wallet.dat
...
 \07keymeta!\03\ff \15\06F0\97a\a4F\e2\15\01\b9;U%\97\d7\8c\0b\0b\e4`\bc\96\b2\ee@\db\8eL
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/461'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff*r{\de5\e4/\a2_\f6+rIv\db(\92r\ee\dd\ce\18\daC\13\df\85\e3l6H
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/254'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffD9Y=\8e\0f\a51\95\ecs\98u\9a\da[A\82v\dd=\95\f9\bbf\a5\eby\18\0b9
 \0a\00\00\00\ca3\91Z\00\00\00\00\0cm/0'/1'/594'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ffF\b3\c9\81\d4\1btNv\e9\ad\b3%\bbb\14\93\ff\e2}\9a\b4\dc\05\dc\b01\a5\87\ac^
 \0a\00\00\00\c93\91Z\00\00\00\00\0cm/0'/0'/638'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07keymeta!\03\ff\80A\e8\ea\c8*\92DU\e5W\fe\b3\e9]Vk2\b3\e4\fep\10\e4i\d5\1f\\\83\\3
 \0a\00\00\00\cb3\91Z\00\00\00\00\0cm/0'/1'/921'j\11Z\e1t%\14\0d\13\0ai8\11^\8dz\e7bC\13
 \07purpose"18V1UjWhRBEq7Kj3cXtLLjh2imd6UPWz9A
 \07receive
 \07purpose"1H1uEWBtsHe4tMzCRqGL6BkZBbu6xfnsmR
 \07receive
 \07purpose"32AvF5Vmnn9zD6VRf1D3vZXFHPuwdgEKqw
 \07receive
 \07purpose"35gnJvBQD424mgH9j2Udu6zoEsSXhzQkky
 \07receive
 \07purpose"3AMa4ZAh6R83sWRKsFvprpXvhzDuZ34yqm
 \07receive
 \07purpose"3M7vPcw9h6KaiRfpNNBRFC6mc918UaW6NF
 \07receive
 \07purpose*bc1qnfqmg4ty3qgft79gk3zgym3w3hqxmznxg9avup
 \07receive
 \07version
 \ac\97\02\00
 \09bestblock
 \ac\97\02\00\00
 \0aminversion
 \9cp\02\00
...
Jump to: