How can i read Block.dat Files without using full node from C#

Anti-Cen

member

Activity: 210

Merit: 26

High fees = low BTC price

Quote from: coder0x15 on January 15, 2018, 06:17:09 PM

@Anti-Cen so how to read blk*.dat files without using full node?

Well me is not into mining, can not stand them myself as they play CPU-Wars but I had this wallet
I installed to give it a test using a type of wireshark program I wrote (Network packet sniffer) but
didn't get that far before I noticed it not only downloaded all the block headers but started to
download all the blocks, well it did until I stopped it so that's where I got em from

Windows (Spyware) version it was but PM me if you want to know who's wallet it was

Quote

Also if you talk about *Key* and *Value* database, you really would be surprised when you know that Bitcoin is not a {*Key* *Value*} database. Especially in RAM.

This LevelDB thing is Key/Value and the index for the blocks uses to to address the blocks is loaded using LevelDB not that
I ever had a chance to play with the beast so I am not sure of the point you are trying to make here

coder0x15

jr. member

Activity: 36

Merit: 3

@Anti-Cen so how to read blk*.dat files without using full node?
Also if you talk about *Key* and *Value* database, you really would be surprised when you know that Bitcoin is not a {*Key* *Value*} database. Especially in RAM.

Anti-Cen

member

Activity: 210

Merit: 26

High fees = low BTC price

Quote from: achow101 on January 15, 2018, 12:45:44 PM

The blk*.dat and rev*.dat files - the files that actually contain the block data - are not database files. They are just files with the raw blocks dumped into them. You don't need BDB or LevelDB in order to read them. Those databases are stored separately.

The format of the blk*.dat files is network magic (4 bytes) followed by length of the block in bytes (4 byte unsigned integer) followed by the block itself serialized in network format. This repeats until the file reaches a certain size. Then the next numbered file is used.

Well thanks for that and you sound like you know what your talking about so I apologize for getting it wrong
See https://bitcoin.stackexchange.com/questions/51435/migration-from-berkeley-db-to-leveldb

Quote

In 2013 a new bitcoin core was released and one of the supposed improvement was migrating from Berkeley DB to LevelDB. According to the release notes at bitcoin.org:

Quote

Bitcoin uses LevelDB to store and retrieve block information. This answer lists the exact schema/databases used:

See https://www.quora.com/How-is-Bitcoin-using-LevelDB

I think the confusion comes from the block-index and chainstate using LevelDB so thanks
for the correction and I was thinking why would anyone use key/value pair type DB on blocks
and should have back tracked sooner.

my chainstate files end with".ldb" and raw text in MANIFEST has "KšhÄß8...leveldb.BytewiseComparator..."
but i don't have any rev*.dat files to read but then again I did not get a full download of the data and
I dare not say where the data came from but i am out to parse more than just block for BTC so I needed
to swim in the raw

Top man you are and thanks

coder0x15

jr. member

Activity: 36

Merit: 3

@achow101 so please tell why you need rev*.dat files to read database?

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: Anti-Cen on January 13, 2018, 06:04:43 PM

The data in these "Block.dat" files is not raw text anymore

It never was in the first place.

Quote from: Anti-Cen on January 13, 2018, 06:04:43 PM

but it seems like i need to read the files using
Berkeley-DB format which does not compile for me so does anyone know where I can download a Berkeley-DB Dll that
will work in VS .Net or any short cuts using .NET file streams

Quote from: Anti-Cen on January 14, 2018, 05:55:09 PM

I think Berkeley-DB for the file format has been dumped and they now use LevelDB that is something
that Google started but getting a copy that will compile and works in MS-Dot.Net is no easy task

The blk*.dat and rev*.dat files - the files that actually contain the block data - are not database files. They are just files with the raw blocks dumped into them. You don't need BDB or LevelDB in order to read them. Those databases are stored separately.

The format of the blk*.dat files is network magic (4 bytes) followed by length of the block in bytes (4 byte unsigned integer) followed by the block itself serialized in network format. This repeats until the file reaches a certain size. Then the next numbered file is used.

Anti-Cen

member

Activity: 210

Merit: 26

High fees = low BTC price

Quote from: pebwindkraft on January 14, 2018, 07:55:27 AM

there are several way to read the files, even into a MySQL database. Couldn't find C#, but maybe you can flip through this link, and find what you need:

https://bitcoin.stackexchange.com/search?q=blk00000.dat+file

I think Berkeley-DB for the file format has been dumped and they now use LevelDB that is something
that Google started but getting a copy that will compile and works in MS-Dot.Net is no easy task

Thanks for the link, will check it out and i was using block.dat when searching so maybe blk0000.dat
will do the trick.

Won't use MySQL because you have to deploy dlls and I tend to use data-tables and I just save as XML (With -Schema)
for things like indexes which work nice and fast I find up to about a million rows of data but then Microsoft
starts falling to bits so i sort of use a derived version of a data-table that's been tested and works real fast
with 2 millions rows in a table and saves as simple CSV to reduce all the XML tags from read/write to file streams.

Chances are LevelDB will blow both me and Microsoft out the window when it comes to speed so
maybe I will just stick with that, who knows but thanks for the help

pebwindkraft

sr. member

Activity: 257

Merit: 343

there are several way to read the files, even into a MySQL database. Couldn't find C#, but maybe you can flip through this link, and find what you need:

https://bitcoin.stackexchange.com/search?q=blk00000.dat+file

Anti-Cen

member

Activity: 210

Merit: 26

High fees = low BTC price

Everything when dealing with Bitcoin seems hard work to me and is over complicated
especially using windows but not only do I want to parse BTC Block chain but also forks
like Bit-Cash, Bit-Gold and even the B2X if it gets started.

The data in these "Block.dat" files is not raw text anymore and I don't want to use NBitcoin or RPC to localhost node
because that won't work on several forked coins but it seems like i need to read the files using
Berkeley-DB format which does not compile for me so does anyone know where I can download a Berkeley-DB Dll that
will work in VS .Net or any short cuts using .NET file streams

I think i will be OK reading the header, transaction with input/outputs if I can just get the plain
text to parse it.

All the on-line REST APIs are very easy to use but it's slow and won't suite this job

Topic: How can i read Block.dat Files without using full node from C# (Read 276 times)