What data have Bitcoin Core blockchain files?

pbies

full member

Activity: 297

Merit: 133

Quote from: ABCbits on June 13, 2022, 07:36:11 AM

Quote from: pbies on June 13, 2022, 05:45:05 AM

RPC is too slow to call it for whole blockchain, and it will eat too much resources. I have synced Bitcoin Core, the blockchain. I need to operate on BerkeleyDB directly.

Since Bitcoin Core RPC-JSON doesn't meet your need, here are few pointers for reading the files directly.
1. https://bitcoindev.network/understanding-the-data/
2. https://github.com/bitcoin/bitcoin/issues/6613
3. https://learnmeabitcoin.com/technical/blkdat
4. https://bitcoin.stackexchange.com/a/48738

P.S. I would still recommend you to check self-hosted block explorer (e.g. mempool.space) and Electrum server (e.g. Fulcrum).

Quote from: pbies on June 13, 2022, 05:45:05 AM

Dolthub is a remote solution, I need local solution.

Only if you run the SQL query on their website rather than cloning the data.

Thank you very much! This is what I needed! Especially link no 1.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: pbies on June 13, 2022, 05:45:05 AM

RPC is too slow to call it for whole blockchain, and it will eat too much resources. I have synced Bitcoin Core, the blockchain. I need to operate on BerkeleyDB directly.

Since Bitcoin Core RPC-JSON doesn't meet your need, here are few pointers for reading the files directly.
1. https://bitcoindev.network/understanding-the-data/
2. https://github.com/bitcoin/bitcoin/issues/6613
3. https://learnmeabitcoin.com/technical/blkdat
4. https://bitcoin.stackexchange.com/a/48738

P.S. I would still recommend you to check self-hosted block explorer (e.g. mempool.space) and Electrum server (e.g. Fulcrum).

Quote from: pbies on June 13, 2022, 05:45:05 AM

Dolthub is a remote solution, I need local solution.

Only if you run the SQL query on their website rather than cloning the data.

pbies

full member

Activity: 297

Merit: 133

I am trying to write my own Python scripts to mess around with BerkeleyDB and Bitcoin Core blockchain.

I am not interested in ready solutions that provide list of addresses - they are always outdated, I need to take data from current blockchain.

To translate C/C++ code of Bitcoin Core to Python is last resort solution.

RPC is too slow to call it for whole blockchain, and it will eat too much resources. I have synced Bitcoin Core, the blockchain. I need to operate on BerkeleyDB directly.

Dolthub is a remote solution, I need local solution.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

The most reliable way to parse the BDB files is simply to copy the DB read/write code of Bitcoin Core and make a CLI frontend that will let you query any data. That's because drivers like db_dump(1) and others are not guarranteed to work, as within the BDB file format, the program can write it in however manner it likes (i.e. it's technically a BDB format but it's unreadable to other applications).

pooya87

legendary

Activity: 3472

Merit: 10611

Quote from: pbies on June 12, 2022, 10:28:24 AM

One thing I would like to do is to dump all public addresses with any additional data.

Unless you are trying to do something special that is not to extract pubkeys to brute force private keys lol you can use existing lists that contain all bitcoin addresses that have ever been used[1] or list of addresses with a balance[2].

[1] https://bitcointalksearch.org/topic/list-of-all-bitcoin-addresses-ever-used-currently-unavailable-on-temp-location-5265993
[2] https://bitcointalksearch.org/topic/list-of-all-bitcoin-addresses-with-a-balance-5254914

pbies

full member

Activity: 297

Merit: 133

Quote from: ?? on ??

1. Use Bitcoin Core RPC JSON to obtain block data.

That is too slow for me. I need access to whole data in hours, not months.

One thing I would like to do is to dump all public addresses with any additional data.

pbies

full member

Activity: 297

Merit: 133

I am interested in writing a Python script to parse BerkeleyDB in which is blockchain on disk working.
I know that this is old BerkeleyDB and I need to use bsddb3 with Python 3.9 or earlier.

What data can I find in these folders?

1. blocks+index
2. chainstate
3. indexes

How should I instruct Python to connect to whole blockchain in these files and then query it?

Any help appreciated.

Topic: What data have Bitcoin Core blockchain files? (Read 163 times)