From node to block explorer? | Bitcointalksearch.org

noblefire

newbie

Activity: 6

Merit: 7

Quote from: BlackHatCoiner on December 15, 2020, 03:57:10 PM

Quote from: Quickseller on December 15, 2020, 03:31:48 PM

A block explorer will use a database to keep track of transactions and will update it as new blocks come in. It will probably not use RPC commands as it receives requests from its users.

This seems a great idea. Why haven't I thought it before... Once a new block is mined, the database will insert a new row of information. Still, though, I don't get how I'll convert the blk.dat files to csv. (So I can import it to a database)

I recently began working on trying to do the same thing. I wanted a python library that would let me change the RPC authentication info so that I could use python to send RPC commands to my node and store the returned data into a CSV file.

There were some libraries I found on Github but they didn't do exactly what I wanted, so I started writing a library myself. I've tested it with a few commands, and it works. Using the requests library and this link https://curl.trillworks.com/ you can convert all the bitcoin-cli curl commands to python. I've been making them into functions so they're callable within other programs.

a_6apcyk

jr. member

Activity: 35

Merit: 10

Quote from: PrimeNumber7 on December 16, 2020, 11:55:49 PM

Quote from: BlackHatCoiner on December 15, 2020, 03:57:10 PM

Quote from: Quickseller on December 15, 2020, 03:31:48 PM

A block explorer will use a database to keep track of transactions and will update it as new blocks come in. It will probably not use RPC commands as it receives requests from its users.

This seems a great idea. Why haven't I thought it before... Once a new block is mined, the database will insert a new row of information. Still, though, I don't get how I'll convert the blk.dat files to csv. (So I can import it to a database)

You would run a series of RPC commands to get the transactions contained in each block. I have pseudo code describing how to import transactions into a database below:

#run for loop through each block
#run for loop to get the txid of each transaction in the blook you are looking at
#get the inputs, outputs, and amounts of inputs/outputs for each transaction
#update database accordingly

Once you have updated your database with existing blocks, you can create a script to run lines 2-4 each time your node receives a new block. You will also need to account for orphaned blocks once your database matches the current blockchain.

There is a great tool for extracting data from different blockchains https://github.com/blockchain-etl/bitcoin-etl. It connects to RPC and extracts blocks, transactions, inputs and outputs data

PrimeNumber7

copper member

Activity: 1666

Merit: 1901

Amazon Prime Member #7

Quote from: BlackHatCoiner on December 15, 2020, 03:57:10 PM

Quote from: Quickseller on December 15, 2020, 03:31:48 PM

A block explorer will use a database to keep track of transactions and will update it as new blocks come in. It will probably not use RPC commands as it receives requests from its users.

This seems a great idea. Why haven't I thought it before... Once a new block is mined, the database will insert a new row of information. Still, though, I don't get how I'll convert the blk.dat files to csv. (So I can import it to a database)

You would run a series of RPC commands to get the transactions contained in each block. I have pseudo code describing how to import transactions into a database below:

#run for loop through each block
#run for loop to get the txid of each transaction in the blook you are looking at
#get the inputs, outputs, and amounts of inputs/outputs for each transaction
#update database accordingly

Once you have updated your database with existing blocks, you can create a script to run lines 2-4 each time your node receives a new block. You will also need to account for orphaned blocks once your database matches the current blockchain.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

You also have https://mempool.space
Github: https://github.com/mempool/mempool

Which in addition to being a block explorer is also showing the fees in the blocks so you can figure out how much you need to get into the next one.

You have some that do not even need a database. You loose some functionality by not having one but it can be done:
https://github.com/janoside/btc-rpc-explorer

Depending on the speed / memory / etc. of your hardware you should be able to have either one of the above running in under an hour.

-Dave

pooya87

legendary

Activity: 3472

Merit: 10611

Quote from: BlackHatCoiner on December 16, 2020, 02:19:16 AM

~
I guess that's reasonable. The only thing nodes do except verifying is sharing their blk.dat files. Since I create a handshake with a node and I send him my block height he is forced to send me his block height too, before sending me any files.

Not just blocks, they can send you a lot more stuff that I already mentioned in my post. Blocks are big (usually 1-2 MB) while they can also give you the new or old block hashes (only 32 byte) or the new block headers (80 bytes each).
Take a look here: https://en.bitcoin.it/wiki/Protocol_documentation#getheaders

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: BitMaxz on December 15, 2020, 07:18:25 PM

~

Great links, thanks. I had seen iquidus block explorer before, but didn't take a look on their github. It's funny that all shitcoins use the same block explorer. Tongue

Quote from: pooya87 on December 15, 2020, 11:30:02 PM

If you just want to get the latest block hash you can still fake it though it requires some additional manual work. You can simply connect to another node, give it the block header hash of one or more of previous header hashes (less than 2000 blocks deep) and receive the most recent hash.

I guess that's reasonable. The only thing nodes do except verifying is sharing their blk.dat files. Since I create a handshake with a node and I send him my block height he is forced to send me his block height too, before sending me any files.

pooya87

legendary

Activity: 3472

Merit: 10611

If you just want to get the latest block hash you can still fake it though it requires some additional manual work. You can simply connect to another node, give it the block header hash of one or more of previous header hashes (less than 2000 blocks deep) and receive the most recent hash.

For example imagine if there were 655,000 blocks and you had hash of block 654,000, 654,001, 654,002 you simply add all 3 in a getheaders message and send it to any node and that node will respond with blocks 654,003 to 655,000 (it is about 80 kilo bytes). Now you can discard anything you don't want and keep block headers 655,000; 654,999; 654,998.
If you stay connected to that node it will send you inv messages containing any new block hashes which you can then download the header of and bump your known height.

I don't know what the point of this setup would be, but it is possible.

BitMaxz

legendary

Activity: 3472

Merit: 3217

Happy New year 🤗

Quote from: BlackHatCoiner on December 15, 2020, 01:35:13 PM

How can you make it a block explorer? I imagine a VPS that runs the bitcoin-cli and somehow sends the bitcoin core's information to the client. I just don't get how you can get this done.

There are some open-source from GitHub that I think you can review and check how they can retrieve information from nodes to your block explorer.

Here's some of the list that I found when searching on Google.

- https://github.com/iquidus/explorer -This one is a well-known open-source block explorer.
- https://github.com/hyperledger/blockchain-explorer
- https://github.com/poanetwork/blockscout
- https://github.com/X9Developers/block-explorer
- https://github.com/romanornr/blockexplorer

Some open-source block explorers need RPC enabled.

Sample the first link developed by iquidus you will need to set up the "settings.json.template" and put your wallet host, port, username, pass and must be set the RPC to true and the same goes to your node(Bitcoin.conf)

You can watch their guide on how to setup block explorer from here https://www.youtube.com/watch?v=laeV2slJgc8

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

The information contained within the blockchain has to be parsed because it isn't ideal for most block explorer to function. Notably, it doesn't interpret the addresses and your block explorer has to process and tabulate them by yourself. In addition, nodes also do not really take into account transactions that are not pertinent to your wallet, which is also why rescan doesn't work on pruned wallets.

You could try looking into the various open sourced block explorers on github. I tried running a block explorer once but it was quite resource intensive so I gave up. Most would require the txindex for the client to index other transactions as well.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: Quickseller on December 15, 2020, 03:31:48 PM

A block explorer will use a database to keep track of transactions and will update it as new blocks come in. It will probably not use RPC commands as it receives requests from its users.

This seems a great idea. Why haven't I thought it before... Once a new block is mined, the database will insert a new row of information. Still, though, I don't get how I'll convert the blk.dat files to csv. (So I can import it to a database)

Quickseller

copper member

Activity: 2996

Merit: 2374

A block explorer will use a database to keep track of transactions and will update it as new blocks come in. It will probably not use RPC commands as it receives requests from its users.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Until now, I used to believe that nodes, did share information with each other and with whoever created a connection with them. For example, I thought that sending a command like bestblockhash to a node and waiting to get a result, was possible, but unfortunately for me it isn't. After having a discussion with Coding Enthusiast, I realized that this can't work since nodes only verify and share their blocks.

I confused the block explorers' API with nodes. A block explorer has its own node and allows us to run commands like getting the balance of an address (most of the times). I want to understand how can you manage your node like that? How can you make it a block explorer? I imagine a VPS that runs the bitcoin-cli and somehow sends the bitcoin core's information to the client. I just don't get how you can get this done.

Topic: From node to block explorer? (Read 221 times)