Quick balance check | Bitcointalksearch.org

citb0in

hero member

Activity: 630

Merit: 731

Bitcoin g33k

Quote from: citb0in on November 14, 2022, 03:25:24 AM

Thanks a bunch. May I ask what's the intention/difference between

[1] https://github.com/romanz/electrs
[2] https://github.com/Blockstream/electrs

Should I favor [1] for electrs ?

Well, I guess I've found an answer to my question:

Quote

Note that this implementation of Electrum server is optimized for personal/small-scale (family/friends) usage. It's a bad idea to run it publicly as it'd expose you to DoS and maybe also other attacks. If you want to run a public server you may be interested in the Blockstream fork of electrs which is better optimized for public usage at the cost of consuming significantly more resources.

Source: https://github.com/romanz/electrs

I'll stick with the original electrs version.

As mempool.space is my favorite block explorer in my daily use I try to go this way. If not successfull I can try SPV blockexplorer and thankful to your great HowTo I have a good guide.Thank you.

@LoyceV: I'm absolutely satisfied with the current setup and my download speed is 1 GBit/s, it does not take much time to get those files

No need for extra work Wink

however, really appreciate your offer. Many thanks

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: citb0in on November 13, 2022, 04:55:59 PM

is there any How-To available which explains in detail how to setup an own block explorer?

I don't have a How-To, but someone created a Mempool.space clone with my name, so I'd go from there.
Do you want to use your own block explorer for privacy reasons? Or because of the large number of addresses you want to check?

Quote

Currently I'm using the data files of LoyceV which I'm very satisfied with. But as mentioned before, it requires regularly downloading those huge files.

Are you using the 1 GB file (with funded addresses) or the 25 GB file (which includes emptied addresses)? If it's the latter, you can download the daily updates instead of the large file. If it's the former, I wouldn't worry about the 1 GB of data per day. Considering you're running full nodes, I assume you're not limited on bandwidth.

If there's a demand for it, I can try to create a "differences file" for the address list, so you can download only the changes each day. It looks like a combination of diff and patch can work, but I'd say downloading a full update is easier.

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: citb0in on November 14, 2022, 03:25:24 AM

Looking forward to hear you comments. I'm not sure which product to choose

Blockstream's electrs is a fork of the original (romanz') electrs.
It may be more professional, I don't know, but it may also be more resource hungry too.
But the setup I've made is based on what I managed back then to actually install without Docker (and I really wanted to avoid Docker). Mempool.Space I didn't manage to install on my configuration. And I didn't know about Blockstream's electrs.

So I will not say that the setup I've made is the best. It's just the one I could install. If you have time for tests, you can try other directions (maybe also document them better). If they don't work, you can use "my" setup as last resort option.

If you are in the mood for tests, Fulcrum can also be an option as Electrum Server (and may or may not save some time, but iirc it will eat up quite some disk space with its db).

citb0in

hero member

Activity: 630

Merit: 731

Bitcoin g33k

Thanks a bunch. May I ask what's the intention/difference between

[1] https://github.com/romanz/electrs
[2] https://github.com/Blockstream/electrs

Should I favor [1] for electrs ?

And may I ask for the reason, why did you choose

https://github.com/janoside/btc-rpc-explorer

for a block explorer? What are pros/cons versus mempool ?

https://github.com/mempool/mempool

Looking forward to hear you comments. I'm not sure which product to choose

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: citb0in on November 13, 2022, 04:55:59 PM

is there any How-To available which explains in detail how to setup an own block explorer?

I see people already have you some hints. I will give you one more: I've made myself a step-by-step tutorial for installing and configuring Electrum Server (electrs) and also Blockchain Explorer (rpc-btc-explorer). It can be done on Windows (by installing WSL and then Debian) or directly under a Linux. Bitcoin + Electrum server + Block explorer under Windows (with WSL and Debian). Good luck!

I didn't use this block explorer's REST API, but I may try that too, however, it's supported according to this page: https://www.npmjs.com/package/btc-rpc-explorer#Features

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

Quote from: citb0in on November 13, 2022, 04:55:59 PM

is there any How-To available which explains in detail how to setup an own block explorer? I have few full-nodes running out there and I'd like to test this as a side-project of a local full-node. Currently I'm using the data files of LoyceV which I'm very satisfied with. But as mentioned before, it requires regularly downloading those huge files. So I'm looking forward to hear some helpful tips how to setup my own block explorer which I could use for querying addresses. I also did not understand why I would need an SPV or Electrum server for that purpose ?

You can take a look at open source block explorers here:

https://github.com/mempool/mempool
https://github.com/blockstream/esplora

and many more here:
https://www.lopp.net/bitcoin-information/block-explorers.html#:~:text=Open%20Source%20Block%20Explorers%3A&text=BTC%20RPC%20Explorer,Iquidus

pbies

full member

Activity: 297

Merit: 133

Quote from: citb0in on November 13, 2022, 04:55:59 PM

is there any How-To available which explains in detail how to setup an own block explorer? I have few full-nodes running out there and I'd like to test this as a side-project of a local full-node. Currently I'm using the data files of LoyceV which I'm very satisfied with. But as mentioned before, it requires regularly downloading those huge files. So I'm looking forward to hear some helpful tips how to setup my own block explorer which I could use for querying addresses. I also did not understand why I would need an SPV or Electrum server for that purpose ?

Own block explorer can be set up as a tree, if you want to search shortly. Adding keys would be fast.
It could be stored in the same format as in memory (a dump) and loaded when you need to use it.
However it would take tons of memory to provide good (quick) results.

Another way is to use indexed MySQL or MS SQL database.
You would load all the private keys and public addresses to it and then use it as a reference for all the blockchain.
Surely there can be no private keys for corresponding public addresses, but also they can exist (if you need to).

Or you can script the current Bitcoin Core data folder, that is BerkeleyDB, for example in Python and use it the same way.

All the same is for transactions and BTC amount.

It can be done.

In result - with some effort - it is easier to get all public addresses with or without balances and grep through it.
That is enough for me.

citb0in

hero member

Activity: 630

Merit: 731

Bitcoin g33k

Quote from: NeuroticFish on October 13, 2022, 02:29:36 AM

Quote from: PawGo on October 13, 2022, 02:18:28 AM

The fastest solution would be to download daily dump of founded addresses (from blockchair or http://addresses.loyce.club/ ), load into local database (for example postgresql) and then have have a program which launches queries to your database. That way you are independent of network connection, API limitations etc. In long term perspective, this is the best solution (of course you need to update your database from time to time).

Another solution would be to host everything locally - full node, spv server like electrs and then rpc explorer.

This is what I'd suggest: a local block explorer that provides the API OP needs.
The dumps from loyce.club are big (over 1GB zipped for the addresses with funds) and I find downloading them on a daily basis counterproductive.

is there any How-To available which explains in detail how to setup an own block explorer? I have few full-nodes running out there and I'd like to test this as a side-project of a local full-node. Currently I'm using the data files of LoyceV which I'm very satisfied with. But as mentioned before, it requires regularly downloading those huge files. So I'm looking forward to hear some helpful tips how to setup my own block explorer which I could use for querying addresses. I also did not understand why I would need an SPV or Electrum server for that purpose ?

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

Quote from: LoyceV on October 15, 2022, 07:33:31 AM

Quote from: bitmover on October 15, 2022, 07:32:03 AM

I believe my next project will to run my own node in an online server. I don't have it now. I will take a look on the costs and what else could I do with it.

I happen to have one running, it's a donated Xeon that's meant for cool projects like this. I don't mind feeding you data dumps in some format.

Do you have an api or something like that? Because a data dump ( a large file?) would be very slow.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Depends on what you're doing. Are they clients' addresses? If so, you should better do this search locally. A simple way is to run a full node and an Electrum server, and feel the addresses to the Electrum client (new wallet -> watch-only with addresses). If you have a strong machine, it should be trivial thing to do.

From my RPi 4, it takes a few minutes to load 34xp4vRoCGJym3xR7yCVPFHoCNxv4Twseo, bc1qgdjqv0av3q56jvd82tkdjpy7gdp9ut8tlqmgrpmv24sq90ecnvqqjwvw97, 1P5ZEDWTKTFGxQjZphgWPQUpe554WKDfHQ, and 3LYJfcfHPXYJreMsASk2jkn69LWEYKzexb for the first time, which are consisted of thousands of outputs.

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: ABCbits on October 15, 2022, 06:38:06 AM

Actual database also use optimized data structure (such as btree or trie), although it comes down to whether you can utilize it properly or not.

That's correct. The indexes at least are optimized. But the access to the database itself also comes with a cost.
I agree, it's after all up to dev skills and preferences.

AnotherAlt

sr. member

Activity: 280

Merit: 259

https://bitcoincleanup.com #EndTheFUD

I would say bitmover's project is cool. OP can check if this meets your needs. Running a server for such a thing doesn't worth it unless you use it daily or for your work. You can also check the BitAC Android app. It's an open-source android app, and the codes can be found on Github. I haven't tried the app yet since I never had to check multiple address balances at once. You may want to try it and let us know if it works Wink

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: bitmover on October 15, 2022, 07:32:03 AM

I believe my next project will to run my own node in an online server. I don't have it now. I will take a look on the costs and what else could I do with it.

I happen to have one running, it's a donated Xeon that's meant for cool projects like this. I don't mind feeding you data dumps in some format.

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

Quote from: ABCbits on October 15, 2022, 06:38:06 AM

I noticed sentence "Powered by mempool.space and Coindesk." on your website. Have you checked whether those free API actually let user check thousand address within short time?

No, you can't use thousands at once. It won't load. You need to make a few each time.

Quote from: LoyceV on October 15, 2022, 07:21:44 AM

Would it work to replace those by your own utxo dump from Bitcoin Core? It takes more resources than using an API, but won't have any limitations on how many addresses you can check.

Certainly this would be the best solution.
I believe my next project will to run my own node in an online server. I don't have it now. I will take a look on the costs and what else could I do with it.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: bitmover on October 14, 2022, 05:35:27 AM

I was created this solution here:

https://www.bitcoindata.science/bitcoin-balance-check.html

I tried 400 addresses, but it never loaded the results.

Quote from: ABCbits on October 15, 2022, 06:38:06 AM

"Powered by mempool.space and Coindesk."

Would it work to replace those by your own utxo dump from Bitcoin Core? It takes more resources than using an API, but won't have any limitations on how many addresses you can check.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: bitmover on October 14, 2022, 05:35:27 AM

I was created this solution here:

https://www.bitcoindata.science/bitcoin-balance-check.html

You can check the balance of many addresses at once, but you can't add an xpub.

I can add this feature in a few weeks

I noticed sentence "Powered by mempool.space and Coindesk." on your website. Have you checked whether those free API actually let user check thousand address within short time?

Quote from: NeuroticFish on October 14, 2022, 01:23:25 PM

Quote from: PawGo on October 14, 2022, 12:50:28 PM

All depends on how it is going to be used. If you want to test a bulk of addresses from time to time - then yes, maybe.
If you want to test 1 address every few (mili)seconds, it will be too slow, dedicated (and properly indexed) database will be better.

If OP goes on this route, he could load up the file into an optimized structure in memory (maybe b-tree?) at the start of the day and then just query his own structure every time. This should be quicker than using an actual database.

Actual database also use optimized data structure (such as btree or trie), although it comes down to whether you can utilize it properly or not.

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: PawGo on October 14, 2022, 12:50:28 PM

All depends on how it is going to be used. If you want to test a bulk of addresses from time to time - then yes, maybe.
If you want to test 1 address every few (mili)seconds, it will be too slow, dedicated (and properly indexed) database will be better.

If OP goes on this route, he could load up the file into an optimized structure in memory (maybe b-tree?) at the start of the day and then just query his own structure every time. This should be quicker than using an actual database.

PawGo

legendary

Activity: 952

Merit: 1386

Quote from: LoyceV on October 14, 2022, 04:42:07 AM

Quote from: PawGo on October 13, 2022, 02:18:28 AM

The fastest solution would be to download daily dump of founded addresses (from blockchair or http://addresses.loyce.club/ ), load into local database (for example postgresql) and then have have a program which launches queries to your database.

The fastest solution is this:

Quote from: LoyceV on June 11, 2020, 10:30:34 AM

How to use
The most likely use is to check a long list of Bitcoin addresses for a remaining balance.
On Linux, use this to find matching addresses (after extrating the compressed .gz file of course):

Code:

comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | sort | uniq)

All depends on how it is going to be used. If you want to test a bulk of addresses from time to time - then yes, maybe.
If you want to test 1 address every few (mili)seconds, it will be too slow, dedicated (and properly indexed) database will be better.

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

I was created this solution here:

https://www.bitcoindata.science/bitcoin-balance-check.html

You can check the balance of many addresses at once, but you can't add an xpub.

I can add this feature in a few weeks

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on October 13, 2022, 02:18:28 AM

The fastest solution would be to download daily dump of founded addresses (from blockchair or http://addresses.loyce.club/ ), load into local database (for example postgresql) and then have have a program which launches queries to your database.

The fastest solution is this:

Quote from: LoyceV on June 11, 2020, 10:30:34 AM

How to use
The most likely use is to check a long list of Bitcoin addresses for a remaining balance.
On Linux, use this to find matching addresses (after extrating the compressed .gz file of course):

Code:

comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | sort | uniq)

Bitcoin_addresses_LATEST.txt: the extracted latest version downloaded from addresses.loyce.club.
mylist.txt: your own list of addresses, one address per line.

This takes only seconds to check millions of addresses. If your text file has Microsoft formatting, you may need to use this instead:

Code:

comm -12 Bitcoin_addresses_LATEST.txt <(cat mylist.txt | fromdos | sort | uniq)

But, that only gets you a list of funded addresses, not their balances. I'll cook something up:

Code:

grep -f mylist.txt blockchair_bitcoin_addresses_and_balance_LATEST.tsv

This is a bit slower and eats RAM: 500,000 input addresses uses 1.5 GB and took 50 seconds. You could mix it with the earlier command to only search the balance for addresses that aren't empty.

Quote from: Sarah Azhari on October 13, 2022, 10:15:58 PM

I don't know exactly what you want, you ask question: how to check the balance for thousands of public addresses.
the loyce link only has data address which you can't check the address that you want.

See the explanations on List of all Bitcoin addresses with a balance.

Topic: Quick balance check (Read 508 times)