Topic: Bitcoin address mass balance checker [Tutorial and code] (Read 755 times)

I don't have such skill, the documentation doesn't mention public key at all and Insight itself is no longer updated. I'm afraid you couldn't accomplish your goal with this method.


4 pipeline? That's crazy, i never used more than 3 pipeline at once.

I am interested in following your method, but the SHODAN part isnt very clear

About your comment:
The first thing we need to do is find other people's Insight servers that are fully exposed to the internet and respond to API-calls. A simple way of locating some of these is by searching Shodan (https://www.shodan.io/) using this phrase: "port:3001 insight bitcoin"

I registered, but I found that I could install and run SHODAN CLI form my linux machine.... in any case, I can not find the correct command line in SHODAN to get the resultant IP addresses...

I used: shodan "port:3001 insight bitcoin"

but returned an error.

Would you mind being a little more specific how you perform that search? Do you use the CLI from your linux machine ar do you search in the actual SHODAN website?

Many thanks.

Actually, I started with this:
But splitting it up reduces CPU load when using it more than once.

I often end up with dozens of pipes on a line. It's very convenient for testing, after each new pipe I instantly see the result and either adjust it or add the next one.

---

---

If anyone's interested, I can set up a mirror for the latest version of both of those, for instance with weekly updates (a bit like I did for Bitcoin block data) and download speeds that aren't stuck in the '90s.

---

---

Update: I'll start downloading gz.blockchair.com/bitcoin/outputs/. It'll take about 5 months. After that, I can share daily updates with decent download speed.

Update:

Yup. Downloading Bitcoin Core, letting it sync and run a blockparser will do the job. It's not fast and easy; in fact, most if not all block parsers I've tried require almost insane hardware and a decent amount of patience. And some bash skills (such as yours) to produce lists.

Provided you have a full node up and running (i.e. a folder containing .blk files), which block parser would you recommend today? I updated my own list of "all used public addresses ever" about a year ago, which gave me a list of around 500 million addresses, and it took some blood, sweat and tears (and curse words). Also, I was never 't able to extract bech32 addresses (bc1...). Since I haven't looked into recent development in the field, I'm interested in knowing which one on github one should go for today. Many thanks.

This works (on Linux) to display addresses that exist in both lists:
Option 1 takes 6 seconds on my old laptop, checking 300,000 addresses against 30 million addresses with balance.

---

---

Blockchair also has all outputs, but at 10 kB/s it's going to take a while. If you can get all addresses ever used from your own version of Bitcoin Core, you're good to go.

This is the best option if you're only interested in the current balance. For various reasons, I'm more interested in transaction history - e.g. "even if the balance is 0, has this public address ever occured on the blockchain?" - and for that you need more input than only Blockchair's list of public addresses that have a positive balance "now".

I've tested a very fast search method before, and it works great:
1. Use the link above to get all funded Bitcoin addresses
2. Make a list of all addresses you want to check
3. Sort the lists and remove accidental duplicates
4. Add both lists together, sort them again, and find all duplicates

All you need for this is Linux commands "sort" and "uniq". It scales very well, checking millions of addresses takes a few seconds and you can search as much as fits your computer's memory.

LoyceV post the code!

If you want to check for all addresses that have ever been funded before, you'll just need to find a list for that. The main drawback is that you won't catch addresses that have been funded in the past hours if you don't update your list real-time.

Hello I mean Public Key's. Can you modyfy your code pease ?

Can someone modify this code for getting public addresses too ?

If you, or anyone else for that matter, is interested in a copy - without download speed limits - I put a copy of the latest version here:

https://www.mediafire.com/file/v30lydmiljcx42w/blockchair_bitcoin_addresses_latest.tsv.gz/file

would it not be faster to get the API key for blockchain.com and run a python script?

I have such tools and it outputs all balances with tx numbers

I realize that fact, but it doesn't change there's ethical problem of using private Bitcoin explorer with sloppy security / without owner's permission.

P.S. this is my last reply to prevent further off-topic / topic derailing

As discussed above, if using the script in the OP, you are most likely searching for thousands (or more likely millions/billions) of addresses, and a public block explorer is going to rate limit your searches.

I think he answered that question:

I didn't know this one existed, many thanks!

EDIT: Aha, so it is capped to 10 KB/s, so that it takes about 21 hours to download the file. Smart business model.

Why not just (wait 24h and) download the file from https://gz.blockchair.com/bitcoin/addresses/
then use any text editor or write a simple script to work on local data.
At least you will not annoy servers with many calls.

You are assuming the business doesn’t ever experience downtime. You are also assuming there are not any edge cases in which the business might not detect the transaction. The OPs script would also help a business audit their holdings to confirm they have coin they believe they have.

I do agree that the intended end user for the OPs script is probably people trying to steal others’ money despite the above. Using others’ insight servers is also questionable. The problem of looking up the balance of many addresses is an interesting engineering problem. I would also prefer it be highlighted that people try to steal brain wallets and that there are “easy to understand” guides on doing so. This should make people think twice before using a brain wallet of their own, and hopefully will lead to less theft from brain wallets over the long term.

a business does this in reverse, meaning when their mempool receives a new transaction they check the output of that against their list of output-scripts-to-watch which is a sorted and a much smaller list. then if that single transaction had any outputs that matched they raise the respective balance update event of the respective account. the number of txs they receive in their mempool isn't that huge to cause any problems either.
what OP is trying to do/teach still mainly concerns those who are trying to steal other people's money even if they are wasting their time.

Someone checking "many" addresses would likely be checking mostly "empty" addresses, that is addresses that have no transactions, or that have no unspent inputs.

if you have a .csv file with 10,000 addresses it might not be terribly difficult to filter which ones have transactions/unspent inputs, however, this will not scale if you have many millions of addresses. The output on your terminal might cause your computer to run out of memory, but even if the terminal output was removed, looking through a .csv file with millions of lines you don't care about takes up resources unnecessarily, and doing so can require intermediary technical knowledge.

I would suggest that output only be recorded if there are transactions, or if the balance is above zero.

A business with many customers who either deposit coin, or who pay for their items with coin would have the need to check many addresses on an ongoing basis.

A fairly common question on this forum is:

"I have a list of thousands of bitcoin addresses - how can I check their balances?"

Presumably, this is asked by people who are still into hacking brainwallets and/or playing around with BrainFlayer (https://github.com/ryancdotorg/brainflayer). Or why else do you need to check the current balances for thousands of bitcoin addresses; not a rhetorical question, please do tell me about other potential uses!

I'll show you a method for it that I sometimes use myself.

For the record, I should say that one politically correct answer is that you first need to install a bitcoin full node, let it download the complete blockchain, and on top on that add an Insight server (https://insight.bitpay.com/) that you can query fast via API, and write and execute a script for it.

Now, I assume that most people are lazy and not willing to pull that off, or simply lack the technical skills required, so let me demonstrate my quick and dirty work-around that requires no installation or downloads. You just need a Linux machine.

The first thing we need to do is find other people's Insight servers that are fully exposed to the internet and respond to API-calls. A simple way of locating some of these is by searching Shodan (https://www.shodan.io/) using this phrase: "port:3001 insight bitcoin"

Provided that you have a registered Shodan account (there are free - do sign up - you need to be registered to search for specific port numbers), you will find about 35 or so such servers. However, not all of them will work. Some hits are false positives, some do not respond to standard API-calls, some are simply down and maybe half or more are configured for other coins than BTC (be especially careful with BCH, as they will give you a wrong response, as they accept legacy bitcoin addresses).

When writing this, one hit that looked promising was IP: 18.141.53.197 Port: 3001 (this particular server may very well be down when you try this, also please don't hammer individual IP addresses!), so the first thing I did was to check this URL in my browser (note that it should be http and not https, and the correct port 3001):

http://18.141.53.197:3001/

Looks good! Now we need to query it with a known Bitcoin (BTC not BCH) that has funds in it. I semi-randomly picked this address, as it looks as it has been around a while and is well-funded:

https://www.blockchain.com/btc/address/39PVJ1sEVmFHJXjEG1D8kQa4Mk7f7jeMvA

So, for confirmation that our discovered Insight server does its job as intended, let's look at this exact URL in our browser:

http://18.141.53.197:3001/api/addr/39PVJ1sEVmFHJXjEG1D8kQa4Mk7f7jeMvA

Bingo! Lot's of JSON data. From a quick and manual glance, the balance corresponds to the one on blockchain.com too - definitely the droid we are looking for.

Repeat this step until you have 3-5 separate IP addresses that all work and display the desired data.

Finally, for automation, I have written a script (further below). I want to strongly encourage you not to reuse the exact IP addresses as in this one, or they will not be up for much longer. DO change the IP addresses in the script to ones that you have discovered yourself.

The script reads a text file with bitcoin addresses, so first create a such a file containing (for example):


and name that file "addresses.txt"

(Note that the list may contain both legacy addresses [starting with 1] and P2SH addresses [starting with 3], but not P2WPKH/Bech32 addresses [starting with bc1] - if you can find a working solution for Bech32 addresses, please let me know.)

In the same folder create a file called "checker" (for example with nano):

and issue the command "chmod +x checker" to make it executable.

Unless you ask for it specifically, I won't comment or explain the code further.

All you need to know it that it outputs a file called "balances.csv" containing all the data. A very good reason to name the output .csv is that you can open it directly with Excel or any other spreadsheet program without any other pre-formation.

Ultimately, when you have everything ready: In the same directory a list of Bitcoin addresses called "addresses.txt" and an executable script called "checker", issue this command:

./checker addresses.txt

and watch the magic happen in real-time. If everything is done right, it should output this in the terminal


and the file "balances.csv" - try and open it in Excel - should contain:



With enough fresh IP addresses, it is in my experience easy to check many thousand bitcoin address balances in a few minutes.

Again - do not hammer individual IP addresses!

Comments? Questions?

We happy?