List of all Bitcoin addresses ever used - currently UNavailable on temp location - page 8.

Casdinyard

hero member

Activity: 1988

Merit: 875

Leading Crypto Sports Betting and Casino Platform

Quote from: LoyceV on August 18, 2020, 05:43:57 AM

Quote from: Casdinyard on August 18, 2020, 05:12:54 AM

Can you also scrape all the Bitcoin Address used here in forum and the user that uses it?

I actually can Cheesy

I found this regexp on Stackoverflow:

Code:

egrep --regexp="^[13][a-km-zA-HJ-NP-Z1-9]{25,34}$" filename

With some slight changes it stops matching parts of Eth-addresses:

Code:

egrep -w --regexp="[13][a-km-zA-HJ-NP-Z1-9]{25,34}" *

I could run this code on 53 million archived posts, but the main problem will be excluding quotes. That's annoying and slow to do, and if I don't exclude them, it will completely mess up the data. On the other hand, quotes may still contain information that was deleted by the user who posted it.
Even without quotes, users still post Bitcoin addresses that aren't theirs, for instance when providing evidence on a scammer.

I think it would be possible if and only if you scraped the following boards:

Services
Bounties
Marketplace in general (both BTC and Alt)
And Marketplaces of all local boards if applicable/available

With that, detection with evidences on a scam wouldn't be a problem to the matter. And yes, it would be hard especially if threads/posts were deleted. But it mustn't be a problem as long as a list can be made to simply be a reference of which user had used nor mentioned any addresses throughout his post history.

Quote from: LoyceV on August 18, 2020, 05:43:57 AM

Quote

I think it would help labeling the users and alt accounts throughout the entire forum, and would make it easier to detect which accounts are linked to each other

A smart user would simply use different addresses. An even smarter user would use different wallets, so they don't create a blockchain trail when they make a payment.
As a quick test, 51 out of 9999 posts contain at least one Bitcoin address (starting with 1 or 3, ignoring Bech32).
For now I won't go continue this search. If I ever do, I'll move this discussion to Reputation.

I'm looking forward to make it happen. Have I already mentioned my project on making an app (a BPIP ripoff) and such data would be helpful in it. I'm still on the planning stage to which should I go first and with many scraped data you've done, it would help me to make less scraping but rather make an API to just look up on your data.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: Casdinyard on August 18, 2020, 05:12:54 AM

Can you also scrape all the Bitcoin Address used here in forum and the user that uses it?

I actually can Cheesy

I found this regexp on Stackoverflow:

Code:

egrep --regexp="^[13][a-km-zA-HJ-NP-Z1-9]{25,34}$" filename

With some slight changes it stops matching parts of Eth-addresses:

Code:

egrep -w --regexp="[13][a-km-zA-HJ-NP-Z1-9]{25,34}" *

I could run this code on 53 million archived posts, but the main problem will be excluding quotes. That's annoying and slow to do, and if I don't exclude them, it will completely mess up the data. On the other hand, quotes may still contain information that was deleted by the user who posted it.
Even without quotes, users still post Bitcoin addresses that aren't theirs, for instance when providing evidence on a scammer.

Quote

I think it would help labeling the users and alt accounts throughout the entire forum, and would make it easier to detect which accounts are linked to each other

A smart user would simply use different addresses. An even smarter user would use different wallets, so they don't create a blockchain trail when they make a payment.

As a quick test, 51 out of 9999 posts contain at least one Bitcoin address (starting with 1 or 3, ignoring Bech32).

For now I won't go continue this search. If I ever do, I'll move this discussion to Reputation.

Casdinyard

hero member

Activity: 1988

Merit: 875

Leading Crypto Sports Betting and Casino Platform

Quote from: LoyceV on August 01, 2020, 05:05:46 AM

~

Can you also scrape all the Bitcoin Address used here in forum and the user that uses it? Yes, some users would have used the same wallet as they are just alts of someone (with a lot of investigation just to be proven correct). And I think it would help labeling the users and alt accounts throughout the entire forum, and would make it easier to detect which accounts are linked to each other and which are disobeying campaign rules and even forum rule (enrolling many accounts in a single bounty or sig campaign)

MrFreeDragon

sr. member

Activity: 443

Merit: 350

Very interesting statistics, thank you!

Quote from: LoyceV on August 01, 2020, 05:06:06 AM

-snip-
Addresses with most receiving transactions
This is the Top 100, the number in front of the address shows how many transactions it has received:
-snip-
326839 d-d0d953f2e7043342540a1407243e49fe
...
289070 d-0e9deef32abfc454392d21725f9defef
...
262539 d-73fd8c31c9fc1d084f44b301bb7adb6a
...
224217 d-752ed0099932a96fbc0a854a4d3a300f
...
219174 s-e3b0c44298fc1c149afbf4c8996fb924
-snip-

Can you please clarify, what is the type of these d- and s- addresses?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Some interesting (?) statistics (updated until blockchair_bitcoin_outputs_20200719.tsv.gz)
Total address count: 1,484,589,749
1... address count: 1,039,899,708
3... address count: 343,485,961
bc1q... address count: 55,006,904
...-... (with a "dash") address count: 46,197,161

Unique address count: 693,180,830
1... address count: 470,943,308
3... address count: 167,941,821
bc1q... address count: 39,137,878
...-... (with a "dash") weird address count: 15,157,808

Addresses with most receiving transactions
This is the Top 100, the number in front of the address shows how many transactions it has received:

Code:

4467608 1HckjUpRGcrrRAtFaaCAUaGjsPx9oYmLaZ
1900428 1NxaBCFQwejSZbQfWcYNwgqML5wWoE3rK4
1601193 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp
1527471 1FoWyxwPXuj4C6abqwhjDWdz6D4PZgYRjA
1204787 1LuckyR1fFHEsXYyx5QK4UFzv3PEAepPMK
1105406 1dice97ECuByXAvqXpaYzSaQuPVvrtmz6
1021575 3CD1QW6fjgTwKq3Pj97nty28WZAVkziNom
1009836 1G47mSr3oANXMafVrR8UC4pzV7FEAzo3r9
 929737 3JXRVxhrk2o9f4w3cQchBLwUeegJBj6BEp
 872274 1J37CY8hcdUXQ1KfBhMCsUVafa8XjDsdCn
 859422 3422VtS7UtCvXYxoXMVp6eZupR252z85oC
 841967 168o1kqNquEJeR9vosUB5fw4eAwcVAgh8P
 832807 1P9RQEr2XeE3PEb44ZE35sfZRRW1JHU8qx
 782811 1VayNert3x1KzbpzMGt2qdqrAThiRovi8
 689574 37Tm3Qz8Zw2VJrheUUhArDAoq58S6YrS3g
 676674 1DUb2YYbQA1jjaNYzVXLZ7ZioEhLXtbUru
 663458 bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej
 631610 17kb7c9ndg7ioSuzMWEHWECdEVUegNkcGc
 595853 1dice9wcMu5hLF4g81u8nioL5mmSHTApw
 580565 1Po1oWkD2LmodfkBYiAktwh76vkF93LKnh
 573787 1LAnF8h3qMGx3TSwNUHVneBZUEpwE4gu3D
 520889 1NDyJtNTjmwk5xPNhjgAMu4HDHigtobu1s
 505956 13vHWR3iLsHeYwT42RnuKYNBoVPrKKZgRv
 448252 1Fi9J5TeaWPHdU5cTJ4e9jr3V58SrWtUuT
 437634 1dice7fUkz5h4z2wPc1wLMPWgB5mDwKDx
 406471 1MPxhNkSzeTNTHSZAibMaS8HS1esmUL1ne
 395663 1dice7W2AicHosf5EL3GFDUVga7TgtPFn
 394249 1LuckyY9fRzcJre7aou7ZhWVXktxjjBb9S
 389038 1D5bPm1YAdn9WvAAixht7PbACU3TtkqtJJ
 376310 17A16QmavnUfCW11DAApiJxp7ARnxN5pGX
 364311 3HNSiAq7wFDaPsYDcUxNSRMD78qVcYKicw
 363898 3MfN5to5K5be2RupWE8rjJHQ6V9L8ypWeh
 357641 3HRZjedwF2AJejNTtgznWnas4E6froNP5r
 354691 1LuckyG4tMMZf64j6ea7JhCz7sDpk6vdcS
 346986 366Dgw4pi3rnvu5zizVWZF6nijWxZWc6RA
 341430 1dice6YgEVBf88erBFra9BHf6ZMoyvG88
 326839 d-d0d953f2e7043342540a1407243e49fe
 325099 38jMiiZs2C5n5MPkyc5pSA7wwW6H4p6hPa
 293567 38ENmTr2AD1avJrmmi9iM7PfS6nZVmuMKf
 289070 d-0e9deef32abfc454392d21725f9defef
 285507 1N52wHoVR79PMDishab2XmRHsbekCdGquK
 282321 3PUuiYu5cFMsagkffArrKZzQFtWdHttU3x
 280691 367f4YWz1VCFaqBqwbTrzwi2b1h2U3w1AF
 280107 1FoxBitjXcBeZUS4eDzPZ7b124q3N7QJK7
 262539 d-73fd8c31c9fc1d084f44b301bb7adb6a
 262317 1Fi57hAqyYYwaQVdA7a9qSKfiukBbt31G3
 253795 1K2SXgApmo9uZoyahvsbSanpVWbzZWVVMF
 252344 1dice5wwEZT2u6ESAdUGG6MHgCpbQqZiy
 251282 3JnFBLxDCutY3bZEZsPTkHAaUA1bxmEMX2
 250862 1diceDCd27Cc22HV3qPNZKwGnZ8QwhLTc
 247797 352zT3Ts9piSDhZpBsDoZMvdtDmJioQNBo
 246472 12JYmnfYU2ghzjwUAspzJsSnmJtK9bZPYR
 243955 1x6YnuBVeeE65dQRZztRWgUPwyBjHCA5g
 240428 3A4U175prUGEn3B1gUDkz32u8fnF9Nx3Ly
 232303 357d4rAjQhDPaWhZrBAFY7aizVPkNSq2DH
 230290 18rdKmjrg1EawxgiVT3ikLExj6GWS2MNCk
 229128 3JjPf13Rd8g6WAyvg8yiPnrsdjJt1NP4FC
 226837 1HWqsgnSd12Gv8SpoUMi1Cj8hp79BTSpW7
 226259 1changemCPo732F6oYUyhbyGtFcNVjprq
 224451 138o15eFWEEPv2ayKW2CZCgVvv5ZaZvomP
 224217 d-752ed0099932a96fbc0a854a4d3a300f
 219697 bc1qnsupj8eqya02nm8v6tmk93zslu2e2z8chlmcej
 219174 s-e3b0c44298fc1c149afbf4c8996fb924
 215870 1Kr6QSydW9bFQG1mXiPNNu6WpJGmUa9i1g
 215691 37p9pUugydmoLpQyFLLqGAgjWmUFERa1Pq
 215520 19iVyH1qUxgywY8LJSbpV4VavjZmyuEyxV
 212059 1dice7EYzJag7SxkdKXLr8Jn14WUb3Cf1
 209001 1F89hmmrtonJfAQNAqDmeDadcw7AsZcvXG
 207701 1NDpZ2wyFekVezssSXv2tmQgmxcoHMUJ7u
 207697 1Bd5wrFxHYRkk4UCFttcPNMYzqJnQKfXUE
 207524 15fXdTyFL1p53qQ8NkrjBqPUbPWvWmZ3G9
 207499 14719bzrTyMvEPcr7ouv9R8utncL9fKJyf
 207424 18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs
 207385 1J4yuJFqozxLWTvnExR4Xxe9W4B89kaukY
 207376 1Bqm5MDo82m1FTxV3qYNUUEKnESPRhk9jd
 207256 1HVpyjYEPwQhvRQ3dL8tGe9kiydti616sX
 207228 17NKcZNXqAbxWsTwB1UJHjc9mQG3yjGALA
 207218 1HjDauL2kth6KJUz5vX198Nvp1xN1hgYRb
 207187 13h1DP2Boo9TAsenphroACxhNy7pGxDYXd
 207138 1MSzmVTBaaSpKDARK3VGvP8v7aCtwZ9zbw
 207053 1GoK6fv4tZKXFiWL9NuHiwcwsi8JAFiwGK
 207006 13HFqPr9Ceh2aBvcjxNdUycHuFG7PReGH4
 206834 1L4EThM6x3Rd2PjNbs1U136FpMq4Gmo3fJ
 206826 14ChPPM8rPYJeHnw6kMVUDnNNKx1KnjYW4
 206808 1AdN2my8NxvGcisPGYeQTAKdWJuUzNkQxG
 206760 1DpsR91YmHUDTtiuH1pPCuG3RqAkmg6YKB
 206707 1PeohaRGaTF8cSzDqP1yYfzDah66xiriEQ
 206664 1JmcV7G3r8k7ev2EkS84MmsvxGyhiRGP84
 206572 1HZHBnH2FbHNWieMxAh4xBPfgfuxW15UPt
 206469 18czPiA9PcCs7rFTBZnhvNAWuh1pEZRpGJ
 206346 12Cf6nCcRtKERh9cQm3Z29c9MWvQuFSxvT
 206344 1MPerpQzTABa1K2eXQxsQTDSZtDQHWf6vk
 206247 1dice1e6pdhLzzWQq7yMidf6j8eAg7pkY
 206243 18XSLnBZ8ydMUkaifU6sQBMJzmm7JvDeUp
 205690 bc1quq29mutxkgxmjfdr7ayj3zd9ad0ld5mrhh89l2
 203334 3QQB6AWxaga6wTs6Xwq8FYppgrGinGu15f
 201993 3M92sq9ssFaNbEwF47uteVKJsbw125juS7
 199135 1AScRhqdXMrJyxNmjEapMZi1PLFsqmLquG
 196271 18p9Ftp3m4435tdpZTvoBsm3yjUgkvTF2b
 193271 33fDiKKhr2F2uRv2jJzdKT3ECuK3wzCq5d

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Background
To follow up on List of all Bitcoin addresses with a balance and this post, I made a list of all Bitcoin addresses that have ever been used.

The data
See alladdresses.loyce.club (new location)
I now have the resources (RAM, CPU power and disk space) and code to show unique addresses in their original order. Each address is only shown once. I have 2 large files:

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):

Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

2. All Bitcoin addresses ever used, sorted by address, without duplicates.
Sample: addresses_sorted.txt.gz: (Warning: 16 GB):

Code:

1111111111111111111114oLvT2
111111111111111111112BEH2ro
111111111111111111112xT3273
.......
s-ffd80dee5966fb23c1a483b28f6bfcbc
s-fff5d0faa9628c188e97661f0e185fce
s-ffff291613d413b4ac128df96a462294

Updates
Sorting a list that doesn't fit in the server's RAM is slow. Therefore I only update both large files (addresses_sorted.txt.gz and addresses_in_order_of_first_appearance.txt.gz) twice a month (on the 6^th and 21^st, updates take more than a day). Check the file date here to see how old it is. If an update fails, please post here.
In between updates, I create daily updates: alladdresses.loyce.club/daily_updates/. These txt-files contain unique addresses (for that day) in order of appearance.
I won't keep older snapshots.

Bandwidth
This server is allowed 5 TB bandwidth per month. If it runs out, it probably gets suspended Tongue

I haven't tried that yet.

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 kB/s (or recently 100 kB/s)) with daily updates. All data presented in this topic comes from Blockchair.

No spam please.
Self-moderated against spam. Discussion and questions are welcome.

Q&A

Quote from: MrFreeDragon on August 17, 2020, 01:56:46 PM

Can you please clarify, what is the type of these d- and s- addresses?

This is how Blockchair.com shows OP_RETURN. From the main page the search field doesn't show them, but you can replace a Bitcoin address in the URL to find them: https://blockchair.com/bitcoin/address/d-d0d953f2e7043342540a1407243e49fe.

Tips and tricks
Some suggestions for Linux/VPS users:

Code:

wget http://alladdresses.loyce.club/addresses_sorted.txt.gz -O - | gunzip > addresses_sorted.txt

This doesn't save the .gz but extracts it while downloading.

Code:

comm -12 <(sort list.txt) addresses_sorted.txt

This outputs all Bitcoin addresses from "list.txt" that have ever been funded.

Code:

comm -12 <(sort list.txt) addresses_sorted.txt > output.txt

This does the same, but writes to output.txt instead of console.
This search is fast, even with millions of addresses in list.txt, it's mainly limited by how fast your computer can read from disk.

Topic: List of all Bitcoin addresses ever used - currently UNavailable on temp location - page 8. (Read 3665 times)