What the addres with prefix s- and prefix m- ?
They are not addresses, they are output scripts that don't have any corresponding address and blockchair.com explorer uses an undocumented method to convert them into these strange looking strings.
Topic: List of all Bitcoin addresses ever used - currently UNavailable on temp location - page 3. (Read 3697 times)
September 19, 2021, 04:00:21 AM
September 19, 2021, 03:56:40 AM
What the addres with prefix s- and prefix m- ?
They are not addresses, they are output scripts that don't have any corresponding address and blockchair.com explorer uses an undocumented method to convert them into these strange looking strings. September 19, 2021, 03:49:56 AM
What the addres with prefix s- and prefix m- and prefix d- ?
August 13, 2021, 06:01:49 AM
This is up to date as of 2021-08-12.
If it's the same file notatether.com published, it was last updated in May.I still haven't found a decent affordable replacement host.
August 12, 2021, 08:17:47 PM
Another way to generate the file yourself, from a local copy of the blockchain: https://github.com/graymauser/btcposbal2csv
August 12, 2021, 06:48:20 PM
For anyone stumbling open this in the future, I have created a torrent for this file, which accesses the web link.
Web Link: https://files.notatether.com/public/loycev/addresses_sorted.txt.gz
Torrent Magnet URL
.torrent file, good until 2022-09
This is up to date as of 2021-08-12. It will not be updated in the future, but is being left here to help, at least for now.
Web Link: https://files.notatether.com/public/loycev/addresses_sorted.txt.gz
Torrent Magnet URL
.torrent file, good until 2022-09
This is up to date as of 2021-08-12. It will not be updated in the future, but is being left here to help, at least for now.
June 13, 2021, 04:33:53 AM
Have you considered creating a script to update your existing list once you have a list of addresses? Your script could start at block_n and add an address if it is not already in your list. This should reduce read/write operations pretty significantly.
That's what I tried to do for the addresses in chronological order, but it gave inconsistent results (and due to the time it takes to test it, I'm still not sure what caused it). I could probably do the same for the sorted addresses, by using comm instead of sort. That means only reading the compressed big file from disk, reading and sorting the weekly addition, and writing the new compressed file to disk. This does make more sense and reduces disk writes 
June 12, 2021, 10:28:25 PM
I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.On second thought, I just had one of my servers' disks fail a couple days ago (all data was lost), so I'm not comfortable running these updating scripts on the rest of my hardware with all that grinding until I can set up a proper backup plan for my TBs of data.
June 12, 2021, 04:16:31 AM
I'm not comfortable running these updating scripts on the rest of my hardware
No worries, I'll just wait for the right hosting offer again.This is the reason I don't want to do a lot of testing on my laptop SSD too.
June 12, 2021, 03:59:14 AM
I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times.On second thought, I just had one of my servers' disks fail a couple days ago (all data was lost), so I'm not comfortable running these updating scripts on the rest of my hardware with all that grinding until I can set up a proper backup plan for my TBs of data.
June 06, 2021, 04:01:35 PM
I'm trying to figure out how much RAM it uses in the worst case.
I can limit sort's memory usage. I'm more worried about the grinding this causes on the hard drive. I have no idea how much data gets read and written to sort 30 GB, but I assume every bit gets pickup up at least several times. June 06, 2021, 02:51:54 PM
Memory requirements can be low (then sort uses more tmp files instead).
This is what I'm worried about. I remember you writing somewhere along the lines of the sort process takes an obscene amount of memory on this file, or maybe that was in the addresses-with-a-balance project. That's why I'm trying to figure out how much RAM it uses in the worst case.
June 06, 2021, 10:29:51 AM
Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.
But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver
It uses a bunch of pipes, so the load it causes is more like 2-3 than 1. Memory requirements can be low (then sort uses more tmp files instead). It should be done within a few hours, and doesn't need frequent updates (once every 2 weeks will be fine). But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver
June 06, 2021, 03:14:23 AM
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
Depends on how much time is "long time".
Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.
But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver
June 06, 2021, 02:06:14 AM
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
Depends on how much time is "long time".
Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.
But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver
June 05, 2021, 02:29:35 PM
My provider gives me a cool 100TB monthly bandwidth cap so I will be fine
Interesting 
Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.
June 05, 2021, 10:45:47 AM
What bandwidth limitations do you have? I don't just want to make problem your problem.
I don't just want to make problem your problem.My provider gives me a cool 100TB monthly bandwidth cap so I will be fine
these boxes are designed for highly intensive torrenting. June 05, 2021, 04:07:25 AM
Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.
Obviously, I won't use a host with "unlimited" bandwidth. That's never real. And I've seen shared hosts that don't allow hosting large files. But for a VPS, I pay for the bandwidth limit, and it's up to the provider to ensure it's profitable for them.Quote
Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.
I've never used a creditcard for anything crypto-related, and I want to keep it that way.I found another reason to pay upfront in crypto instead of using my creditcard: The tale of the July 4th surprise $2700 AWS bill.
Quote
There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.
They're also in the money making business, and their paying customers pay for the bandwidth used by, well, people like me 
June 04, 2021, 02:53:48 PM
Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.
"Your download will take ~4 hours to complete"Code:
- 0%[ ] 139.10M 33.9MB/s eta 10m 15s
Quote from: PN7
IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.
There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.
Quote from: PN7
Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.
My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com. June 04, 2021, 10:38:19 AM
I've got the file, thank you so much for sharing.
You're welcome Judging by the 64 GB outgoing traffic, the file was downloaded 3 times. If anyone else wants it:
I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB.
Jump to: