List of all Bitcoin addresses ever used - currently UNavailable on temp location - page 4.

PrimeNumber7

copper member

Activity: 1652

Merit: 1901

Amazon Prime Member #7

Quote from: NotATether on June 06, 2021, 01:06:14 AM

Quote from: LoyceV on June 05, 2021, 01:29:35 PM

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver Undecided

Memory/thread constraints should not be an issue. Executing a script remotely on a server optimized for script requirements is trivial, and uploading an output file a single time to your server should not be an issue for a ~20 GB file.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: LoyceV on June 05, 2021, 01:29:35 PM

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

Depends on how much time is "long time".

Leaving it single threaded (as most shell commands already are) will probably be alright as long as it doesn't take more than a few hours.

But if it has obscene memory requirements then it'll be too much for my box. I only got 9 out of 16GB ram free and I hate force-rebooting the notatether.com webserver Undecided

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: NotATether on June 05, 2021, 09:45:47 AM

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine

Interesting Cheesy

Can it also handle the occasional update? I have "2 methods" now: one that takes a long time but works, and one that's much faster but gives different results once in a while. And because testing takes so long, I haven't found the problem yet.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: LoyceV on June 04, 2021, 04:28:23 AM

Quote from: NotATether on June 04, 2021, 12:54:36 AM

Or you can just ask me nicely and I'll host it on my site's public directory.

What bandwidth limitations do you have? Cheesy

I don't just want to make problem your problem.

My provider gives me a cool 100TB monthly bandwidth cap so I will be fine

these boxes are designed for highly intensive torrenting.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PrimeNumber7 on June 04, 2021, 01:53:48 PM

Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.

Obviously, I won't use a host with "unlimited" bandwidth. That's never real. And I've seen shared hosts that don't allow hosting large files. But for a VPS, I pay for the bandwidth limit, and it's up to the provider to ensure it's profitable for them.

Quote

Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.

I've never used a creditcard for anything crypto-related, and I want to keep it that way.
I found another reason to pay upfront in crypto instead of using my creditcard: The tale of the July 4th surprise $2700 AWS bill.

Quote

There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.

They're also in the money making business, and their paying customers pay for the bandwidth used by, well, people like me Tongue

PrimeNumber7

copper member

Activity: 1652

Merit: 1901

Amazon Prime Member #7

Quote from: LoyceV on June 04, 2021, 04:28:23 AM

Quote from: PrimeNumber7 on June 03, 2021, 11:30:42 PM

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

I get this (in England):

Code:

-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s

The 4 hour quote appears to be the result of my crappy WiFi connection on my back porch. I was able to reproduce a ~10 minute download estimate via a datacenter. I have been able to transfer ~a half terabyte worth of videos stored in a storage bucket in seconds.

Quote from: LoyceV on June 04, 2021, 04:28:23 AM

Quote from: PN7

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

There's 2 problems with that: I don't want to use a creditcard, and I don't want anyone who downloads it to require a creditcard. If I need to charge a few dollars per download, I'd rather set it up myself to accept Bitcoin payments.

That is a reasonable desire, however it is something that is more difficult as you are making big data available to the public. Service providers have limited network infrastructure, and need to pay for data sent to the internet, regardless of if they have data caps, or charge you for egress/outgoing data. If you are freely sharing a 10 or 20 GB file(s) using a service provider that does not charge per data transferred, you will eventually get kicked off from that service provider.

Another point is that many people look at bitcoin-related data today. The fact that someone is looking at blockchain data is not the privacy leak that it might have been 10 years ago.

Quote from: LoyceV on June 04, 2021, 04:28:23 AM

Quote from: PN7

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.

My other project (List of all Bitcoin addresses with a balance) is closing in on it's 2 TB montly bandwidth limit. I'd hate to have to setup a payment system, especially since this is basically just mirroring data from Blockchair.com.

There is a reason why blockchair throttles downloads, and why they charge as much as they do for an API key.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: mike2077 on June 04, 2021, 09:29:59 AM

I've got the file, thank you so much for sharing.

You're welcome

Judging by the 64 GB outgoing traffic, the file was downloaded 3 times. If anyone else wants it:

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB.

~~I'll nuke this VPS tomorrow.~~ It's gone, until I find a more permanent solution.

mike2077

newbie

Activity: 18

Merit: 10

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

Quote from: mike2077 on June 03, 2021, 09:20:25 AM

BTW, I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

I've got the file, thank you so much for sharing.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PrimeNumber7 on June 03, 2021, 11:30:42 PM

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

I get this (in England):

Code:

-                     0%[                    ] 139.10M  33.9MB/s    eta 10m 15s

Quote from: NotATether on June 04, 2021, 12:54:36 AM

Or you can just ask me nicely and I'll host it on my site's public directory.

What bandwidth limitations do you have? Cheesy

I don't just want to make problem your problem.

I remember I had another offer:

Quote from: renedx on January 19, 2021, 11:23:15 PM

For living we host high-end enterprise, just in case you need some space or mirrors, you’re welcome if ever in need.

Is this offer still valid?

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: PrimeNumber7 on June 03, 2021, 11:30:42 PM

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

Or you can just ask me nicely and I'll host it on my site's public directory.

PrimeNumber7

copper member

Activity: 1652

Merit: 1901

Amazon Prime Member #7

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

Update: I got you http://107.191.98.18/addresses_sorted.txt.gz ! It's 19 GB. Please let me know when I can nuke the VPS again.

"Your download will take ~4 hours to complete"

IMO, you should upload the file to a GCS/AWS/Azure/Oracle/etc storage bucket, set the permissions to "anyone can access" but set the object so that the "requestor pays" for downloads. This will result in you paying under a dollar per month in storage costs, but anyone who accesses your file will pay a few dollars to get your data in seconds.

Maintainng a multigigabyte file that is accessible to the public for free, that can be accessed unlimited times is really not feasible.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: LoyceV on June 03, 2021, 09:55:10 AM

Quote from: mike2077 on June 03, 2021, 09:20:25 AM

BTW, I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.

Most cloud storage sites can't upload files several GB large well without constantly breaking the connection, and will throttle the download speed even more which makes them unsuitable for downloading those files as well.

Also it's not 50GB of free storage, it's much smaller than that. Most of that free storage is temporary and expires after a year.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: mike2077 on June 03, 2021, 09:20:25 AM

BTW, I think mega - mega.co.nz give you something like 50GB of storage for free.

That's a terrible site, I used it once to download a large file, it forced me to install their program first. So I prefer a VPS.
I'll let you know when it's available.

Update: I got you ~~http://107.191.98.18/addresses_sorted.txt.gz~~ ! It's 19 GB. Please let me know when I can nuke the VPS again.
Update: link expired.

mike2077

newbie

Activity: 18

Merit: 10

If you can and its not too hard for you, version 1 would be awesome!

Quote from: LoyceV on June 02, 2021, 09:03:40 AM

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):
Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

Thanks.

BTW, I think mega - mega.co.nz give you something like 50GB of storage for free.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: mike2077 on June 02, 2021, 07:40:38 AM

Any news from the provider, when your server is coming back up?

Nope, they're awefully quiet Sad

Quote from: LoyceV on August 01, 2020, 04:05:46 AM

1. All Bitcoin addresses ever used, in chronological order, without duplicates.
Sample: addresses_in_order_of_first_appearance.txt.gz: (Warning: 18 GB):

Code:

1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa
12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX
1HLoD9E4SDFFPDiYfNYnkBLQ85Y51J3Zb1
.......
3GFfFQAFgXKiA1qqUK6rqBpEpG4vZDos6t
3Mbtv47gZ2eN6Fy7owpgHHwSLYHS42P56P
38JyF2RQknBUMETyRT2yGndDJFYSp6hJNg

2. All Bitcoin addresses ever used, sorted by address, without duplicates.
Sample: addresses_sorted.txt.gz: (Warning: 16 GB):

Code:

1111111111111111111114oLvT2
111111111111111111112BEH2ro
111111111111111111112xT3273
.......
s-ffd80dee5966fb23c1a483b28f6bfcbc
s-fff5d0faa9628c188e97661f0e185fce
s-ffff291613d413b4ac128df96a462294

Which one would you prefer? The sorted version is much more practical for most uses, so unless you have a specific reason to want the addresses in chronological order, I'd say go for the sorted file.

mike2077

newbie

Activity: 18

Merit: 10

Hi LoyceV,

Any news from the provider, when your server is coming back up?
Do yo have temp location I can download files from ?

Thanks

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: mike2077 on May 25, 2021, 11:55:39 PM

I can suggest sharing files via torrent. Technically you'd have to upload a lot less since a lot of data will be transferred between people downloading.

I haven't tried that yet for these reasons: I don't want to upload from my desktop, so I still need a VPS. I don't expect many simultaneous downloads, so most of the uploads will still come from me. Every update will make an existing torrent useless again (and I don't want to keep posting new magnet links).

Quote

Linux question

what is the benefit of two directional syntax -

Code:

cat <(gunzip -c addresses_sorted.txt.gz) > secondgunzip

instead of just

Code:

gunzip addresses_sorted.txt.gz

or

Code:

gunzip -c addresses_sorted.txt.gz > out.txt

?

I was isolating a part from the longer code:

Code:

This:
<(gunzip -c addresses_sorted.txt.gz)
Came from:
....t -f2) <(comm -13 <(gunzip -c addresses_sorted.txt.gz) <(s....

I didn't edit it so it's clear where it came from.

Quote

or another exmple:

Same reason

I am now pretty sure the inconsistent results were caused by using sort S, --buffer-size=SIZE. I was trying to be smart enforcing efficient memory usage, but I now believe this sometimes showed an error, which was then piped into the next command. If I omit the -S40% part, it works fine. This is actually good news, because it's much faster than my previous method.

mike2077

newbie

Activity: 18

Merit: 10

I can suggest sharing files via torrent. Technically you'd have to upload a lot less since a lot of data will be transferred between people downloading.
You can share magnet link here, seed for some time and then let people share it, just a suggestion.

Linux question

what is the benefit of two directional syntax -

Code:

cat <(gunzip -c addresses_sorted.txt.gz) > secondgunzip

instead of just

Code:

gunzip addresses_sorted.txt.gz

or

Code:

gunzip -c addresses_sorted.txt.gz > out.txt

?

or another exmple:

Code:

cat <(sort -u daily_updates/*.txt) > firstsort

instead of

Code:

cat daily_updates/*.txt |sort -u > firstsort

Thanks

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: mike2077 on May 25, 2021, 04:57:34 AM

all links are down.

Sorry, I forgot to post this here:

Quote from: ?? on ??

This server is currently offline. I don't know why (yet).

Still no response from my webhost Sad

mike2077

newbie

Activity: 18

Merit: 10

Hi there, first of all thanks for nice data source.

Could you check the links , all links are down.
Thanks

Topic: List of all Bitcoin addresses ever used - currently UNavailable on temp location - page 4. (Read 4068 times)