Pages:
Author

Topic: Bitcoin block data (728 GB): inputs, outputs and transactions - page 3. (Read 3204 times)

legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Big update

First: the Torrents won't last. I got an anonymous sponsor for a dedicated server. The Seedbox expires on April 6, so the Torrens won't work much longer:
New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

The data is back at it's original location:
This server is has a 50 TB/month bandwidth limit. So far, at most a few people per month downloaded this (crazy amount of) data, so it should be sufficient.

Note: Some files are missing, those are still being updated now. By tomorrow automated daily updates should be on track again.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust.
For projects like this topic I don't really need AWS-level uptime, so I gladly go for the budget hosts. I run this project at Racknerd (good deals via Lowendtalk.com), and (so far) I'm quite happy with it. If I look at prices on their own site, it's much higher.
You could also go for RamNode, pay by the hour and less cheap, but from what I've seen RamNode has very solid performance too.
I don't think any of those are run from their garage Wink I run this project at Gullo's Hosting (again: see Lowendtalk.com for deals), which has the unique feature that it's run by one guy. Servers are international, so not from his garage, but you'll always deal with the same guy. He'll try his best, which for this project is everything I need.

I also found out that several of the higher range VPS providers don't accept Bitcoin, or demand a copy of my passport (yeah, right!) first.

Quote
That's not a VPS, so won't help me much.
legendary
Activity: 952
Merit: 1385
See:
Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.
(nobody ever reads the OP)

;-) Your point.
The same applies to the privacy policy and the washing machine manual.

Quote
Quote
For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.
Memory usage to get balances: ~18GB. That would take big VPS.

Ops. I did not use it, only transactions dump. It is up to you if you want to use it, even partially.

Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust. Once I tried one hosting just to compare with my main one, but service and reliability was terrible. I had impression all that companies which offer servers for BTC are running their "datacenters" somewhere in the garage.
Have you seen https://www.sync.com/pricing-individual/ ?
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
How do you currently extract the data you publish?
See:
Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.
(nobody ever reads the OP)

Quote
For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.
Memory usage to get balances: ~18GB. That would take a strong VPS.
legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

I would demand partial/full refund.
legendary
Activity: 952
Merit: 1385

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.
Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

Wait, I am lost. How do you currently extract the data you publish?
For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
I've updated all links to the new host. Torrents should work again too:



You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.
Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Ugh ... that leaves me with no other choice than use touch.
I used wget instead, this should preserve file dates.

Is that true that if you tell average user to leave vi
It can't be: the average user doesn't use vi.
legendary
Activity: 952
Merit: 1385
(f*ck vi Tongue " :wq!" wtf)
What's wrong with vi? I literally use it every day Cheesy

Is that true that if you tell average user to leave vi, you will get a natural RNG or at least a good example of password (compliant with any strong password policy)?

legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
I've finalized my update, and after 10 months, I'm happy to announce This project is back online!


Feel free to test (and/or seed) the torrents if you have some bandwidth and disk space to waste Smiley

My VPS doesn't have enough storage for "inputs" torrent, so i didn't try until download is finished. But it starts almost immediately with similar speed on previous speed.
legendary
Activity: 2464
Merit: 3158
However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.
I actually prefer rsync, because scp changes the original file date, which is undesirable.

Ugh ... that leaves me with no other choice than use touch.



(Actually I'd prefer to do that .. can't do rsync straight to that appliance and I would rather avoid setting up a buffer VM just for rsync)



Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Ha. Makes sense. Smiley
Some time ago I started coding a script to get something similar to the all addresses balance dump.
Unfortunately the initial processing of the blockchain was too resource expensive. The process never completed.

However ... This got me thinking..
If the "input outputs transaction" dump is something like a csv file, just compiling the "missing" data with Bitcoin Core and append it to the dump could do the trick.

(f*ck vi Tongue " :wq!" wtf)
What's wrong with vi? I literally use it every day Cheesy



The people who showed me the path to emacs warned me about vi users.
Everytime you use the dd p shortcut, the init entity turns off the numlock on boomer's keyboard so they can no longer login to their windows session. And can't figure out why.
You monster !  Angry
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.
I actually prefer rsync, because scp changes the original file date, which is undesirable.

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(f*ck vi Tongue " :wq!" wtf)
What's wrong with vi? I literally use it every day Cheesy
legendary
Activity: 2464
Merit: 3158
Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.


Nothing better than the good old foundations of IT. Smiley
Like, emacs is more pure and "C-itonic" as it's been delivered to us mere mortals by the founder of GNU himself. (f*ck vi Tongue " :wq!" wtf)

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.




Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions" Cheesy

Yes, this is the one I meant to use for my project. (Bitcoin addresses with a balance)

I can find the 700 GB and host "inputs, outputs and transactions" Smiley
Luckily the bandwidth is unmetered so I can also provide a mirror for some of the other dumps.


It could work, but having to upload updates instead of just processing them on the server is far from ideal.

How is the data processed ? Running on Bitcoin Core ?
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
I need a VPS, this looks a bit like "shared hosting". I guess I wasn't specific enough. It could work, but having to upload updates instead of just processing them on the server is far from ideal.
A bigger deal breaker is the fact that Hetzner doesn't accept Bitcoin. I don't want to link my credit card to anything crypto-related. From what I've read, I'd expect Hetzner to offer good quality hosting.
legendary
Activity: 952
Merit: 1385
If anyone finds a good offer on 1 TB storage, feel free to post it.

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
https://www.hetzner.com/storage/storage-box

legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
People still use FTP these days? I remember using FTP with FileZilla long time ago.
I still use FTP on my local network, where I don't mind the lack of encryption.

Quote
I would demand partial/full refund.
It's paid per month, and really not worth the effort. I'd much rather have them fix the problems so I can finally keep this data online. I've uploaded it too many times already, and the whole point of having it on a server is so I can access it when needed.
If anyone finds a good offer on 1 TB storage, feel free to post it.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ? Grin
Lol. I'll have to stand in the hallway to count the blinks.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:
Code:
Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.842800] [Hardware Error]: Corrected error, no action required.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.846023] [Hardware Error]: CPU:8 (15:1:2) MC4_STATUS[-|CE|MiscV|-|AddrV|-|CECC]: 0x9c44c00001080a13

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.849212] [Hardware Error]: Error Addr: 0x0000001f8d366350

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.852463] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.855864] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no timeout)

Quote
More seriously, I can help hosting "balances" if that interests you.
Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions" Cheesy

Quote
Let me know, I can provide an FTP access to you.
FTP, scp, rsync, anything works for me Smiley
legendary
Activity: 2464
Merit: 3158
I'm not amused.

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ? Grin



More seriously, I can help hosting "balances" if that interests you.
I will need this archive for a personal project anyway, it's not a big deal to make it available on port 80.

I have some extra storage space avail on my setup. Smiley

Let me know, I can provide an FTP access to you.
legendary
Activity: 2870
Merit: 7490
Crypto Swap Exchange
Quote
Testing it now.
Only one out of 2 Seedboxes is currently uploading (at a very decent speed).

I just noticed i finished download both torrent. Here are some details,
1. I remove tracker on both torrent before i start download process.
2. "output" downloaded started some time (about 15 - 30 minutes) after i add it.
3. Average speed was 24-30MB/s. Although "transaction" have 2x speed after it connected to 2 seeder.
4. I use qbittorent-nox with default configuration.

I think i can conclude that you made the torrent files correctly and there's no problem with your seedbox.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Barely 3 days later, I received an email:
Quote
There is a backend software glitch which can cause all data on /home to be removed.
I still have "outputs" and "transactions" on the other server, but this means I'll have to upload 469 GB from my local backup again. It also means they've moved me to another server, which means the above links are no longer valid.
I'm not amused.
Pages:
Jump to: