Pages:

Author

Topic: Bitcoin block data (728 GB): inputs, outputs and transactions - page 3. (Read 3190 times)

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Big update

First: the Torrents won't last. I got an anonymous sponsor for a dedicated server. The Seedbox expires on April 6, so the Torrens won't work much longer:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

The data is back at it's original location:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Back online! The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (469 GB)
blockdata.loyce.club/outputs/ (145 GB)
blockdata.loyce.club/transactions/ (52 GB)

This server is has a 50 TB/month bandwidth limit. So far, at most a few people per month downloaded this (crazy amount of) data, so it should be sufficient.

Note: Some files are missing, those are still being updated now. By tomorrow automated daily updates should be on track again.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 16, 2022, 07:21:04 AM

Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust.

For projects like this topic I don't really need AWS-level uptime, so I gladly go for the budget hosts. I run this project at Racknerd (good deals via Lowendtalk.com), and (so far) I'm quite happy with it. If I look at prices on their own site, it's much higher.
You could also go for RamNode, pay by the hour and less cheap, but from what I've seen RamNode has very solid performance too.
I don't think any of those are run from their garage Wink

I run this project at Gullo's Hosting (again: see Lowendtalk.com for deals), which has the unique feature that it's run by one guy. Servers are international, so not from his garage, but you'll always deal with the same guy. He'll try his best, which for this project is everything I need.

I also found out that several of the higher range VPS providers don't accept Bitcoin, or demand a copy of my passport (yeah, right!) first.

Quote

Have you seen https://www.sync.com/pricing-individual/ ?

That's not a VPS, so won't help me much.

PawGo

legendary

Activity: 952

Merit: 1385

Quote from: LoyceV on March 16, 2022, 07:05:08 AM

See:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(nobody ever reads the OP)

;-) Your point.
The same applies to the privacy policy and the washing machine manual.

Quote

For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

Memory usage to get balances: ~18GB. That would take big VPS.

Ops. I did not use it, only transactions dump. It is up to you if you want to use it, even partially.

Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust. Once I tried one hosting just to compare with my main one, but service and reliability was terrible. I had impression all that companies which offer servers for BTC are running their "datacenters" somewhere in the garage.
Have you seen https://www.sync.com/pricing-individual/ ?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 16, 2022, 06:57:28 AM

How do you currently extract the data you publish?

See:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(nobody ever reads the OP)

Quote

For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

Memory usage to get balances: ~18GB. That would take a strong VPS.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: yogg on March 14, 2022, 07:30:19 PM

Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.

Quote from: LoyceV on March 15, 2022, 05:01:59 AM

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

I would demand partial/full refund.

PawGo

legendary

Activity: 952

Merit: 1385

Quote from: LoyceV on March 16, 2022, 06:26:04 AM

Quote from: ABCbits on March 16, 2022, 05:57:02 AM

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.

Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

Wait, I am lost. How do you currently extract the data you publish?
For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

I've updated all links to the new host. Torrents should work again too:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Quote from: ABCbits on March 16, 2022, 05:57:02 AM

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.

Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 15, 2022, 06:08:08 PM

Ugh ... that leaves me with no other choice than use touch.

I used wget instead, this should preserve file dates.

Quote from: PawGo on March 16, 2022, 03:22:39 AM

Is that true that if you tell average user to leave vi

It can't be: the average user doesn't use vi.

PawGo

legendary

Activity: 952

Merit: 1385

Quote from: LoyceV on March 15, 2022, 02:10:30 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

Is that true that if you tell average user to leave vi, you will get a natural RNG or at least a good example of password (compliant with any strong password policy)?

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: LoyceV on March 10, 2022, 02:35:52 PM

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Feel free to test (and/or seed) the torrents if you have some bandwidth and disk space to waste

My VPS doesn't have enough storage for "inputs" torrent, so i didn't try until download is finished. But it starts almost immediately with similar speed on previous speed.

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: LoyceV on March 15, 2022, 02:10:30 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Ugh ... that leaves me with no other choice than use touch.

(Actually I'd prefer to do that .. can't do rsync straight to that appliance and I would rather avoid setting up a buffer VM just for rsync)

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Ha. Makes sense.

Some time ago I started coding a script to get something similar to the all addresses balance dump.
Unfortunately the initial processing of the blockchain was too resource expensive. The process never completed.

However ... This got me thinking..
If the "input outputs transaction" dump is something like a csv file, just compiling the "missing" data with Bitcoin Core and append it to the dump could do the trick.

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

The people who showed me the path to emacs warned me about vi users.
Everytime you use the dd p shortcut, the init entity turns off the numlock on boomer's keyboard so they can no longer login to their windows session. And can't figure out why.
You monster ! Angry

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 15, 2022, 01:54:13 PM

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: ABCbits on March 15, 2022, 07:15:18 AM

Quote from: yogg on March 14, 2022, 07:30:19 PM

Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.

Nothing better than the good old foundations of IT.

Like, emacs is more pure and "C-itonic" as it's been delivered to us mere mortals by the founder of GNU himself. (f*ck vi Tongue

" :wq!" wtf)

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

Quote from: LoyceV on March 15, 2022, 05:01:59 AM

Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions" Cheesy

Yes, this is the one I meant to use for my project. (Bitcoin addresses with a balance)

I can find the 700 GB and host "inputs, outputs and transactions"

Luckily the bandwidth is unmetered so I can also provide a mirror for some of the other dumps.

Quote from: LoyceV on March 15, 2022, 09:14:02 AM

It could work, but having to upload updates instead of just processing them on the server is far from ideal.

How is the data processed ? Running on Bitcoin Core ?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 15, 2022, 09:01:32 AM

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.

I need a VPS, this looks a bit like "shared hosting". I guess I wasn't specific enough. It could work, but having to upload updates instead of just processing them on the server is far from ideal.
A bigger deal breaker is the fact that Hetzner doesn't accept Bitcoin. I don't want to link my credit card to anything crypto-related. From what I've read, I'd expect Hetzner to offer good quality hosting.

PawGo

legendary

Activity: 952

Merit: 1385

Quote from: LoyceV on March 15, 2022, 07:39:48 AM

If anyone finds a good offer on 1 TB storage, feel free to post it.

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
https://www.hetzner.com/storage/storage-box

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ABCbits on March 15, 2022, 07:15:18 AM

People still use FTP these days? I remember using FTP with FileZilla long time ago.

I still use FTP on my local network, where I don't mind the lack of encryption.

Quote

I would demand partial/full refund.

It's paid per month, and really not worth the effort. I'd much rather have them fix the problems so I can finally keep this data online. I've uploaded it too many times already, and the whole point of having it on a server is so I can access it when needed.
If anyone finds a good offer on 1 TB storage, feel free to post it.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 14, 2022, 07:30:19 PM

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ? Grin

Lol. I'll have to stand in the hallway to count the blinks.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

Code:

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.842800] [Hardware Error]: Corrected error, no action required.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.846023] [Hardware Error]: CPU:8 (15:1:2) MC4_STATUS[-|CE|MiscV|-|AddrV|-|CECC]: 0x9c44c00001080a13

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.849212] [Hardware Error]: Error Addr: 0x0000001f8d366350

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.852463] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.855864] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no timeout)

Quote

More seriously, I can help hosting "balances" if that interests you.

Quote

Let me know, I can provide an FTP access to you.

FTP, scp, rsync, anything works for me

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: LoyceV on March 13, 2022, 10:10:51 AM

I'm not amused.

More seriously, I can help hosting "balances" if that interests you.
I will need this archive for a personal project anyway, it's not a big deal to make it available on port 80.

I have some extra storage space avail on my setup.

Let me know, I can provide an FTP access to you.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: LoyceV on March 09, 2022, 07:49:56 AM

Quote

Testing it now.

Only one out of 2 Seedboxes is currently uploading (at a very decent speed).

I just noticed i finished download both torrent. Here are some details,
1. I remove tracker on both torrent before i start download process.
2. "output" downloaded started some time (about 15 - 30 minutes) after i add it.
3. Average speed was 24-30MB/s. Although "transaction" have 2x speed after it connected to 2 seeder.
4. I use qbittorent-nox with default configuration.

I think i can conclude that you made the torrent files correctly and there's no problem with your seedbox.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Barely 3 days later, I received an email:

Quote

There is a backend software glitch which can cause all data on /home to be removed.

I still have "outputs" and "transactions" on the other server, but this means I'll have to upload 469 GB from my local backup again. It also means they've moved me to another server, which means the above links are no longer valid.
I'm not amused.