Pages:

Author

Topic: Bitcoin block data (728 GB): inputs, outputs and transactions - page 3. (Read 2869 times)

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

I've updated all links to the new host. Torrents should work again too:

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Quote from: ABCbits on March 16, 2022, 05:57:02 AM

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.

Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 15, 2022, 06:08:08 PM

Ugh ... that leaves me with no other choice than use touch.

I used wget instead, this should preserve file dates.

Quote from: PawGo on March 16, 2022, 03:22:39 AM

Is that true that if you tell average user to leave vi

It can't be: the average user doesn't use vi.

PawGo

legendary

Activity: 952

Merit: 1367

Quote from: LoyceV on March 15, 2022, 02:10:30 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

Is that true that if you tell average user to leave vi, you will get a natural RNG or at least a good example of password (compliant with any strong password policy)?

ABCbits

legendary

Activity: 2856

Merit: 7410

Crypto Swap Exchange

Quote from: LoyceV on March 10, 2022, 02:35:52 PM

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Feel free to test (and/or seed) the torrents if you have some bandwidth and disk space to waste

My VPS doesn't have enough storage for "inputs" torrent, so i didn't try until download is finished. But it starts almost immediately with similar speed on previous speed.

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: LoyceV on March 15, 2022, 02:10:30 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Ugh ... that leaves me with no other choice than use touch.

(Actually I'd prefer to do that .. can't do rsync straight to that appliance and I would rather avoid setting up a buffer VM just for rsync)

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Ha. Makes sense.

Some time ago I started coding a script to get something similar to the all addresses balance dump.
Unfortunately the initial processing of the blockchain was too resource expensive. The process never completed.

However ... This got me thinking..
If the "input outputs transaction" dump is something like a csv file, just compiling the "missing" data with Bitcoin Core and append it to the dump could do the trick.

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

The people who showed me the path to emacs warned me about vi users.
Everytime you use the dd p shortcut, the init entity turns off the numlock on boomer's keyboard so they can no longer login to their windows session. And can't figure out why.
You monster ! Angry

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 15, 2022, 01:54:13 PM

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Quote from: yogg on March 15, 2022, 01:54:13 PM

(f*ck vi

" :wq!" wtf)

What's wrong with vi? I literally use it every day Cheesy

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: ABCbits on March 15, 2022, 07:15:18 AM

Quote from: yogg on March 14, 2022, 07:30:19 PM

Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.

Nothing better than the good old foundations of IT.

Like, emacs is more pure and "C-itonic" as it's been delivered to us mere mortals by the founder of GNU himself. (f*ck vi Tongue

" :wq!" wtf)

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

Quote from: LoyceV on March 15, 2022, 05:01:59 AM

Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions" Cheesy

Yes, this is the one I meant to use for my project. (Bitcoin addresses with a balance)

I can find the 700 GB and host "inputs, outputs and transactions"

Luckily the bandwidth is unmetered so I can also provide a mirror for some of the other dumps.

Quote from: LoyceV on March 15, 2022, 09:14:02 AM

It could work, but having to upload updates instead of just processing them on the server is far from ideal.

How is the data processed ? Running on Bitcoin Core ?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 15, 2022, 09:01:32 AM

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.

I need a VPS, this looks a bit like "shared hosting". I guess I wasn't specific enough. It could work, but having to upload updates instead of just processing them on the server is far from ideal.
A bigger deal breaker is the fact that Hetzner doesn't accept Bitcoin. I don't want to link my credit card to anything crypto-related. From what I've read, I'd expect Hetzner to offer good quality hosting.

PawGo

legendary

Activity: 952

Merit: 1367

Quote from: LoyceV on March 15, 2022, 07:39:48 AM

If anyone finds a good offer on 1 TB storage, feel free to post it.

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
https://www.hetzner.com/storage/storage-box

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ABCbits on March 15, 2022, 07:15:18 AM

People still use FTP these days? I remember using FTP with FileZilla long time ago.

I still use FTP on my local network, where I don't mind the lack of encryption.

Quote

I would demand partial/full refund.

It's paid per month, and really not worth the effort. I'd much rather have them fix the problems so I can finally keep this data online. I've uploaded it too many times already, and the whole point of having it on a server is so I can access it when needed.
If anyone finds a good offer on 1 TB storage, feel free to post it.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: yogg on March 14, 2022, 07:30:19 PM

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ? Grin

Lol. I'll have to stand in the hallway to count the blinks.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

Code:

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.842800] [Hardware Error]: Corrected error, no action required.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.846023] [Hardware Error]: CPU:8 (15:1:2) MC4_STATUS[-|CE|MiscV|-|AddrV|-|CECC]: 0x9c44c00001080a13

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.849212] [Hardware Error]: Error Addr: 0x0000001f8d366350

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.852463] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
kernel:[220101.855864] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no timeout)

Quote

More seriously, I can help hosting "balances" if that interests you.

Quote

Let me know, I can provide an FTP access to you.

FTP, scp, rsync, anything works for me

yogg

legendary

Activity: 2464

Merit: 3158

Quote from: LoyceV on March 13, 2022, 10:10:51 AM

I'm not amused.

More seriously, I can help hosting "balances" if that interests you.
I will need this archive for a personal project anyway, it's not a big deal to make it available on port 80.

I have some extra storage space avail on my setup.

Let me know, I can provide an FTP access to you.

ABCbits

legendary

Activity: 2856

Merit: 7410

Crypto Swap Exchange

Quote from: LoyceV on March 09, 2022, 07:49:56 AM

Quote

Testing it now.

Only one out of 2 Seedboxes is currently uploading (at a very decent speed).

I just noticed i finished download both torrent. Here are some details,
1. I remove tracker on both torrent before i start download process.
2. "output" downloaded started some time (about 15 - 30 minutes) after i add it.
3. Average speed was 24-30MB/s. Although "transaction" have 2x speed after it connected to 2 seeder.
4. I use qbittorent-nox with default configuration.

I think i can conclude that you made the torrent files correctly and there's no problem with your seedbox.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

Barely 3 days later, I received an email:

Quote

There is a backend software glitch which can cause all data on /home to be removed.

I still have "outputs" and "transactions" on the other server, but this means I'll have to upload 469 GB from my local backup again. It also means they've moved me to another server, which means the above links are no longer valid.
I'm not amused.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 10, 2022, 04:00:06 PM

For me it works instantly.

That's something

Overnight, "transactions" completed downloading on the other Seedbox, and later "outputs" did the same. I didn't try "inputs" due to disk space limitations. So I think I can conclude it works, and it may or may not start instantly.
I'll stop fiddling and let it run for now, anyone who's in a rush can always go for the http download.

ABCbits

legendary

Activity: 2856

Merit: 7410

Crypto Swap Exchange

Quote from: LoyceV on March 09, 2022, 07:09:53 AM

Quote

Yeah, usually i just rely on DHT and share my IP:Port (if needed) so other user can it directly to list of peers on their torrent software.

How would that IP/port-sharing work? Usually I just click the Torrent or Magnet link, and don't need to manually add IPs.

You manually send it to other people who wish to download the file. Some client have such option after you add torrent file/magnet link. But it's not practical option.

Quote from: LoyceV on March 09, 2022, 07:09:53 AM

Quote

My VPS have spare bandwidth, do you want me to try to see whether i can complete the download?

Feel free to test it

Testing it now.

PawGo

legendary

Activity: 952

Merit: 1367

Quote from: LoyceV on March 10, 2022, 03:53:37 PM

Quote from: PawGo on March 10, 2022, 03:34:47 PM

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

For me it works instantly.

Quote from: LoyceV on March 10, 2022, 03:53:37 PM

Quote

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

udp://tracker.opentrackr.org:1337/announce
udp://denis.stalker.h3q.com:6969/announce
udp://opentracker.i2p.rocks:6969/announce
udp://tracker.internetwarriors.net:1337/announce
udp://tracker.coppersurfer.tk:6969/announce

And indeed, only the first one seems to recognize the torrent.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: PawGo on March 10, 2022, 03:34:47 PM

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

Quote

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

And indeed, only the first one seems to recognize the torrent.

ABCbits

legendary

Activity: 2856

Merit: 7410

Crypto Swap Exchange

Quote from: LoyceV on March 08, 2022, 08:15:26 AM

Quote from: ABCbits on March 08, 2022, 07:07:04 AM

Really? It's weird since rTorrent (the backend) support DHT.

These are the only options it gives:
Image loading...

Since it include option "Private Torrent", usually it means DHT is used by default. And when i tried it on my VPS (where i remove all tracker before start the torrent), the DHT is working and i can start the download.

Quote from: LoyceV on March 08, 2022, 08:15:26 AM

Quote

3. Take note not all tracker accept any kind of torrent.

That's the tricky part: I couldn't find much about it, only a list of recommended trackers. That's what makes me think the slow download start might be related to the trackers.

Yeah, usually i just rely on DHT and share my IP:Port (if needed) so other user can it directly to list of peers on their torrent software.

Quote from: LoyceV on March 09, 2022, 04:26:47 AM

If I can't get the Torrents to work, the files can still be downloaded directly through http, but I haven't even finished uploading everything yet. And if barely anyone is going to use this, I might just have to give up on this project entirely at some point.

My VPS have spare bandwidth, do you want me to try to see whether i can complete the download?

PawGo

legendary

Activity: 952

Merit: 1367

Quote from: LoyceV on March 10, 2022, 02:35:52 PM

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

Quote from: LoyceV on January 07, 2021, 12:33:04 PM

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Torrents works well now, download is fast. Is it normal that only one tracker returns 'success' and peers data?