Pages:
Author

Topic: [BETA] Bitcoin blockchain torrent - page 5. (Read 57756 times)

donator
Activity: 1218
Merit: 1079
Gerald Davis
October 12, 2012, 12:42:25 PM
#25
Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).

Why would a byte for byte copy be a "slightly different length"?

How does it know the length of the torrented file? (Note that it is "identical", not a copy) Though from what jgarzick says in the post above there is some kind of checkpointing that it either knows or gets fed into it?

It doesn't need to know the length of the file.  The script doesn't make a dat file of your entire blockchain just through the last checkpoint.  Currently the last checkpoint is block 193,000.   If you run that script on any node it will produce the same file.   Exactly the same file.

Now currently the script has 193,000 hardcoded but I could see future version either getting the checkpoint from the client or making an API call to get the block #.  The script could be included with the client or even better it could be built into the client so you click an [export blockchain > torrent] button and it generates the proper file based on the current checkpoint.

Lots of interesting options.
legendary
Activity: 2576
Merit: 2267
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
October 12, 2012, 12:35:44 PM
#24
Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).

Why would a byte for byte copy be a "slightly different length"?

How does it know the length of the torrented file? (Note that it is "identical", not a copy) Though from what jgarzick says in the post above there is some kind of checkpointing that it either knows or gets fed into it?
legendary
Activity: 1596
Merit: 1100
October 12, 2012, 12:31:09 PM
#23
Quote
Each time the blockchain torrent is updated, seeders may run that script to guarantee they have 100% of bootstrap.dat immediately.  A nice, decentralized solution Smiley

Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).

Once the height of the next checkpoint is publicly known, everyone may run that script to independently generate bootstrap.dat with the exact same file size and SHA256 checksum.

sr. member
Activity: 476
Merit: 250
Tangible Cryptography LLC
October 12, 2012, 12:21:42 PM
#22
Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).

Why would a byte for byte copy be a "slightly different length"?
legendary
Activity: 2576
Merit: 2267
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
October 12, 2012, 12:16:31 PM
#21
It will result in seeders needing to re-download a torrent for each update though.

It will result in seeders already having 90% of any new torrent, thus having to fill in the remaining 10%.

Yes, it's just that it's an extra step, not automatic. This is a limitation of the torrent protocol and not a criticism of this project, just pointing it out.


Quote
I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.

Yes, this is open data and an open file format.  Seeders may independently generate a byte-for-byte identical bootstrap.dat by running https://github.com/jgarzik/pynode/blob/master/mkbootstrap.py

Each time the blockchain torrent is updated, seeders may run that script to guarantee they have 100% of bootstrap.dat immediately.  A nice, decentralized solution Smiley


Excellent. Though presumably it will be a slightly different length than whatever is torrented (which will be taken care of by the recheck-data option).
legendary
Activity: 1596
Merit: 1100
October 12, 2012, 12:12:31 PM
#20
It will result in seeders needing to re-download a torrent for each update though.

It will result in seeders already having 90% of any new torrent, thus having to fill in the remaining 10%.

Quote
I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.

Yes, this is open data and an open file format.  Seeders may independently generate a byte-for-byte identical bootstrap.dat by running https://github.com/jgarzik/pynode/blob/master/mkbootstrap.py

Each time the blockchain torrent is updated, seeders may run that script to guarantee they have 100% of bootstrap.dat immediately.  A nice, decentralized solution Smiley

legendary
Activity: 2576
Merit: 2267
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
October 12, 2012, 12:06:16 PM
#19
Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

Hmm.  I didn't realize that was possible.  I assumed that changing the torrent would result in no seeds (and all prior seeders needing to download again).  I was thinking many wouldn't and thus it would be a challenge to keep the number of seeders high.  I guess bittorrent is "smarter" than I realized.  

It will result in seeders needing to re-download a torrent for each update though. They'll also have to download all the intervening data (though they will be seeding from what they already have right away). I wonder if it could be generated from their existing blockchain (is it basically the same file?) which will presumably be pretty up-to-date.
legendary
Activity: 1596
Merit: 1100
October 12, 2012, 12:01:57 PM
#18
Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

Hmm.  I didn't realize that was possible.  I assumed that changing the torrent would result in no seeds (and all prior seeders needing to download again).  I was thinking many wouldn't and thus it would be a challenge to keep the number of seeders high.  I guess bittorrent is "smarter" than I realized.  

This is special to our use case:  bootstrap.dat is essentially an append-only file.  Blocks are simply concatenated onto the end.

Today's torrent at height 193000 is 2,491,771,562 bytes in size.

The next torrent, a few months from now, will have the same first 2,491,771,562 bytes.

Thus, to bittorrent, the next torrent will simply appear to be a truncated / not fully downloaded bootstrap.dat.  Bittorrent is built to fill in the missing pieces of a file, so that is what it does here Smiley

legendary
Activity: 2576
Merit: 2267
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
October 12, 2012, 11:59:43 AM
#17

...
Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

D'oh. This is "stuff I know" and have used from time to time. My bad.
legendary
Activity: 2576
Merit: 2267
1RichyTrEwPYjZSeAYxeiFBNnKC9UjC5k
October 12, 2012, 11:33:19 AM
#16
At last, finally somebody made bootstrapping Bitcoin easier.

It was a major pain in the ass for people running full BTC client.

It has been possible to download the blockchain. Torrent is in some ways an improvement in that it improves decentralization but in some ways a step back as it's not really suited to frequent updates. Say, for example, I download and seed v1, eventually v2 will come out. Although I already have all of the data in v2 in my running blockchain, there's no easy way to contribute to seeding v2 without downloading the whole thing again. A way to extract a version of the torrented blockchain from my live blockchain would be good. What might be better would be a custom client that could seed from the live blockchain itself.
donator
Activity: 1218
Merit: 1079
Gerald Davis
October 12, 2012, 11:16:05 AM
#15
Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

Hmm.  I didn't realize that was possible.  I assumed that changing the torrent would result in no seeds (and all prior seeders needing to download again).  I was thinking many wouldn't and thus it would be a challenge to keep the number of seeders high.  I guess bittorrent is "smarter" than I realized.  
legendary
Activity: 1596
Merit: 1100
October 12, 2012, 11:13:01 AM
#14
Maybe a silly question but would it be possible to separate the torrent into something like daily or weekly segments so the individual parts never need to be updated?

At present, that sort of setup would be challenging for users to use effectively.  It is far easier to have one file to import (automatically, in 0.7.1+) than a collection of files.

Longer term, the idea is to fix any issues with bitcoin P2P peer selection, so that downloading recent blocks from your peers is faster and not burdensome.

It could be as simple as making a new torrent for each checkpoint. Say in 6 months we are at block 240,000 and there is a checkpoint 220,000 a "torrent2" could be added which contains blocks 193,001 to 220,000.   New users could download both torrents and be within 20,000 blocks of "current".  I think at most quarterly updates is all that is necessary.

That's the plan, with a slight change:  each new torrent will contain all blocks from zero to X.

For bitcoin users, the largest user population served by this is those that have zero blocks, and are jumpstarting a fresh node installation.  Other bitcoin users will likely have blocks fresher than 3-6 months old; catching up via P2P network is fine for them.  There is only a tiny remaining segment of bitcoin users who would then be served by new-torrent-for-checkpoint, those that only turn on their bitcoin clients once every ~6 months.

Note:  For existing torrent seeders, they may simply swap out the .torrent file, perhaps kick their torrent client to manually re-verify a file, and bootstrap.dat in their Uploads directory will simply be extended.  Seeders will automatically already have 90% of each new torrent's bootstrap.dat.

donator
Activity: 1218
Merit: 1079
Gerald Davis
October 12, 2012, 11:02:05 AM
#13
Maybe a silly question but would it be possible to separate the torrent into something like daily or weekly segments so the individual parts never need to be updated?

Great work, looking forward to testing this client over the weekend.

That probably is a good idea but I don't think updates that often are necessary.  Think of the torrent as just a jumpstarter.  It doesn't need to get you within days of the current block just rapidly build the "ancient history".  Even having a single base torrent to block 193,000 allows new users to bootstrap ~90% of the blockchain.  

It could be as simple as making a new torrent for each checkpoint. Say in 6 months we are at block 240,000 and there is a checkpoint 220,000 a "torrent2" could be added which contains blocks 193,001 to 220,000.   New users could download both torrents and be within 20,000 blocks of "current".  I think at most quarterly updates is all that is necessary.

On edit: looks like it will take me ~30 min to download the torrent.  If a user can download 80% to 90% of the blockchain in half an hour and then index it in another hour that is a pretty nice jumpstart.
legendary
Activity: 1400
Merit: 1013
October 12, 2012, 10:56:38 AM
#12
but it does look like users occasionally get partitioned off into "Azureus island" or "rtorrent island" or "everybody else's island."
I wonder why the Azureus island is so lonely right now. All I can see are two seeds and no peers.
legendary
Activity: 1596
Merit: 1100
October 12, 2012, 10:34:27 AM
#11
Won't this torrent only benefit new users if people actually seed it?

The swarm doesn't appear to be particularly large at the moment...

Correct.  It is not much use, without participating seeders.  We definitely have several right now... but it does look like users occasionally get partitioned off into "Azureus island" or "rtorrent island" or "everybody else's island."

legendary
Activity: 1596
Merit: 1100
October 12, 2012, 10:33:24 AM
#10
Nice one.  I will dedicated some bandwidth for it.  I could imagine with a couple hundred seeds new users could bootstrap very quickly.  Even if you only updated the torrent once a year it would still provide a significant portion of the blockchain at a high speed.

Yes, that's the hope.  Already had reports of 7 MB/s downloads...

legendary
Activity: 1400
Merit: 1013
October 12, 2012, 10:18:33 AM
#9
Won't this torrent only benefit new users if people actually seed it?

The swarm doesn't appear to be particularly large at the moment...
donator
Activity: 1218
Merit: 1079
Gerald Davis
October 12, 2012, 10:15:27 AM
#8
Nice one.  I will dedicated some bandwidth for it.  I could imagine with a couple hundred seeds new users could bootstrap very quickly.  Even if you only updated the torrent once a year it would still provide a significant portion of the blockchain at a high speed.
legendary
Activity: 1050
Merit: 1002
October 12, 2012, 10:08:31 AM
#7
Very nice! This project seems to continue to have the best technology associated with it.
legendary
Activity: 1470
Merit: 1006
Bringing Legendary Har® to you since 1952
October 12, 2012, 03:09:34 AM
#6
At last, finally somebody made bootstrapping Bitcoin easier.

It was a major pain in the ass for people running full BTC client.
Pages:
Jump to: