Any thought to reduce downloading time of blockchain ?

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: achow101 on October 17, 2019, 03:29:16 PM

Checkpoints are still there. They are effectively consensus required. So if someone wanted to make a fake bootstrap.dat, they would have to fork from the most recent checkpoint (block 200000 something) and still do all of the work to mine valid blocks.

hmmm, this is correct. Longest most work chain would still be the genuine chain subsequent to the last checkpoint. My bad, sorry everybody

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: Carlton Banks on October 17, 2019, 01:46:22 PM

right, but not everyone is necessarily going to do a search to find out what the origin of the torrent link is, or to find out whose PGP created the signature.

The thread was linked in the message...

Quote from: Carlton Banks on October 17, 2019, 01:46:22 PM

and the fact that it's hosted on bitcoin.org is a little meaningless on it's own, the admin/owner of that site started to behave a little strangely in public over the last few years, you should know that Bitcoin releases are released primarily through https://bitcoincore.org, as you know.

Sure, but it's not some random website or link. Bitcoin.org is still somewhat trustworthy. And tbf, Bitcoin Core releases are still posted to bitcoin.org in addition to bitcoincore.org.

Quote from: Carlton Banks on October 17, 2019, 01:46:22 PM

and was the checkpoint code not substituted for the assumevalid model of chain authentication? i.e. is it not possible for someone to download a genuine Bitcoin client from http://bitcoincore.org, be given a bootstrap dat for a fake chain that never reaches the assumevalid blockheight, then get conned into e.g. buying BTC and receiving outputs that would be invalid on the genuine chain?

Checkpoints are still there. They are effectively consensus required. So if someone wanted to make a fake bootstrap.dat, they would have to fork from the most recent checkpoint (block 200000 something) and still do all of the work to mine valid blocks.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: achow101 on October 17, 2019, 01:08:48 PM

Quote from: Carlton Banks on October 06, 2019, 05:19:33 PM

Quote from: DaCryptoRaccoon on October 06, 2019, 04:15:16 PM

https://bitcoin.org/bin/block-chain/

You can bootstrap download the chain from here just make sure to check the signatures it will help you.

It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection.

this is a bad idea, do NOT listen to the above advice under any circumstances.

You risk being connected to a fake blockchain if you download random torrents without knowing what you're doing

@MagicByte your post is irresponsible

It actually isn't. It's just the original bootstrap.dat file (that is no longer being updated) that people previously used. Notice how it is hosted on bitcoin.org.

The original thread announcing it is at https://bitcointalksearch.org/topic/ann-bitcoin-blockchain-data-torrent-145386. That message is signed with Jeff Garzik's old PGP key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3710408162759fc5a4296536e7a58e337adca079

right, but not everyone is necessarily going to do a search to find out what the origin of the torrent link is, or to find out whose PGP created the signature.

and the fact that it's hosted on bitcoin.org is a little meaningless on it's own, the admin/owner of that site started to behave a little strangely in public over the last few years, you should know that Bitcoin releases are released primarily through https://bitcoincore.org, as you know.

and was the checkpoint code not substituted for the assumevalid model of chain authentication? i.e. is it not possible for someone to download a genuine Bitcoin client from http://bitcoincore.org, be given a bootstrap dat for a fake chain that never reaches the assumevalid blockheight, then get conned into e.g. buying BTC and receiving outputs that would be invalid on the genuine chain?

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Quote from: Carlton Banks on October 06, 2019, 05:19:33 PM

Quote from: DaCryptoRaccoon on October 06, 2019, 04:15:16 PM

https://bitcoin.org/bin/block-chain/

You can bootstrap download the chain from here just make sure to check the signatures it will help you.

It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection.

this is a bad idea, do NOT listen to the above advice under any circumstances.

You risk being connected to a fake blockchain if you download random torrents without knowing what you're doing

@MagicByte your post is irresponsible

It actually isn't. It's just the original bootstrap.dat file (that is no longer being updated) that people previously used. Notice how it is hosted on bitcoin.org.

The original thread announcing it is at https://bitcointalksearch.org/topic/ann-bitcoin-blockchain-data-torrent-145386. That message is signed with Jeff Garzik's old PGP key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3710408162759fc5a4296536e7a58e337adca079

Quote from: DaCryptoRaccoon on October 17, 2019, 05:59:03 AM

"It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection."

and how is it any more irresponsible to be downloading and running pull request code that is "such a risky move" and "perhaps lower risk"..

If you validate the downloads there is no issue.

TBH I don't know why they don't offer a more up-to-date way to download this and then validate there are a lot of people who still struggle and bootstrapping the data helps.

Because it isn't any faster to use the bootstrap.dat file. It has not been faster to use it since 0.10.0 when headers first sync and out of order download were first introduced. The bootstrap.dat file was useful prior to that, but it no longer is.

The reason it is slower to use the bootstrap.dat file is because it just contains the blocks. Your node still has to read them off disk and validate them. It also happens to be one big file and the node does not know where each block is in that file, so it can only go through it one at a time. This is much slower than how syncing works now where multiple blocks are downloaded in parallel, and because the node is the one writing them to disk, it also knows exactly where in each file the blocks are stored so they can be pulled up later for the final step of validation. This parallelization also allows validating things that are independent of each other to be done in parallel which speeds up the sync.

It is important to note that the primary slow down in syncing is due to the processing, storage (disk I/O), and validation of blocks, not the network speed.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

@magic byte

you're replying to the wrong post

(I predict more confusing replies from Magic Byte to follow Wink

)

DaCryptoRaccoon

hero member

Activity: 1220

Merit: 612

OGRaccoon

Quote from: Carlton Banks on October 01, 2019, 07:19:03 AM

If you're feeling incredibly brave, you can test out the pull requests from this user on github which all together can apparently cut the IBD time by ~14%.

But 14% is not big enough to justify such a risky move, these are testing changes, and as such are not for real-world use in their current state. Although this particular change has been pulled into 0.19.0 release candidate, and is perhaps lower risk. I would still advise against it if you're expecting stability, there is a big risk that the full consequences of these changes have not yet been observed, as they've only seen limited testing.

Did your even read what I posted..

"It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection."

and how is it any more irresponsible to be downloading and running pull request code that is "such a risky move" and "perhaps lower risk"..

If you validate the downloads there is no issue.

TBH I don't know why they don't offer a more up-to-date way to download this and then validate there are a lot of people who still struggle and bootstrapping the data helps.

Wind_FURY

legendary

Activity: 2898

Merit: 1823

Quote from: allwelder on October 15, 2019, 02:34:27 AM

The moderator delete my post,post again,DO NOT DELETE AGAIN,THIS IS NOT SPAM.

There are already coins(Tera) that can reduce the time.
It download blockchain data from end rather than beginning,so when user start client,no need to wait for downloading whole blockchain data,just use it,and let the downloading run at the same time.

In context of this topic. You still can't use it, it's not a full node, because your client hasn't completely validated everything.

The mod deleted your post because he probably believes you don't understand.

Quote

Even though this did not reduce the whole time needed for downloading,but reduce waiting time.

You're still required to wait.

Quote

This is also can be use to many coins,including bitcoin.

No. I can use Electrum while my node is syncing.

OP, to help reduce the initial blockchain download as time passes by? Smaller blocks.

allwelder

legendary

Activity: 1512

Merit: 1004

The moderator delete my post,post again,DO NOT DELETE AGAIN,THIS IS NOT SPAM.

There are already coins(Tera) that can reduce the time.
It download blockchain data from end rather than beginning,so when user start client,no need to wait for downloading whole blockchain data,just use it,and let the downloading run at the same time.

Even though this did not reduce the whole time needed for downloading,but reduce waiting time.

This is also can be use to many coins,including bitcoin.

naska21

hero member

Activity: 1358

Merit: 635

Quote from: Rath_ on September 25, 2019, 03:10:33 AM

....

Also, upgrading to an SSD makes a difference in the sync time.

Yeah, in terms of synch the major time consumer is disk's I/O operations and in this respect NVMe-compatible SSD would perform even better than say SATA SSD.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: Macadonian on October 06, 2019, 05:40:24 PM

Downloading from a torrent although quicker

it's not quicker, since about 4-5 years ago. There's literally no point in using bittorrent to download the blockchain since version 0.10 (which was released in 2015)

Macadonian

sr. member

Activity: 467

Merit: 578

You could download the Blockchain from a torrent however it is extremely advised not too because of the risk that you run of it being infected or altered. Even if it comes from Bitcoin.org I would still advise downloading it natively through the network because that version coming directly from the network is checked by nodes to assure that everything is synchronized and nothing has been altered. Downloading from a torrent although quicker is risky and I would advise members of the forum not to advise others of downloading from there even if there is severe limitations on internet and disk space.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: DaCryptoRaccoon on October 06, 2019, 04:15:16 PM

https://bitcoin.org/bin/block-chain/

You can bootstrap download the chain from here just make sure to check the signatures it will help you.

It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection.

this is a bad idea, do NOT listen to the above advice under any circumstances.

You risk being connected to a fake blockchain if you download random torrents without knowing what you're doing

@MagicByte your post is irresponsible

I was wrong about this, apologies to MagicByt3

it's still pointless to use this old torrent bootstrap, it won't speed up IBD. And it only includes the first 2-3 years of Bitcoin's blockchain, so it adds steps for no benefit.

DaCryptoRaccoon

hero member

Activity: 1220

Merit: 612

OGRaccoon

https://bitcoin.org/bin/block-chain/

You can bootstrap download the chain from here just make sure to check the signatures it will help you.

It is advisable to download from the network but this is a way for those who struggle with downloading or have poor connection.

Code:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Torrent info hash: 36719ba2cecf9f3bd7c5abfb7a88e939611b536c
Torrent magnet link:
magnet:?xt=urn:btih:36719ba2cecf9f3bd7c5abfb7a88e939611b536c&dn=bootstrap.dat&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=http%3A%2F%2Fbttracker.crunchbanglinux.org%3A6969%2Fannounce

Block height: 317000
Filename: bootstrap.dat
Byte size: 22566124235
SHA1: 2e6e5dd20cd9c21960ba94f7265f42c4b6443758
SHA256: d05062052be4dd6fbaeac5f17598e52f7ad545f9f938acefaf7067ecce1d7b5a

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: GPGTools - http://gpgtools.org

iQEcBAEBCgAGBQJT+PcrAAoJEL8a4JGi25zKmj4H/i8xKIXW/xnw5i5Pc+ciUuc2
Zo+KxQd1YGRmJSvGsyTgTqK4IJUBD8s8tIGTtznCJ2/Q3wIdZmoqxkST7r4rqJly
cOyJ/n19K5beYvVekIgHVXFmhdJhZf8KNnyuV+DpHmmLKYyZtvkoY9b3hEBHVtSM
AkjuWyA9IEPeghxBX32JAToXOZGs9vZghNyIMg/fWbGLuv1oX7QCjAAua0NAMbLX
diSvVcLXskTRy38W4whz1DX6ruL8mPRYH5Ko6/cmtVqEkr53aEwYf1XJGHHAz3+J
nEHecxBUbiRhDwMdCVtqzKudXWVDtkgVQmRxrkKTw63NTVMge4eEziE2+7JQgq8=
=p9TY
-----END PGP SIGNATURE-----

Code:

As of Bitcoin Core version 0.10.0 and later, the block chain bootstrap
torrent hosted here takes more time to download and import than it would
to simply start Bitcoin Core and let it sync itself.

Forum thread on BitcoinTalk:
https://bitcointalk.org/index.php?topic=145386.0

Bitcoin-Qt/bitcoind version 0.7.1 and later supports a special import feature: If the file "bootstrap.dat" is found in the bitcoin data directory, it will validate and import all blockchain data found in that file. The following torrent presents a bootstrap.dat file for that feature.

----------------------
What is bootstrap.dat?

It is a flat, binary file containing bitcoin blockchain data, from the genesis block through a recent height.

Versions 0.7.1+ automatically validates and imports a file in the data directory named "bootstrap.dat".

Special note: Version prior to 0.8.0 have a bug which will only import 2G of data from a file. This is fixed in 0.8.0.

------------------------
Who wants bootstrap.dat?

Anyone bringing up a new node using the reference client. This is one method of accelerating the initial blockchain download process, while helping the bitcoin P2P network by offloading data download traffic from public P2P nodes.

This download is not for those who are already running the bitcoin client.

---------------------------------------
How often will this torrent be updated?

Assuming this project is deemed useful and worth continuing... the torrent will be updated once every few months, when the checkpoints are updated in the reference client source code.

--------------------------------------
Why not update the torrent more often?

A torrent works best when it is a large, static dataset that changes infrequently. That maximizes the ability to seed the data, enabling even part-timer seeders to contribute meaningfully. Less frequent changes also minimizes the risk that a malicious torrent will appear, with a long, malicious side chain. The current policy only updates the torrent after blocks are buried many thousands deep in the chain.

-----------------------
Why should I trust you?

You don't have to: This data is raw block chain data. The client will verify this data during import.

Independent third parties may generate their own bootstrap.dat, up to a recent height, and verify that the sha256sum matches that posted above. The file format is simple and publicly known:

<4-byte pchMessageStart><32-bit length>

----------------------------------------
How can I help? Do you need more seeds?

Yes, we need as many long term seeds as possible. This ensures we can meet torrent download demand immediately at high speeds, and remain idle the remainder of the time.

Macadonian

sr. member

Activity: 467

Merit: 578

Quote from: Jet Cash on September 30, 2019, 02:38:37 PM

Get off Windows 10 if you are using it. I've been able to sync' my blockchain without any problems for a few years, and that is using public WiFi. Following the latest major upgrade of Windows 10, i'm getting dumped by quite a few peers, and it is becoming difficult to maintain the blockchain. I need to see if Mint supports the Brave browser, and go back to Linux.

This is probably due to all the background processes within Windows 10 but for this to have any effect your internet would have to be incredibly slow in the first place. Windows 10 should not be dropping anymore peers than Windows 7 would in theory. The only thing I can think of is the way Windows 10's firewall is implemented and its blocking certain peers from connecting via the Bitcoin core program.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

If you're feeling incredibly brave, you can test out the pull requests from this user on github which all together can apparently cut the IBD time by ~14%.

But 14% is not big enough to justify such a risky move, these are testing changes, and as such are not for real-world use in their current state. Although this particular change has been pulled into 0.19.0 release candidate, and is perhaps lower risk. I would still advise against it if you're expecting stability, there is a big risk that the full consequences of these changes have not yet been observed, as they've only seen limited testing.

Jet Cash

legendary

Activity: 2814

Merit: 2472

https://JetCash.com

Get off Windows 10 if you are using it. I've been able to sync' my blockchain without any problems for a few years, and that is using public WiFi. Following the latest major upgrade of Windows 10, i'm getting dumped by quite a few peers, and it is becoming difficult to maintain the blockchain. I need to see if Mint supports the Brave browser, and go back to Linux.

franky1

legendary

Activity: 4410

Merit: 4766

Quote from: CounterEntropy on September 27, 2019, 06:01:40 PM

Quote from: HeRetiK on September 26, 2019, 04:16:50 AM

If downloading the Bitcoin blockchain is too much of a hassle to you, please consider using Electrum instead as suggested by ETFbitcoin and mocacinno:
https://electrum.org/

I guess, it can be done with Bitcoin Core itself through pruning. To enable block pruning set prune=N on the command line or in bitcoin.conf, where N is the number of MiB to allot for raw block and undo data.

Details: https://bitcoin.org/en/full-node#reduce-storage

pruning does not help the user as a full network helper
first you still need to download the entire blockchain first and then the node sets itself to only keep the latest few days/weeks of blocks and deletes the rest.
thus making the whole point of even bothering to download the blockchain useless waste of time. might aswell just be a litenode and have a utxo snapshot.
also pruned nodes dont have the blockchain so they wont then be seeders to help others. thus just being a selfish leacher and sitting on the relay network while not helping the network fully.
its one of th stupid things i thought core devs added as a feature of their FULL NODE implementation because they are not full nodes if they cannot be a seed for others to download from.
so yea if you dont wanna be a seed and dont want peers grabbing data from you. just dont bother trying to be a full node and use a lite node instead

CounterEntropy

full member

Activity: 214

Merit: 278

Quote from: HeRetiK on September 26, 2019, 04:16:50 AM

If downloading the Bitcoin blockchain is too much of a hassle to you, please consider using Electrum instead as suggested by ETFbitcoin and mocacinno:
https://electrum.org/

I guess, it can be done with Bitcoin Core itself through pruning. To enable block pruning set prune=N on the command line or in bitcoin.conf, where N is the number of MiB to allot for raw block and undo data.

Details: https://bitcoin.org/en/full-node#reduce-storage

1Referee

legendary

Activity: 2170

Merit: 1427

Quote from: franky1 on September 27, 2019, 03:49:43 AM

separately
.. the actual issue people have is not that it takes them 4 weeks to download. but that their node is stuck with a 'downloading blockchain' message and no other function until the blockchain is uptodate.

This. I have helped a relative last year after complaining about the process of 'downloading' blocks taking too long, and that with a 100mbit cable connection. Turns out, the validation process is what took him so long, and this is directly where most new people go wrong. They assume that downloading the blockchain is similar to downloading a movie file or something.

People have no clue about how heavy it is on the system to validate every bit of data that goes back like 10 years. In Bitcoin's history there have been almost 460 million transactions and 1.23 billion outputs. If people start to understand how much computing goes into validating that data, and how regular hdd's further slow down that process, they will probably refrain from running their own node and just go with an SPV client (which for most ordinary users is the best option).

pereira4

legendary

Activity: 1610

Merit: 1183

A nice trick to speed up your synchronization time is to get the best possible peers. This website uses an index with an algorithm (open source) and they generate this rank:

https://bitnodes.earn.com/nodes/leaderboard/

You can add nodes either on the debug console with addnode add or in the peers.dat file, by simplying copy pasting the list of nodes there. This really makes a change so try it out.

Topic: Any thought to reduce downloading time of blockchain ? (Read 540 times)