Topic: [new feature discussion] -fastboot (Read 2374 times)

I very like the idea of using Bittorrent, but I'm not sure about few things:

1. we're adding big dependency on other program (bittorrent client)
2. I don't know any good and portable  C / C++  lib for bittorrent (any ideas?)
3. it's still centralised (bittorrent clients need to connect to tracker), but of course it's much better in terms of download speed and load balancing.
4. Bitcoin clients are already using some protocol to download blocks - perhaps we can resuse it instead of adding another protocol?

All in all: idea looks good, but I wasn't not sure how to deal with those problem, so I just implemented version with mirrors. If we find a solution, I'll try to improve fastboot ;-)

The general idea is very very good, but the current "snapshots" of the network should be put on torrent sites.

Torrents offer many advantages over normal ways of distribution:

- Automatic checking of authenticity (torrents have checksums integrated)
- Greatest speed of download
- Decentralization: no central servers needed (same as with Bitcoin)

If there is an evil maintainer and I am running his code, there are many more nefarious things he could be doing with my computer that will be harder for me to notice than giving me an invalid block database that I will probably quickly notice (or hear about in the forums).  It's a risk I'm probably willing to take so I don't have to wait a day to start using the software.

sure, but then - even if evil maintainer prepared faked blockchain - client shouldn't download more blocks from p2p. correct?

Of course it is. That's not what I meant. What you are doing is useful, but it does not have the same security guarantees. Right now, through hardcoded block hashes in the client, the maintainer has the ability to force a particular chain onto you, but that chain must still exist and be valid. If the hash of the entire block file is hardcoded in the client, the maintainer can effectively make you get an invalid block database.

beta version for linux:

https://github.com/kkurach/bitcoin/tree/fastboot

any comments?  (files:  src/fastboot.h  src/fastboot.cpp)

Hey casascius, I think that's a great idea. Perhaps (for the first version) I can just hardcode signatures of some people (responsible for github/bitcoin), and then - in client - just verify signatures of downloaded files. This will allow to update file with blocks every day (and because of that function should provide even faster boot), without changing client source code.

What do you guys think about hardcoding signatures instead of md5sum of first N blocks?

I have thought of this... in my ideal world, this is how it would be implemented.

The client has a few public keys which it trusts (developer maintained), or has a screen where trusted keys can be manipulated by the user (developer maintains the defaults but user is ultimately in charge).

At some point, the blocks themselves can contain checkpoint transactions that vouch for everything up through and including the blocks that contain them.  They don't have to be anything special, they could be nothing more than standard Bitcoin transactions involving a signature from the hardcoded key.

When the client sees such a checkpoint, it can assume that everything before it is safe.  With the checkpoints in the block chain, there is no need to keep hard-coding checkpoints in the software (or, at least, the number of hard-coded checkpoints can be vastly reduced for the same effect.)

Finally....(separate feature).... the software ought to support a sort of "flat file import" that contains a recent copy of the blockchain in "blob" format, signed the same way.  If a well-formed and validly-signed "distribution blob" is available, the client should take in everything from the blob and not ask for a single block from a peer until the entire blob has been "downloaded" (read: copied into the preferred format of the client's choice, and indexed the way the client likes it...as the "blob" shall have no indexes).  Given that, the blob should be optimized for space as much as possible (e.g. deduplication of public keys, not wasting bytes wherever possible, etc.)

Blobs, when obtained from a trusted party, could also give rise to pruned blockchains to save space.  When accepting a pruned chain, one must trust that the chain doesn't omit any transactions that would lead the client to believe that spent coins are actually unspent.  A trusted party (developer whose public key is harcoded) could be the person to sign a trusted pruned blockchain, and a third party open source utility could allow the paranoid among us to compare the signed pruned version with the full version to confirm for everyone else that it is indeed not misrepresenting the unspent status of any coins.

Hey ArtForz, I think you're right. Anyway, I think it's safer, cleaner and easier to implement this with hardcoded checksums. First, we are discovering very early that downloaded blocks are invalid (because someone changed them or there were problems with download). It allows to retry download from a different mirror as soon as md5hash is calculated.

Secondly, I'm not sure how exactly works the logic inside client (handling broken chain), and I'd prefer to not touch it, just to be on a safe side.  With checksum approach I just write a class with one method, which downloads and verifies files. This method can be run from any point of client, and the rest of the client is untouched.

Last but not least, we get another (perhaps redundant) protection with this md5sum.

But here we are not validating blockchain block-by-block (which takes a lot of CPU). I just verify md5um of a file which contains 140000 blocks.  It's  much much faster.


it can take argument, but I think it should also have some default mirrors hardcoded (user usually just download and run a program, and don't care about reading docs and looking for URLs, etc...). I already have 3 people who volunteered to provide mirrors, which may be enough for beta version.

Thinking out loud...
Do we even need to check the ECDSA signatures in blocks older than the last checkpoint?
Assuming the checkpoint hash is okay, you can't change any transactions in blocks before it without breaking the chain.
Assuming the checkpoint hash was replaced (eve hacked the devs box or something)... we're pretty much fucked anyways as the client will reject the legit chain and stay on the attackers chain.

Any input?

-rescan does not revalidate the block chain. It only synchronizes the block chain with the wallet, adding missing transactions to it.

And indeed, most of the time is spent validating the block chain, not downloading. Before 0.3.24, larger blocks often caused disconnects when sending because of an anti-spam measure. That is/was also a cause of delay when starting up, but is slowly improving, as more nodes (and thus potential download partners) upgrade to 0.3.24.

That said, a -fastboot argument that takes a URL or filename to import a blockchain from may be useful in certain situations.

ok, I did the homework.

I used Matt's blockchain (thanks Matt for sharing this!!) on fresh client.
(http://bitcoin.bluematt.me/bitcoin-nightly/blockchain-nightly/)

download time: 7 minutes
bunzip2 and tar -  ~ 2 minutes
./bitcoin -rescan :  1 minute


total: 10 minutes
result:  140600 blocks done

@zellfaze   hm...but don't we need to run  -rescan? If I download 140k blocks,  what about my transactions which were included in those blocks? Will they show up or I need to rescan?   
Anyway, I believe that still download + rescan is faster than p2p.

I think that if you just download the block chain and stick it in the bitcoin directory then the Bitcoin client doesn't check if they are valid.  It just assumes they are.

Isn't that what the -rescan option is for?  Re-validating the block-chain?  Or am I mistaken?

Is it true also for newer blocks? I know that it's pretty fast to download first  60-70k,  but later it's getting slower and slower....


BTW. perhaps we can store already processed blocks? (as binary blob ),  of course with md5 verification?  then we get everything as fast as download

@sirk390

1. you do can -  git push  of course I was talking only about updating md5 string in the code. (so, older clients will still work but have to download more blocks from p2p than newer clients [equal to the number of blocks between releases])

2. You don't need to as the blockchain is allready "protected" against block changes.

It's just another protection. Calculating hash is not a big deal, and I'm not sure how exactly blockchain protection works when attacker can change almost all blocks.

I think you're solving the wrong problem. Most of the time is not spend downloading the block chain but processing it. Take a look at your CPU usage.

Excellent idea! This will be much needed for bitcoin to become useful for the average person.
That being said, I think you should add a -slowboot option and have the fast download as the default behavior.

Maybe you could combine the idea from RaTTuS with the download: have one thread load the block backward from the p2p network, while another thread downloads the cache files from the central servers. Now let those run until they meet each other. That could even solve the validation of the centrally stored mirror files, because if those are bogus the two chains would never match.

It would be nice but I'm not sure you know what you are talking about.
1. You can't update the code every few weeks(How ?)
2. You don't need to as the blockchain is allready "protected" against block changes.