Torrent like client for the Bitcoin block chain

jl777

legendary

Activity: 1176

Merit: 1134

Quote from: TierNolan on November 27, 2015, 05:39:07 PM

The reference client uses "headers-first".

Each header is 80 bytes and there are 385k of them. This gives around 30MB of data for the headers.

The header chain has to be downloaded in series. But once you have downloaded the 30MB header chain from one peer, your node has the headers for all the blocks that you need to download.

You can ask each peer for a different block. This is parallel just like bittorrent. Each peer sends you blocks and you can verify that they match the header. If a node takes to long to send you a block, then you disconnect from that node and ask a different node for the block.

Once they chain has been downloaded, your node then just downloads each new block from whichever node(s) notifies it about the block.

iguana does parallel sync similar to the above, the hard part is processing all the data at network speeds. Currently I am completing the full blockchain download in about 12 minutes, but it takes about half hour to finish the processing. I create a set of read-only files out of bundles of blocks, which means they can all be put into a compressed filesystem, ie with squashfs. I estimate the total HDD needed to be around 20GB, though probably 30GB is needed during the peak usage of sync as there is a queue of unprocessed data.

I used to think blockchain size was an immediate issue, but with the parallel sync and a 1gbps connection, it isnt and wont be for quite a while. The processing of it is almost all totally in parallel, so even as the blockchain increases in size the time for a full sync is very close to linear.

The cost is that the time to get account balances will grow linearly over time also, but it is always possible to create another layer of data on top of the raw data. Currently I directly create the data structures that can be used for all the account queries needed.

James

koad

member

Activity: 64

Merit: 10

Quote from: samspaces on November 23, 2015, 03:01:09 PM

Hi all,

Downloading a full Bitcoin node is now at almost 50 Gigabytes. This is going to increase a lot of course with the broader adoption of Bitcoin and increasing number of transactions.

My question is, why isn't there a Bitcoin wallet interface yet, that accesses the block chain via a torrent like distribution. That would still be in the spirit of decentralisation and that way, you won't have to dedicate so much storage capacity. Also I'd think that via a torrent assembly the block chain would still be safeguarded against malpractice.

I'd like to read your thoughts on this,

Thanks, Sam

Edit: with 'torrent like' I mean decentralised local access to the entire block chain.

I really like your idea, implement your approach into an electrum thin client and it would become a hybrid/.

the blocks would be downloaded as per the users requirement to double check the results of the electrum server data & would also download blocks based on how many copies of each block exist in the p2p network of hybrid clients and where the network requires more coverage. You could effectively choose to limit the amount of blocks you carry and could (in a fashion) bootstrap your thin client with electrum (full node) and maintain full data access to the ENTIRE chain via DHT(p2p). Considering the confirmation times of snailcoin I dont think it is too much to ask for a few hundred extra milliseconds to query your data. Validating the chain will always be the big time spender; the only way around it is to trust (someone else) however I think this idea could add leverage to the p2p network and add some trust-ability into it at a thin-client level. I'm excited to see what the future has in store, things will change as their needs evolve.

/koad

samspaces

legendary

Activity: 1453

Merit: 1030

Quote from: ?? on ??

They said latest version of Core has been optimised and it will download faster blocks, in my experience when I installed Core the bottleneck was CPU, Dual Core 2.3GHz first blocks 2009/2010/2011/2012 was OK, when was confirming and validating other blocks it used 100% of CPU.

Running some older pc's as well, it takes forever to sync up a few 100 blocks.

calkob

hero member

Activity: 1106

Merit: 521

lets be honest the average user dosnt need to download the entire 50gb, as long as they can trust a certain number of nodes each independant of each other. now i don't mean just a few, i mean a few hundred at least.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

I guess another thing to point out is that 0.12 will ditch OpenSSL verification for libsecp256k, which removes a significant bottleneck from the tx verification part of sync'ing the chain (performance increase is something like 100%-400%). Nothing to do with the download rate, but it should still confer an overall speedup in sync'ing up.

cr1776

legendary

Activity: 4228

Merit: 1313

Quote from: samspaces on November 29, 2015, 10:54:01 AM

Quote from: enthus on November 27, 2015, 09:36:45 PM

The blockchain is very big....There must be a quicker alternative to downloading chain, I hope something is done soon, Torrent will be a great idea.

I think you know what I mean. An interface that perhaps stores headers locally, but makes a sync via torrent to the complete block chain, so it runs as a full node.

Since 0.10.0 there is no need to sync via torrent due to the headers first sync. If you read the thread above (among many other threads and discussions about this) and understand it, it wouldn't improve performance, just add something that isn't needed.

However, if you think it is a good idea for whatever reason, it is something that you can implement and then see if it improves anything or just adds complication.

samspaces

legendary

Activity: 1453

Merit: 1030

Quote from: enthus on November 27, 2015, 09:36:45 PM

The blockchain is very big....There must be a quicker alternative to downloading chain, I hope something is done soon, Torrent will be a great idea.

I think you know what I mean. An interface that perhaps stores headers locally, but makes a sync via torrent to the complete block chain, so it runs as a full node.

USB-S

sr. member

Activity: 574

Merit: 250

In XEM we trust

Quote from: enthus on November 27, 2015, 09:36:45 PM

The blockchain is very big....There must be a quicker alternative to downloading chain, I hope something is done soon, Torrent will be a great idea.

It was said before, bitcoin core wallet already uses torrent like system, where the blockchain is downloaded from other core clients via p2p. We know that the blockchain size is a issue. It won't be long until blockchains get to a point where it gets so big, that regular everyday users won't handle the capacity of it, especially when the block size increases.
There are wallets that only download block headers, which would be a way to go in the future.

enthus

full member

Activity: 140

Merit: 100

The blockchain is very big....There must be a quicker alternative to downloading chain, I hope something is done soon, Torrent will be a great idea.

TierNolan

legendary

Activity: 1232

Merit: 1094

The reference client uses "headers-first".

Each header is 80 bytes and there are 385k of them. This gives around 30MB of data for the headers.

The header chain has to be downloaded in series. But once you have downloaded the 30MB header chain from one peer, your node has the headers for all the blocks that you need to download.

You can ask each peer for a different block. This is parallel just like bittorrent. Each peer sends you blocks and you can verify that they match the header. If a node takes to long to send you a block, then you disconnect from that node and ask a different node for the block.

Once they chain has been downloaded, your node then just downloads each new block from whichever node(s) notifies it about the block.

shorena

copper member

Activity: 1498

Merit: 1528

No I dont escrow anymore.

Quote from: Decoded on November 27, 2015, 04:07:13 AM

I think bitcoin core has already utilized that function. It basically syncs with every other node using bitcoin core

I don't really think we should use a torrent-like system for everything, though. Maybe there should be two versions of bitcoin core, or two settings. The full version, just like hope bitcoin core is now, and the lite one, which downloads relevant snippets of the blockchain from a random 100 connected nodes using the full version. The downside to this version is lag when downloading the snippet. This will give an incentive for fully syncing, but if you really need to access a wallet fast, you can use the lite version.

Your "lite client" sounds like SPV to me -> https://multibit.org/en/help/hd0.1/how-spv-works.html

Decoded

legendary

Activity: 1232

Merit: 1030

give me your cryptos

I think bitcoin core has already utilized that function. It basically syncs with every other node using bitcoin core

I don't really think we should use a torrent-like system for everything, though. Maybe there should be two versions of bitcoin core, or two settings. The full version, just like hope bitcoin core is now, and the lite one, which downloads relevant snippets of the blockchain from a random 100 connected nodes using the full version. The downside to this version is lag when downloading the snippet. This will give an incentive for fully syncing, but if you really need to access a wallet fast, you can use the lite version.

RodeoX

legendary

Activity: 3066

Merit: 1147

The revolution will be monetized!

Quote from: Carlton Banks on November 23, 2015, 03:33:43 PM

Quote from: samspaces on November 23, 2015, 03:01:09 PM

Edit: with 'torrent like' I mean decentralised local access to the entire block chain.

You're confusing decentralised access to the blockchain with.... decentralised access to the blockchain.

Exactly what you want already exists: the Bitcoin client downloads different parts of the blockchain from multiple users simultaneously, the same principle as when downloading a file using the Bittorrent protocol.

That's your answer. As far as the huge size of the blockchain... Well, you can use a wallet that references a "trusted" copy of the blockchain. However you are introducing the risk of having to trust someone.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: samspaces on November 23, 2015, 03:01:09 PM

Edit: with 'torrent like' I mean decentralised local access to the entire block chain.

You're confusing decentralised access to the blockchain with.... decentralised access to the blockchain.

Exactly what you want already exists: the Bitcoin client downloads different parts of the blockchain from multiple users simultaneously, the same principle as when downloading a file using the Bittorrent protocol.

shorena

copper member

Activity: 1498

Merit: 1528

No I dont escrow anymore.

Quote from: samspaces on November 23, 2015, 03:01:09 PM

Hi all,

Downloading a full Bitcoin node is now at almost 50 Gigabytes. This is going to increase a lot of course with the broader adoption of Bitcoin and increasing number of transactions.

My question is, why isn't there a Bitcoin wallet interface yet, that accesses the block chain via a torrent like distribution.

Bitcoin core does sync with all connected nodes since 0.10. Its essentially "torrent like".

Quote from: samspaces on November 23, 2015, 03:01:09 PM

That would still be in the spirit of decentralisation and that way, you won't have to dedicate so much storage capacity. Also I'd think that via a torrent assembly the block chain would still be safeguarded against malpractice.

I'd like to read your thoughts on this,

Thanks, Sam

The data still needs to be verified and in order to avoid doing that every time (downloading 50GB, verify it) and still be a full node, the data needs to be stored locally.

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

I'm not sure what you mean by torrent like distribution (although I'm not particularly up to date on how torrents work), can you explain that?

There are wallets that use Simplified Payment Verification which download the block headers (significantly smaller than the entire blockchain) and the transactions that pertain to the addresses in the wallet. They only need a small amount of data and can retrieve the information they want from any full node using Bloom filters.

samspaces

legendary

Activity: 1453

Merit: 1030

Hi all,

Downloading a full Bitcoin node is now at almost 50 Gigabytes. This is going to increase a lot of course with the broader adoption of Bitcoin and increasing number of transactions.

My question is, why isn't there a Bitcoin wallet interface yet, that accesses the block chain via a torrent like distribution. That would still be in the spirit of decentralisation and that way, you won't have to dedicate so much storage capacity. Also I'd think that via a torrent assembly the block chain would still be safeguarded against malpractice.

I'd like to read your thoughts on this,

Thanks, Sam

Edit: with 'torrent like' I mean decentralised local access to the entire block chain.

Topic: Torrent like client for the Bitcoin block chain (Read 2786 times)