Speed up syncing of bitcoin core when the blockchain becomes colossal

jackg

copper member

Activity: 2856

Merit: 3071

https://bit.ly/387FXHi lightning theory

Quote from: Subspace_Labs on March 13, 2019, 11:16:24 AM

Quote from: Jet Cash on January 29, 2019, 10:23:58 AM

I don't think the hardware is the problem. Given that my netbook is a low spec "celery", and it seems to be quite happy sync'ing, I suspect the problem lies with network connectivity. Give things another few years, and babies will be fitted with blockchain implants shortly after birth.

This is not necessarily true. It's possible to have a hardware configuration that is the bottleneck. This is especially true when running Raspberry Pi or similar. Also, slow USB drives can become the bottleneck as well. Most modern desktops and laptops won't run into this though.

EDIT: To answer OPs original question. Yes, I'm pretty sure dedicated chips could speed things up but as another poster said, the initial investment would be large. Also, they would only be useful on initial sync, they wouldn't really be necessary after that.

It's $3 a chip. I probably eat that in 12 hours ($3), not something completely rediculous.

I tried running core on a USB external hard drive and it was definitely a lot slower than running it straight off the machine. (although we need an ecdsa chip for stuff to get really ramped up Cheesy

).

Subspace_Labs

newbie

Activity: 3

Merit: 2

Quote from: Jet Cash on January 29, 2019, 10:23:58 AM

I don't think the hardware is the problem. Given that my netbook is a low spec "celery", and it seems to be quite happy sync'ing, I suspect the problem lies with network connectivity. Give things another few years, and babies will be fitted with blockchain implants shortly after birth.

This is not necessarily true. It's possible to have a hardware configuration that is the bottleneck. This is especially true when running Raspberry Pi or similar. Also, slow USB drives can become the bottleneck as well. Most modern desktops and laptops won't run into this though.

EDIT: To answer OPs original question. Yes, I'm pretty sure dedicated chips could speed things up but as another poster said, the initial investment would be large. Also, they would only be useful on initial sync, they wouldn't really be necessary after that.

hosseinamin

jr. member

Activity: 39

Merit: 25

Quote from: jackg on January 29, 2019, 04:51:24 PM

Chips are cheap for one.

Quote

Or we can offer our Clarke ASIC chip 90 GH/s, 75mJ/GH at USD 3.50 per chip.

I made an enquiry as to the price of a bit fury chip. A while ago.

That's mining chip. ASIC (Application Specific Integrated Circuit). It's only for mining. Doing it for validation needs alot more capital. Some estimated about 100mUSD.

jackg

copper member

Activity: 2856

Merit: 3071

https://bit.ly/387FXHi lightning theory

Chips are cheap for one.

Quote

Or we can offer our Clarke ASIC chip 90 GH/s, 75mJ/GH at USD 3.50 per chip.

I made an enquiry as to the price of a bit fury chip. A while ago.

So 90GHs is more than enough.
A block has about 4000 transactions when full.
2 hashes per transaction is 8000.
Then 8000+4000+2000+1000+500+250+125...= <16000
If we don't do any validation of signatures, it becomes 5.6M blocks per second.

hosseinamin

jr. member

Activity: 39

Merit: 25

Quote from: jackg on January 29, 2019, 03:42:30 PM

Quote from: hosseinamin on January 29, 2019, 02:03:03 PM

I agree implementing GPU validation is the cheapest and simplest way to increase validation speed. I have limited experience on writing code for GPU. But I'm interested.

When I run a program on Windows it spossible to run the executable on a gpu. Given Linux has its talents of multitasking I'd have thought it already pushes that sort of process onto a gpu?

@carlton, isn't there something on the Intel instruction set that increases the efficiency of hashing algorithms also. I'm sure someone mentioned that a while back.

Baking ASICs for validating bitcoin would give the highest efficiency. There are two reason for not doing it.

1. ASIC cannot be changed once it's out there.
2. Cost of manufacturing an ASIC is not realistic at the moment. Also it may raise a security issue.

Instead using small compute units is the perfect solution to scale. It improves efficiency against a CPU core. And it's general purpose computing unit.
Look at the thread i did open. https://bitcointalksearch.org/topic/concurrent-validation-engine-for-bitcoin-core-5103538

hosseinamin

jr. member

Activity: 39

Merit: 25

Quote from: jackg on January 29, 2019, 03:42:30 PM

Quote from: hosseinamin on January 29, 2019, 02:03:03 PM

I agree implementing GPU validation is the cheapest and simplest way to increase validation speed. I have limited experience on writing code for GPU. But I'm interested.

When I run a program on Windows it spossible to run the executable on a gpu. Given Linux has its talents of multitasking I'd have thought it already pushes that sort of process onto a gpu?

@carlton, isn't there something on the Intel instruction set that increases the efficiency of hashing algorithms also. I'm sure someone mentioned that a while back.

bitcoin scripting language should get ported to opencl. For it to be able to run on GPU. I'm sure we can find shortcuts to automate most of the process of porting.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: jackg on January 29, 2019, 03:42:30 PM

isn't there something on the Intel instruction set that increases the efficiency of hashing algorithms also

right, SSE4 (or is it 4.1/2 ?). Bitcoin's been using those instructions for SHA256 on x86 since 0.15.0

jackg

copper member

Activity: 2856

Merit: 3071

https://bit.ly/387FXHi lightning theory

Quote from: hosseinamin on January 29, 2019, 02:03:03 PM

I agree implementing GPU validation is the cheapest and simplest way to increase validation speed. I have limited experience on writing code for GPU. But I'm interested.

When I run a program on Windows it spossible to run the executable on a gpu. Given Linux has its talents of multitasking I'd have thought it already pushes that sort of process onto a gpu?

@carlton, isn't there something on the Intel instruction set that increases the efficiency of hashing algorithms also. I'm sure someone mentioned that a while back.

Carlton Banks

legendary

Activity: 3430

Merit: 3080

Quote from: jackg on January 29, 2019, 10:46:40 AM

Ah. I think it is more the signature s that take a while to validate then in comparison to the hashes.

SHA256 is already accelerated on x86 platforms using SSE4 instructions (since Bitcoin 0.15.0)

Signature verification performance can be improved with batch validation (part of the proposed Schnorr sigs soft fork, hopefully coming this year)

hosseinamin

jr. member

Activity: 39

Merit: 25

Quote from: ABCbits on January 29, 2019, 01:06:13 PM

Storage I/O speed and network connectivity usually are the main bottleneck on most cases. Computational power bottleneck usually only happen on barebone PC (which have few cores and .5-2GB) such as Raspberry Pi, at least on initial sync, not when on full nodes.

Bitcoin core already use multi-core/thread for signature verification anyway & ASIC for hashing or/and signature verification only useful for initial sync as CPU should be good enough. IMO GPGPU / GPU acceleration is better and few core developer suggest it, but no one contribute to do it until now since it was suggested on 2013 (https://bitcointalksearch.org/topic/m.1451582)

I agree implementing GPU validation is the cheapest and simplest way to increase validation speed. I have limited experience on writing code for GPU. But I'm interested.

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Storage I/O speed and network connectivity usually are the main bottleneck on most cases. Computational power bottleneck usually only happen on barebone PC (which have few cores and .5-2GB) such as Raspberry Pi, at least on initial sync, not when on full nodes.

Bitcoin core already use multi-core/thread for signature verification anyway & ASIC for hashing or/and signature verification only useful for initial sync as CPU should be good enough. IMO GPGPU / GPU acceleration is better and few core developer suggest it, but no one contribute to do it until now since it was suggested on 2013 (https://bitcointalksearch.org/topic/m.1451582)

jackg

copper member

Activity: 2856

Merit: 3071

https://bit.ly/387FXHi lightning theory

Ah. I think it is more the signature s that take a while to validate then in comparison to the hashes.

I don't think the current main bottle neck jet cash states as the network is the true bottleneck for a private network.

qwk

donator

Activity: 3542

Merit: 3413

Shitcoin Minimalist

Signature validation is only one of a few bottlenecks, and it's unlikely that hardware optimization for that would result in a completely different "user experience".
For a simple user, running a full node and thence requiring both the download and processing of somewhere on the order of 150GB of data ist simply not the best option.
A lightweight SPV-client will usually serve the needs of a regular user best.

Just for reference, a number of bottlenecks to consider:
https://bitcoincore.org/en/2017/03/13/performance-optimizations-1/

Jet Cash

legendary

Activity: 2814

Merit: 2472

https://JetCash.com

I don't think the hardware is the problem. Given that my netbook is a low spec "celery", and it seems to be quite happy sync'ing, I suspect the problem lies with network connectivity. Give things another few years, and babies will be fitted with blockchain implants shortly after birth.

jackg

copper member

Activity: 2856

Merit: 3071

https://bit.ly/387FXHi lightning theory

Something occurred to me a few days ago while I was daydreaming in a lecture, I focused on a slide that said that hardwired hardware is much faster than modifible hardware inside a cpu.

Does this also work with bitcoin?

If there was for example: an asic chip for sha256 and an asic or a cpu core dedicated to the signature validation, a 256Gbyte ram (and any variant upwards) and a hard drive to push everything back onto once the sync was done would this be a possible way for speeding up the initial sync so that only the network becomes the bottleneck or have I missed something?

Topic: Speed up syncing of bitcoin core when the blockchain becomes colossal (Read 317 times)