Split blockchain download in different devices

serveria.com

legendary

Activity: 2422

Merit: 1191

Privacy Servers. Since 2009.

Quote from: takuma sato on January 26, 2023, 12:09:09 AM

Let's say you have 3 drives X Y Z

X has 100 GB of free space you would be ok using for the blockchain
Y has 300 GB
Z has 100 GB

Would it be possible to tell the client, "allocate 100GB to X, 300GB to Y, 100GB to Z" and have the files downloaded and verified in the right order?
And you could just keep adding further drives in case you ran out of space. Right now if you run out of space you have to buy a bigger device that allows at least 500GB at least once in case you used pruned mode, but for someone that wants to keep the entire blockchain this would be good. It would also give some usage for smaller drives you don't use much.

Yes, you can theoretically use LVM or some other type of distributed filesystem but that'd be too complicated, especially if you'd like to keep other data on the same drives. I'd recommend to use a bigger single drive. 1TB drives (even SSD) don't cost a fortune nowadays (and don't require any advanced sysadmin skills). Besides, it'll most probably be a more reliable solution (less points of failure). Cool

ABCbits

legendary

Activity: 2870

Merit: 7490

Crypto Swap Exchange

Quote from: AverageGlabella on January 26, 2023, 01:50:57 PM

Quote from: DaveF on January 26, 2023, 11:27:10 AM

--snip--

3 points of failure increases the chances of something breaking but another way to look at it would be if you have 3 devices you would lose what that device was holding and downloading that again would be better then if you lost the entire blockchain on one hard drive because depending on your internet connection that can take a long time. Days or weeks. I think I would prefer to lose a small amount of data and download that then losing the entire thing because it is on one drive.

But does any Bitcoin full node software have such capability (re-download only missing block)? I know Bitcoin Core can do that when only newer block is missing[1], but i don't know whether same behavior apply when early/middle block is missing.

Quote from: LoyceMobile on January 26, 2023, 02:36:23 PM

You can probably symlink the block files to another location, but it's really not worth the effort. Who still uses 100 GB disks?

OP question would make sense if he has several old SSD (usually has 32-256GB capacity).

[1] https://bitcointalksearch.org/topic/blkdat-and-revdat-5433062

LoyceMobile

hero member

Activity: 1659

Merit: 687

LoyceV on the road. Or couch.

You can probably symlink the block files to another location, but it's really not worth the effort. Who still uses 100 GB disks?

NeuroticFish

legendary

Activity: 3668

Merit: 6382

Looking for campaign manager? Contact icopress!

Quote from: takuma sato on January 26, 2023, 12:09:09 AM

Let's say you have 3 drives X Y Z

X has 100 GB of free space you would be ok using for the blockchain
Y has 300 GB
Z has 100 GB

Would it be possible to tell the client, "allocate 100GB to X, 300GB to Y, 100GB to Z" and have the files downloaded and verified in the right order?
And you could just keep adding further drives in case you ran out of space. Right now if you run out of space you have to buy a bigger device that allows at least 500GB at least once in case you used pruned mode, but for someone that wants to keep the entire blockchain this would be good. It would also give some usage for smaller drives you don't use much.

As said, drives are cheap. So there's no reasonably good reason for implement anything like you said.
For now there are settings in config for having different folders in different places or you can use symlinks. However, this means that you'll still need a drive for storing the blocks folder, which is ~477 GB and another drive(s) for the rest.

Right now getting a HDD for the bulk of data (I've bought not long ago a 2TB a CMR HDD exactly for this) is not expensive and, if you set up the chainstate (and indexes) stay on SSD (at least temporarily) the IBD is also relatively fast. So imho doing odd setups with the HDDs for the sake of free space for bitcoin doesn't worth it.

AverageGlabella

legendary

Activity: 1232

Merit: 1080

Quote from: DaveF on January 26, 2023, 11:27:10 AM

Just because there are ways of doing it that does not mean it should be done.
Drives are cheap, used drives with low hours that show good in crystal disk and similar apps in the 1TB range are in the $20 range. External 1TB are under $25
You are going to spend more time trying to get oddball configurations to work then it's worth.

WITH THAT BEING SAID.

As others have pointed out depending on your OS / hardware there are a lot of ways to do it. But at that point, if you are using 3 drives you now have 3 points of failure any one of which has an issue will take your node down.

-Dave

3 points of failure increases the chances of something breaking but another way to look at it would be if you have 3 devices you would lose what that device was holding and downloading that again would be better then if you lost the entire blockchain on one hard drive because depending on your internet connection that can take a long time. Days or weeks. I think I would prefer to lose a small amount of data and download that then losing the entire thing because it is on one drive.

DaveF

legendary

Activity: 3500

Merit: 6320

Crypto Swap Exchange

Just because there are ways of doing it that does not mean it should be done.
Drives are cheap, used drives with low hours that show good in crystal disk and similar apps in the 1TB range are in the $20 range. External 1TB are under $25
You are going to spend more time trying to get oddball configurations to work then it's worth.

WITH THAT BEING SAID.

As others have pointed out depending on your OS / hardware there are a lot of ways to do it. But at that point, if you are using 3 drives you now have 3 points of failure any one of which has an issue will take your node down.

-Dave

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Quote from: NotATether on January 26, 2023, 06:20:39 AM

RAID0 only works with two disks that must be the same size, and be attached to the same RAID controller that often requires the two disks to be almost identical models, if I recall correctly.

You could run them in JBOD mode (just-a-bunch-of-disks), but I'm not sure if Windows itself supports that - it is a feature you can enable for some external drives like WD's if you have their drivers installed though.

It's okay to have different models of drive, only thing affected is the IO speed but thats about it. Valid point on same size, would work if it isn't just that space will be wasted. Storage Spaces would work for > Windows 10, JBOD isn't needed.

Quote from: ?? on ??

These days using RAID controller is optional. Many motherboard these days provide RAID support on software level, which AFAIK have inferior performance and occasionally unstable.

I think the reliability got better for motherboard RAID so it isn't that big of an issue. IRST does help with it on Intel chipsets.

Zubayer

member

Activity: 181

Merit: 10

Working as a Web Developer and Cyber specialist.

Quote from: takuma sato on January 26, 2023, 12:09:09 AM

Let's say you have 3 drives X Y Z

X has 100 GB of free space you would be ok using for the blockchain
Y has 300 GB
Z has 100 GB

Would it be possible to tell the client, "allocate 100GB to X, 300GB to Y, 100GB to Z" and have the files downloaded and verified in the right order?
And you could just keep adding further drives in case you ran out of space. Right now if you run out of space you have to buy a bigger device that allows at least 500GB at least once in case you used pruned mode, but for someone that wants to keep the entire blockchain this would be good. It would also give some usage for smaller drives you don't use much.

It is technically possible to tell a client to allocate specific amounts of storage space to different drives and have the blockchain files downloaded and verified in the correct order. This could be done by specifying the file paths for each drive when configuring the client, and then using a script or program to ensure that the correct files are being downloaded and verified on each drive.

However, it's important to note that managing the storage space across multiple drives in this way can be complex and may require a significant amount of technical knowledge. It would also require the use of a client that has this functionality built-in or can be configured to work in this way.

Additionally, it's important to note that keeping the entire blockchain requires a significant amount of storage space, and even with multiple drives, it may not be practical for some users. For users that want to keep the entire blockchain, it may be more cost-effective to invest in a larger drive or a storage solution specifically designed for this purpose.

Also, it's worth to mention that different blockchain client software have different requirements and capabilities for storage, and not all of them may support this type of configuration.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: ranochigo on January 26, 2023, 03:26:49 AM

You can also use Storage Spaces on Windows to create logical volumes out of multiple disk, or alternatively RAID0 would work as well.

RAID0 only works with two disks that must be the same size, and be attached to the same RAID controller that often requires the two disks to be almost identical models, if I recall correctly.

You could run them in JBOD mode (just-a-bunch-of-disks), but I'm not sure if Windows itself supports that - it is a feature you can enable for some external drives like WD's if you have their drivers installed though.

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

You can also use Storage Spaces on Windows to create logical volumes out of multiple disk, or alternatively RAID0 would work as well. Since the block data are redundant after validation (barring reindex, import, etc), I believe that it would be possible to have them split into multiple volumes, though it doesn't provide any substantial benefits. That being said, I highly doubt that splitting the block data natively would work anytime soon, because there is no need.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

It is better to combine those storage devices using a file system. In fact, if you are using Linux, then you can already achieve this using LVM by creating a volume group out of all those physical disk and then creating a logical volume that makes a big block device out of all those disks, after which you can put a filesystem on it.

It is not particularly simple to rewire Bitcoin Core to download the blockchain onto multiple disks.

takuma sato

sr. member

Activity: 317

Merit: 448

Let's say you have 3 drives X Y Z

X has 100 GB of free space you would be ok using for the blockchain
Y has 300 GB
Z has 100 GB

Would it be possible to tell the client, "allocate 100GB to X, 300GB to Y, 100GB to Z" and have the files downloaded and verified in the right order?
And you could just keep adding further drives in case you ran out of space. Right now if you run out of space you have to buy a bigger device that allows at least 500GB at least once in case you used pruned mode, but for someone that wants to keep the entire blockchain this would be good. It would also give some usage for smaller drives you don't use much.

Topic: Split blockchain download in different devices (Read 150 times)