Hard Drive Speed running a Core Node

cellard

legendary

Activity: 1372

Merit: 1252

Quote from: ranochigo on April 29, 2018, 10:30:35 AM

Quote from: Jet Cash on April 25, 2018, 03:15:04 AM

I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed.

Pruning solves the issue with his limited storage on his macbook. I wasn't addressing his external harddrive speed since that's harder to solve.

Quote from: cellard on April 26, 2018, 11:10:39 AM

Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

Rescan doesn't make the client validate the blockchain. I believe that deleting the chainstate alone won't force the client to check all the blocks again. If you want to be on the safe side, delete the block index too. Either way, it would take somewhat the same amount of time to validate again as your initial synchronization.

I believe chainstate should force validation of blocks again... otherwise these files would lack sense.

If your bottleneck is your computer and you have a fast internet connection then yes, it will not make that much difference if you can download 165 GB pretty fast, but if it's the opposite (fast computer but slow connection) then it will make an huge difference.

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

Quote from: Jet Cash on April 25, 2018, 03:15:04 AM

I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed.

Pruning solves the issue with his limited storage on his macbook. I wasn't addressing his external harddrive speed since that's harder to solve.

Quote from: cellard on April 26, 2018, 11:10:39 AM

Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

Rescan doesn't make the client validate the blockchain. I believe that deleting the chainstate alone won't force the client to check all the blocks again. If you want to be on the safe side, delete the block index too. Either way, it would take somewhat the same amount of time to validate again as your initial synchronization.

cellard

legendary

Activity: 1372

Merit: 1252

Quote from: Jet Cash on April 26, 2018, 02:32:27 PM

I'm pretty certain that the Ubuntu machine has only got 2Gb of ram. and I think it's a Celeron processor. It was a cheap machine that I bought to surf and post whilst I was in the van. I can't see how the blockchain can be corrupted, as Core verifies it at startup. Installing a new hard drive, and a new version of Core will avoid ny chance of infection. I'd need second opinion on the use of the wallet and other files associated with the blockchin, but as they aren't executables, I don't see how they could introduce a vulnerability.

I wouldn't run a full node with 2 gigs of RAM... that is overkill. Ubuntu is too resource consuming if you want to go down that route. If I were to use a 2GB old Celeron for the task I would for sure use Lubuntu, which is a super minimalist version of Ubuntu, ideal for old machines.

Wallet file's risk resides in if they ever sat in an online machine. You want to ideally create a new wallet in a machine that has never seen, and will never see the internet, and use it as cold storage, and use a separate computer to broadcast transactions.

Jet Cash

legendary

Activity: 2814

Merit: 2472

https://JetCash.com

I'm pretty certain that the Ubuntu machine has only got 2Gb of ram. and I think it's a Celeron processor. It was a cheap machine that I bought to surf and post whilst I was in the van. I can't see how the blockchain can be corrupted, as Core verifies it at startup. Installing a new hard drive, and a new version of Core will avoid ny chance of infection. I'd need second opinion on the use of the wallet and other files associated with the blockchin, but as they aren't executables, I don't see how they could introduce a vulnerability.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: cellard on April 26, 2018, 11:10:39 AM

How much RAM? safe (older) laptops tipically will only allow 8 GB and it's ddr2 ram.

4+8 now, but it's not from before 2008 if that's what you mean with "safe".

Quote from: LoyceV on April 26, 2018, 10:46:02 AM

I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, don't want to risk it (and I don't have the space to make a copy first). Any thoughts?.

Quote

You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

Are you sure about this?

Code:

  -rescan
       Rescan the block chain for missing wallet transactions on startup

As far as I know, this only checks your wallet, but doesn't check the blocks. Bitcoin Core assumes the blocks on disk are correct. I would expect rescanning them to use just as much CPU and disk access as when you're downloading them for the first time.

cellard

legendary

Activity: 1372

Merit: 1252

Quote from: LoyceV on April 26, 2018, 10:46:02 AM

I used to have that experience too, but after adding much more RAM and speed improvements within Bitcoin Core, I really don't mind anymore. I'm not going to test it, but I estimate it'll be done within 2 days (on a cheap laptop that's more than 3 years old).

How much RAM? safe (older) laptops tipically will only allow 8 GB and it's ddr2 ram.

Quote from: LoyceV on April 26, 2018, 10:46:02 AM

I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, don't want to risk it (and I don't have the space to make a copy first). Any thoughts?.

Yes removing chainstate files forces a rescan and it's pretty safe as far as I know. You can force a rescan by adding "rescan=1" in bitcoin.conf file or with the terminal adding -rescan.

I will just keep the block files for now, can't be bothered with a full download from scratch. Chainstate files obviously were not imported from windows to linux setup so in theory the blocks that I imported should be valid.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: cellard on April 26, 2018, 10:12:54 AM

How safe is to re-use blockchain files? could they have been compromised if they were in a windows setup in the past?

What if the node you're downloading from is compromised?
The answer is the same: as long as your version of Bitcoin Core isn't compromised, it can verify all blocks again.

Quote

to avoid downloading from scratch which is hell

I used to have that experience too, but after adding much more RAM and speed improvements within Bitcoin Core, I really don't mind anymore. I'm not going to test it, but I estimate it'll be done within 2 days (on a cheap laptop that's more than 3 years old).

Quote

but a part of me is paranoid about the fact that the block files were hosted in a Windows setup for a long time...

I thought it would be a trivial thing to give a rescan-the-blockchain option at startup, but all it can do is rescan the wallet.
I've read that removing the chainstate forces a rescan, but I don't want to risk it (and I don't have the space to make a copy first). Any thoughts?

Quote

Im considering making a clean download from Linux from scratch but just thinking about it gives me an headache by how long it will take for it to sync.

If you have enough space, you can just rename your current .bitcoin and download a new one. If it takes too long, you can always move it back.

cellard

legendary

Activity: 1372

Merit: 1252

Quote from: Jet Cash on April 26, 2018, 07:48:50 AM

My current experience more or less confirms the comments above. I decided to sync the external HDD using the small Ubuntu machine, which uses the external SSD normally. No problems connecting it, but it reports that it is 7 weeks behind, and at one point, it said that it would take 12 weeks to complete the sync.

It's now dropped to 6 days, but I don't think I'll bother. The SSD is reasonably fast, and that's what I use daily. The 2Tb external HDD is really just a backup. The next plan is to create a Bitcoin1 and a Bitcoin2 directory on the HDD, and to use a file management program to move the Core sub-directories into them alternately.

For me, it is important to keep a backup of the blockchain, as I don't think I could handle downloading it again, especially as it is double the size of my original download.

How safe is to re-use blockchain files? could they have been compromised if they were in a windows setup in the past? At the beginning I was using windows to run Bitcoin Core, then I got a Linux setup and moved my block files in there to avoid downloading from scratch which is hell, but a part of me is paranoid about the fact that the block files were hosted in a Windows setup for a long time... I don't see how a hacker would be able to extract value from messing with the blocks file but still a bit paranoid about that. Im considering making a clean download from Linux from scratch but just thinking about it gives me an headache by how long it will take for it to sync.

Jet Cash

legendary

Activity: 2814

Merit: 2472

https://JetCash.com

My current experience more or less confirms the comments above. I decided to sync the external HDD using the small Ubuntu machine, which uses the external SSD normally. No problems connecting it, but it reports that it is 7 weeks behind, and at one point, it said that it would take 12 weeks to complete the sync.

It's now dropped to 6 days, but I don't think I'll bother. The SSD is reasonably fast, and that's what I use daily. The 2Tb external HDD is really just a backup. The next plan is to create a Bitcoin1 and a Bitcoin2 directory on the HDD, and to use a file management program to move the Core sub-directories into them alternately.

For me, it is important to keep a backup of the blockchain, as I don't think I could handle downloading it again, especially as it is double the size of my original download.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: cellard on April 25, 2018, 09:25:51 AM

achow101 doesn't recommend using symlinks, it can lead to blockchain corruption. Everything must be on the same folder, blocks and chainstate files.

It's never happened to me, but even if it would happen, I don't really mind losing my blockchain directory. When it comes to that, I get a new disk and download it again.
I'm not running any critical business environment, it's just my home PC.

cellard

legendary

Activity: 1372

Merit: 1252

Quote from: LoyceV on April 25, 2018, 05:12:47 AM

Quote from: biggieM on April 24, 2018, 07:03:25 PM

Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s.

This doesn't make much sense, as the disk access time is much more important than sequential read write speed. Do you have a link to this recommendation? Could it be the internet speed they recommend to run a node?

To share my 2 cents out of my own experience: I would just install Bitcoin Core, let it create it's default ~/.bitcoin directory on your SSD, then close it.
Then, inside the .bitcoin-directory, move the blocks-directory to your external HDD, and create a simlink to the new location. While you're doing this anyway, create a backup of your wallet.dat.
Start Bitcoin Core again. Most disk activity happens in the chainstate-directory, which is now on your fast SSD. The blocks directory is on your large HDD.

Note though that you should make absolutely sure your USB disk doesn't disconnect while running Bitcoin Core!

achow101 doesn't recommend using symlinks, it can lead to blockchain corruption. Everything must be on the same folder, blocks and chainstate files. Even after the latest updates where you can select where to put the wallet.dat, this doesn't change chaintate and blocks folder must be on the same place.

In my opinion one should try to save money monthly until you can afford an SSD of 1TB of storage, which should last you for years to host the blockchain. HDD disks take a massive toll when validating the blockchain due high i/o activity, the headers may break eventually due wear and tear, maybe you end up saving money long term investing in an SSD.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: biggieM on April 24, 2018, 07:03:25 PM

Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s.

This doesn't make much sense, as the disk access time is much more important than sequential read write speed. Do you have a link to this recommendation? Could it be the internet speed they recommend to run a node?

To share my 2 cents out of my own experience: I would just install Bitcoin Core, let it create it's default ~/.bitcoin directory on your SSD, then close it.
Then, inside the .bitcoin-directory, move the blocks-directory to your external HDD, and create a simlink to the new location. While you're doing this anyway, create a backup of your wallet.dat.
Start Bitcoin Core again. Most disk activity happens in the chainstate-directory, which is now on your fast SSD. The blocks directory is on your large HDD.

Note though that you should make absolutely sure your USB disk doesn't disconnect while running Bitcoin Core!

Jet Cash

legendary

Activity: 2814

Merit: 2472

https://JetCash.com

I've got 3 copies of the blockchain. One on an internal HDD, one on an external SSD, and the third is an external HDD that I use for backup. I've got two computers one running Ubuntu, and one with Windows 10. The fastest combo for sync'ing the blockchain is the external SSD on the Ubuntu machine. I wouldn't bother with pruning, as it isn't the space that is your problem, but the sync speed. I'd be tempted to buy an external 256Gb SSD to download the blockchain, and then copy it onto your HDD. Put everything ( blockchain and wallet ) into a Bitcoin directory, and you can do the download on a faster computer to get started.

ranochigo

legendary

Activity: 3038

Merit: 4418

Crypto Swap Exchange

The synchronizing is going to take forever since its constantly reading and writing to the disk. Why don't you consider pruning it? If you've got at least 550MB on your MacBook, you can prune it without sacrificing too much of the functionality.

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

Syncing Bitcoin Core for you is going to take a long time with slow write speeds. Also, because it is an external drive, the blockchain will have a higher chance of becoming corrupted. So you could store the blockchain on your external drive, but I would not recommend it.

biggieM

newbie

Activity: 1

Merit: 0

Hey, I'm getting a Bitcoin Core node running on an old MacBook Air I have lying around. It doesn't have enough SSD space to store the blockchain, so I want to use an old 1TB external hard drive I have. Its reads and writes are around 25MB/s. Is this fast enough to run a core node? On Bitcoin.org it recommends 100 MB/s. Do you think I could get away with 25? Thanks in advance

-BiggieM

Topic: Hard Drive Speed running a Core Node (Read 253 times)