Why does catching up with the blockchain hammer the disk so much?

paraipan

legendary

Activity: 924

Merit: 1004

Firstbits: 1pirata

Quote from: fghj on December 24, 2012, 06:37:47 AM

Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
_{Also remember to copy it back to hard drive when finished}

Better, buy a 64gb SSD drive (around 50€) and move bitcoin datadir onto it. You should see the difference right away Wink

fghj

member

Activity: 65

Merit: 10

Sorry for necroing this thread, posting something that is obvious to most people (was for me in hindsight) and that will be moot when v0.8 will be released but I didn't found it written anywhere. If you don't have enough RAM for storing entire blockchain in RAMdisk you can put blkindex.dat into RAM and do a symlink to it from datadir. This will also speed up syncing and will require only about 1.8GB of ram instead of 6+GB.
_{Also remember to copy it back to hard drive when finished}

deepceleron

legendary

Activity: 1512

Merit: 1036

Quote from: stakhanov on March 31, 2012, 10:40:37 AM

Quote from: Pieter Wuille on March 31, 2012, 10:14:05 AM

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

You'd believe how much it writes to the disk if you make a new ~8gb disk image, defrag & push all the data to front, and then simply install and run Bitcoin. The blockchain will be fragmented all over the disk from all the writes and erases to the database.

stakhanov

full member

Activity: 175

Merit: 101

Quote from: Pieter Wuille on March 31, 2012, 11:02:21 AM

Quote from: Revalin on March 31, 2012, 10:51:14 AM

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome

Just to be clear, this is a perfectly acceptable answer to me! I just wanted to make sure I wasn't missing something obvious.

Pieter Wuille

legendary

Activity: 1072

Merit: 1181

Quote from: Revalin on March 31, 2012, 10:51:14 AM

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Patches very welcome

Revalin

hero member

Activity: 728

Merit: 500

165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g

The fact that it's possible to download the whole thing onto a RAM disk, manually copy it to disk, and sync with orders of magnitude less of an IO hit suggests that the DB is doing something pedagogically.

Pieter Wuille

legendary

Activity: 1072

Merit: 1181

Quote from: stakhanov on March 31, 2012, 10:40:37 AM

Quote from: Pieter Wuille on March 31, 2012, 10:14:05 AM

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

It takes about half an hour for me, when loading from a local file (a patch that will probably be in 0.7) instead of from the network.

If you have a lot of RAM, you can run it in a RAM drive (if your OS supports that) and watch it fly. Of course, you'll miss the one hard part: not losing or corrupting the on-disk index when your computer crashes.

Revalin

hero member

Activity: 728

Merit: 500

165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g

I was just kidding, but everything you say is correct. The startup and shutdown times are very high. SOME disk hit is expected, but it's still quite excessive.

stakhanov

full member

Activity: 175

Merit: 101

Quote from: Pieter Wuille on March 31, 2012, 10:14:05 AM

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

Well, I suppose you've given it a lot of thought already. But I still have a hard time believing downloading the block chain and creating an index requires all that disk hammering. On my computer, the .bitcoin directory is 1.6GB big. It's big, but not that big. It even fits in the memory of most computers! How can building it require hammering the disk for hours?

Revalin

hero member

Activity: 728

Merit: 500

165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g

They used to be much too close; hence the adjustment.

Obviously there's still considerable room for improvement.

Edit: And why would you ever quit Bitcoin? You smell like a traitor.

Pieter Wuille

legendary

Activity: 1072

Merit: 1181

Quote from: stakhanov on March 31, 2012, 10:09:49 AM

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.

The hard part is not downloading or storing the block chain. It's building, maintaining and guaranteeing the on-disk consistency of the block and transaction index that is necessary for validation.

stakhanov

full member

Activity: 175

Merit: 101

Thanks for your answer. I do have version 0.6x

I wonder if the database commits are simply too close to each other? The blockchain is public information and can always be redownloaded in the even of a crash, there is really no reason to write to the disk several times per second to secure it.

Revalin

hero member

Activity: 728

Merit: 500

165YUuQUWhBz3d27iXKxRiazQnjEtJNG9g

Database transactions require writing a journal, syncing, writing the data, syncing again, and then deleting the journal. That's a much harder disk hit than linear writes.

Try using the just-released 0.6.0. The settings have been changed for better disk performance.

stakhanov

full member

Activity: 175

Merit: 101

I'm using bitcoin-qt on Arch linux, and catching up from a long time ago. I've noticed that the hard disk is working very hard all the time. It seems pretty fishy to me, since I'm downloading from a pretty slow DSL line, the disk should be working far less than that. What is going on?

Topic: Why does catching up with the blockchain hammer the disk so much? (Read 3200 times)