Automated Bitcoin startup with backup

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: TheArchaeologist on August 23, 2018, 02:21:08 AM

Quote

Did you know you can store bitcoins blockchain on USB stick or external hard drive as the startup directory?

Sure you can do this. But given the size of the entire thing it would be hard to find a USB stick which can hold the entire blockchain!

Actually, flash memory is extremely cheap right now.

You can buy a 256 gb stick (which is enough for the whole blockchain) for 35$.
And a 126 gb stick only costs around 20$.

With the current overthrow of the storage market, most prices are hilariously low. Especially flash memory (SSDs / USBs).

DrakonX

jr. member

Activity: 99

Merit: 4

Quote from: TheArchaeologist on August 23, 2018, 02:28:31 AM

Is that because your on a slow connection? Or do you use a slow drive?

Presumably both is true.
I don't use the wallet.dat from Bitcoin Core, but Bitcoin Armory. Right now I'm not motivated to try ist with the -dbcache parameter. I'll try as soon as I have to rebuild the blockchain.

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote from: DrakonX on July 16, 2018, 05:59:58 AM

The only reason for me to backup daily is that it tooks extremely long to download the last few 10,000 blocks again when I need to redownload the whole blockchain after an blockchain corruption.

Is that because your on a slow connection? Or do you use a slow drive? Did you try to set the -dbcache parameter to a higher value because in my experience this makes a hugh difference when rebuilding the entire blockchain.

I can recreate the entire blockchain in a couple of hours even with -txindex when setting -dbache to 16384 and storing the result on an SSD.

Backups off the public data are literally everywhere, I would worry most about your personal data: the wallet.dat file.

TheArchaeologist

sr. member

Activity: 310

Merit: 727

---------> 1231006505

Quote

So what if your hard drive crashes?

Then you get a new hard drive and sync the entire blockchain again. All full nodes have a copy of the entire chain so there are plenty of backups!

Quote

Did you know you can store bitcoins blockchain on USB stick or external hard drive as the startup directory?

Sure you can do this. But given the size of the entire thing it would be hard to find a USB stick which can hold the entire blockchain! But you can store the data everywhere and then reference it to bitcoind on startop (or at the bitcoin.conf file):

Code:

bitcoind --datadir='/path/to/alternative/'

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: DrakonX on August 12, 2018, 10:18:49 AM

I made a backup of everything, I deleted the original data directory and afterwards I copied the backup of the chainstate folder (and only this) back to its original place.

If you have deleted the data directory and only copied the backup of the chainstate into it, you did delete your local copy of the blockchain.
You need to keep your blocks (*.blk files) if you aren't running core with pruning.

Quote from: DrakonX on August 12, 2018, 10:18:49 AM

The result was that I had to redownload (rsync as you name it) the whole blockchain with consumption of all the time a complete download (and rebuild) requires.

What i was revering to with 'syncing' was the process of core reading the block files and building the chainstate (which usually takes way more time than the download process).

DrakonX

jr. member

Activity: 99

Merit: 4

Quote from: bob123 on August 07, 2018, 09:13:56 AM

If you don't want to resync the whole blockchain (which might take some time depending on the specs), you'll need to backup the chainstate folder.
In case of a corruption of your DB, you will be able to remove the block files and redownload the blocks again. And instead of needing to completely process the whole blockchain to build the chainstate, you will be able to use your backup instead.

This is exactly what I intended, but it doesn't seem to work. I made a backup of everything, I deleted the original data directory and afterwards I copied the backup of the chainstate folder (and only this) back to its original place. The result was that I had to redownload (rsync as you name it) the whole blockchain with consumption of all the time a complete download (and rebuild) requires.

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: DrakonX on August 07, 2018, 09:03:58 AM

So I need to backup everything, but in case of a failure I just need to recover the chainstate folder with the backup?

This depends on what you define as 'need to backup'.
The only thing you really need to backup is the wallet.dat. This file contains all of your private keys (which are needed to access your BTC).
Anything further is helpful, but not necessary to be backed up.

If you don't want to resync the whole blockchain (which might take some time depending on the specs), you'll need to backup the chainstate folder.
In case of a corruption of your DB, you will be able to remove the block files and redownload the blocks again. And instead of needing to completely process the whole blockchain to build the chainstate, you will be able to use your backup instead.

Instead of downloading the corrupted blocks, you might also store the whole blockchain as a backup. But this does need quite a lot of space and doesn't bring that much time advantage with it.
Downloading a few files isn't the problem. Building the chainstate if it got corrupted takes the most work (CPU, I/O, ..) and therefore time.

DrakonX

jr. member

Activity: 99

Merit: 4

So I need to backup everything, but in case of a failure I just need to recover the chainstate folder with the backup?

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: DrakonX on August 04, 2018, 03:46:01 PM

Would you please tell me, how long it takes to reconstruct the whole blockchain with just the chainstate folder?

You can not reconstruct the blockchain out of this data. Your individiual chainstate folder stores data related to your personal transactions / balances. This is being derived from the blockchain data.
This basically is your 'backup' of 'synching the blockchain'.

To restore if something fails, delete and redownload (or use your backup the blockchain) the corrupted files. In case of something happening which leads to your chainstate to break (would require another sync) you can simply copy your backup of the chainstate folder into the correct directory.

DrakonX

jr. member

Activity: 99

Merit: 4

I suppose, it is right what you say about the chainstate folder. Would you please tell me, how long it takes to reconstruct the whole blockchain with just the chainstate folder?

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: DrakonX on July 30, 2018, 04:33:37 AM

Keeping just the chainstate folder seems not to be enough, becaus it seems to build up the whole blockchain from scratch if I delete all the files and folders except the chainstate folder.

I hope you didn't delete your wallet.dat with 'all the files and folders' you have deleted.
If you want to save space, you can run core with the prune-command and set a limit on the maximum available space.

Core is building the chainstate by downloading and verifying all blocks. You can't just delete all files except the chainstate folder and expect it to still work properly.
But you can backup the chainstate directory to have a full functional backup of the current state (balance, UTXOs, ..) the next time it somehow gets corrupted.
It is a fast way to make a backup and to return to a functional state if your system somehow doesn't work properly anymore. But the chainstate folder itself does not replace the whole data directory.

DrakonX

jr. member

Activity: 99

Merit: 4

Keeping just the chainstate folder seems not to be enough, becaus it seems to build up the whole blockchain from scratch if I delete all the files and folders except the chainstate folder.

DrakonX

jr. member

Activity: 99

Merit: 4

We're living in a world where storage space is less an issue than time is. Wink

HCP

legendary

Activity: 2086

Merit: 4361

That all seems a bit redundant, given that the blockchain is effectively backed up by all the nodes... I'd be more worried about making sure I had backups of wallet.dat than the blocks/chainstate.

But I guess if you don't have storage space issues then I guess it doesn't hurt to have 3 copies of the blockchain!!?! Tongue

DrakonX

jr. member

Activity: 99

Merit: 4

The only reason for me to backup daily is that it tooks extremely long to download the last few 10,000 blocks again when I need to redownload the whole blockchain after an blockchain corruption. It happened at least once before. I even backup the backup every month, which I do manually.

Mayby I should proof the method with the chainstate folder though.

bob123

legendary

Activity: 1624

Merit: 2481

Quote from: DrakonX on July 16, 2018, 05:25:42 AM

I use the Windows taskmanager to start automaticly backup the blockchain every I logon on with windows

There isn't a real reason to backup the blockchain daily (except maybe if you are offering an online service which requires an intact blockchain to operate normally).
Usually it should be enough if you just backup the chainstate folder. It contains all relevant information (UTXO's, current verification state of the blockchain, .. .) and it is just a few GB big.

You can find it in the data directory of core (usually: C:\Users\YourUserName\Appdata\Roaming\Bitcoin for windows and ~/.bitcoin/ for linux).

Is there a specific reason you need to backup the blockchain daily ?

DrakonX

jr. member

Activity: 99

Merit: 4

I use the Windows taskmanager to start automaticly backup the blockchain every I logon on with windows from my internal hdd to the external one. Afterwards Bitcoin Core starts automaticly too.

Here is how it goes:

Download FreeFileSync and install it.
Start it and select source hdd and destination hdd and for type "mirroring".
Save as batch and note its location and name.
Start the task scheduler and click on library on the left within it.
Click on new task on the right.
Go to trigger tab and click on new.
Choose "with logon" and click on ok.
Go to actions tab and click on new.
Click on search and choos the FreeFileSync.exe
Insert the noted location and name of the batchfile saved with FreeFileSync into the arguements field.
Click Ok and and click new.
Click on search and choose your Bitcon Core executable (bitcoin-qt.exe).
Insert "-min -testnet=0 -regtest=0" into the arguments field.

Click ok, go to options tab and uncheck "End task if execution is longer than ..."*

Klick ok and ok again and exit the task scheduler.

*Note: I inserted the colored step later with an update of this guide.

Note: The first execution might take several hours.

Topic: Automated Bitcoin startup with backup (Read 337 times)