Putting the database on a USB key | Bitcointalksearch.org

randy-waterhouse

newbie

Activity: 41

Merit: 0

Quote from: kjj on September 04, 2012, 10:00:58 PM

Hard to say if the problem is the USB bus or the NAND speed (or bus). I'll do a little research, see if I can find a fast USB 3.0 drive to test with.

It's not that hard really .... bitcoin absolutely thrashes the disks ... it's like running Windows instead of Linux

Raize

donator

Activity: 1419

Merit: 1015

Wow, this is bizarre, I was just working on converting my ACARD RAM drive to be a standalone db store as I started reading this thread. I assume those of you setting up new nodes are using more powerful RAM drives, so what are you using? Anyone planning on something that can do 128 GB or more in anticipation of the blockchain increasing vastly in size in a short period of time?

racerguy

sr. member

Activity: 270

Merit: 250

USB is plenty fast enough for bitcoin, it's just that lots of writes are the achilles heel of usb flash storage.

kjj

legendary

Activity: 1302

Merit: 1026

Hard to say if the problem is the USB bus or the NAND speed (or bus). I'll do a little research, see if I can find a fast USB 3.0 drive to test with.

Vladimir

hero member

Activity: 812

Merit: 1001

-

Nope, trust me, even the best MLC USB drives do not live in servers long. With all the trashing bitcoind gives em they will not last long at all. It is of course unless the system is built so that writes are rare, but this certainly not the case with bitcoind. Also USB interface is surely not the bottleneck, erasing/rewriting blocks is what takes time AFAIK.

SLC USB drives is another matter of course, but try to find one and even if you do, it would cost an arm and a leg and whatever your miner mines during 2013. Wink

grondilu

legendary

Activity: 1288

Merit: 1080

Quote from: kjj on September 04, 2012, 08:15:38 PM

I have serious doubts about killing flash drives. Modern flash chips can take a pretty hefty number of writes and should last for a decent number of years, even if written to at full interface speed nonstop.

That said, for running a full node, USB flash drives are too damn slow. At least the cheap ones I tried were.

I'm pretty sure it's not a problem with the flash storage device, but rather with the USB connection which just can't transmit data fast enough. I guess the problem would be the same with any kind of USB external disk drive.

kjj

legendary

Activity: 1302

Merit: 1026

Quote from: Vladimir on September 04, 2012, 03:35:44 PM

Quote from: grondilu on August 29, 2012, 08:23:28 AM

Has anyone succeeded in using a usb key to store the bitcoin database ? If so, which format ?

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

I have serious doubts about killing flash drives. Modern flash chips can take a pretty hefty number of writes and should last for a decent number of years, even if written to at full interface speed nonstop.

That said, for running a full node, USB flash drives are too damn slow. At least the cheap ones I tried were. I use RAM drives for everything now too.

Vladimir

hero member

Activity: 812

Merit: 1001

-

RAM is dirt cheap these days

grondilu

legendary

Activity: 1288

Merit: 1080

Quote from: Vladimir on September 04, 2012, 03:35:44 PM

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

Ok. Good to know. But if you use a memory based fs, you need quite a lot of RAM, right? I only have 1Go so I guess that won't do.

Vladimir

hero member

Activity: 812

Merit: 1001

-

Quote from: grondilu on August 29, 2012, 08:23:28 AM

Has anyone succeeded in using a usb key to store the bitcoin database ? If so, which format ?

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

wabber

member

Activity: 85

Merit: 10

well actually the error says your client version is too old. Try to get the latest from bitcoin.org

grondilu

legendary

Activity: 1288

Merit: 1080

I've recently tried to install my OS (debian sid) on a USB key (ext4 formatted). As this key has not much storage room, I thought of using a second removable media (SD card or USB) where to put the bitcoin database.

Problem: it never actually work well enough. At first, everything's fine: the database and the index get filled, but eventually everything is stuck and no more block is added in the block chain. And I get a Warning in the client saying:

WARNING: Displayed transactions may not be correct! You may need to upgrade, or other nodes may need to upgrade.

I look in the debug.log and to me it doesn't seem like a connection problem. I get a repetitive pattern of messages like:

askfor tx 5fce383170099b52a632 0 sending getdata: tx 5fce383170099b52a632 askfor tx 5fce383170099b52a632 1346243977000000 ERROR: FetchInputs() : 5fce383170 mempool Tx prev not found 04e6a51ecb ERROR: CTxMemPool::accept() : FetchInputs failed 5fce383170 stored orphan tx 5fce383170 (mapsz 10001) mapOrphan overflow, removed 1 tx

My guess is that with low bitrate media, things screw up at some point. When I use my internal, non-SSD hard drive, everything works fine and I can update the database up to the latest block.

I used ext4 for my USB key and I suspect that it would work better with vfat but then I could not use symbolic links to point to my wallet.

Has anyone succeeded in using a usb key to store the bitcoin database ? If so, which format ?

Topic: Putting the database on a USB key (Read 1761 times)