Topic: [Solved]Getting "Error reading from database, shutting down" after several hours (Read 2296 times)

My guess is that there was another computer that was running something (e.g antivirus) on the files in the shared folder which was causing all of your problems.

Well, it's been up for well over a day now so it appears moving the folder solved the issue.

Thanks everyone. 

Well, so far it's been working ok, but it doesn't mean it's fixed.  The amount of time it takes did vary some before, though right now it's lasted more than 12 hours and is still going strong.

Guess I'll let it run still, see how it goes.

Thanks for the help so far, everyone. 

Not a fix,

My 0.11.0 bitcoin-qt on 64-bit linux showed comparable erratic and unreliable "error reading from database" shutdowns.
This diagnoses that it is not a windows specific fault.
Please could the devs fit either a button or an hourly process cleanup to do whatever needs doing; it could be the boost multithreaded equivalent of a few C program fclose(all); or it could be littering the database with unclosed files.

Does anyone else get similar?

Oh, and just another odd tidbit of info.  I compared when Bitnodes said it "stalled" to the log entries above.  According to the log it stopped at midnight, but according to Bitnodes it was still working and responding to queries up to 3am.  5 hours later it shows when it came back after I restarted it just after 8am.

Seems it worked from memory for those three hours after the logged error.

I moved it from /storage/Backups/bitcoin to /storage/bitcoin.  I also changed the owner/group to my own, still with the rwxrwxrwx permissions.  I really need the folder to stay in that raid array, since I don't have that much room in the root filesystem.  But where it is now is not shared, so guess we'll see what happens overnight.

Thats certainly faster than a complete redownload from scratch with the 64 bit version.

The only cron job I can think of is the indexing done for all files, not just the shared folders.  Thing is it was the same thing happening on Windows, on a drive and folder that wasn't shared.  That would only happen with the 64-bit one, it would quit after a while, usually with the same error as in the title, sometimes with other weird errors.  32-bit would work fine.

I can move the folder to somewhere that is not part of the SMB share just to test it though.

I dont think its related to 32/64-bit. Could this an SMB issue? Is there a cronjob that is going through the files every 12 hours for indexing or something along those lines?

Yes, I do.  ;-)

Permissions on the folder has a special group, since some of these folders are shared through SMB, of group smbusers and user smbguest (smbguest is the owner).  My account is a member of smbusers.  The permissions are rwxrwxrwx for all files in the folder, and the sticky bit is set for the group so it inherits automatically for any file or folder made inside it.

Like I said earlier, what doesn't make sense is I can click that error message from bitcoin-qt, which then closes the window, then immediately double-click the desktop icon to start it back up and it works fine again, for at least 8-12 hours.

It's done the same thing every day since I set up the 64-bit version.

I can't get the 32-bit one to even start up (doesn't even get to start checking the Blockchain), since there are some 32-bit libraries missing from the Centos install (I set it up as pure 64-bit to start with).  When I tried running it the first complaint was it couldn't find libfontconfig.so (might have been .so.0 on the end, not sure now).  So I installed the 32-bit package for that, which cleared that error.  But then it says it cannot find a file, but doesn't say what file, so I can't go any farther with it.  That's why I went with the 64-bit version when I set it up under Centos this time.  I would give the 32-bit one a try to see if it is stable like it's been everywhere else, but there is nothing in the docs that says what the software dependencies are.

Does the user that is running core have r/w access to the file(s)?

When I had it running on my Windows 7 computer, I had the 64-bit installed first (since it was 64-bit Windows).  Since it was REALLY unstable (wouldn't go more than 12 hours without crashing, sometimes corrupting the database and redownloading the blockchain) I then loaded the 32-bit and made sure it loaded the old data files (which were working in the 64-bit version when I shut it down).  That worked solid, no problems for at least a month.

I kinda doubt there might be an issue with the files between 32 and 64 bit, at least for Windows anyway.  Maybe for Linux though.

When I set up the Fedora VM, I made it 32-bit from the start since I didn't want to mess with the stability issues like I saw with the Windows 64-bit version.  That install had been there for months, starting with the data files that the Windows box had (copied them over and replaced the contents of the Data folder in the linux install).

Hmm, this is odd.  Here's the three lines in debug.log from around when it happened:
2015-12-21 06:00:05 LevelDB read failure: IO error: /storage/Backups/bitcoin/data/chainstate/3047481.ldb: Permission denied
2015-12-21 06:00:05 IO error: /storage/Backups/bitcoin/data/chainstate/3047481.ldb: Permission denied
2015-12-21 14:06:29 Error reading from database: Database I/O error

Interesting that it doesn't record the error that is prompted until I clicked it this morning (just after 8am CST US time, which matches up to the last time stamp since that is UTC shown).

The line just before the LevelDB entry is from normal operation (lots of entries about the rate limiter kicking in on a free transaction).  Not sure why it thinks there is a permission denied issue, since immediately restarting the client works fine.

For reference, the /storage folder is a mount point for a software RAID5 array of four 2 TB drives, formatted EXT4. I've got others things (like a Windows VM running in Virtualbox) that have data files stored on that same volume, which were all working fine at the same time this happened.

I checked the kernel's messages log at midnight, matching up to that time stamp, and there's nothing odd listed.

There might be compatibility issues between how the 32 bit version stored data and how the 64 bit version does. Your best option would probably be to do a full resync, at the very least you will need to reindex.

Can you provide the debug.log?

Forgot to mention, this is the tgz download of the compiled core.  I can try compiling from source if needed.

Lately I changed where my Bitcoin Core install was located, moving the data out from a Fedora VM to the host system and setting up Bitcoin Core 0.11.2 64-bit.  The host system is running 64-bit Centos 6 (whatever latest is, kept up to date against repos) on a Core2Quad 2.4 GHz CPU with 8GB RAM in an old Optiplex 755 (server is stable, up 24/7 and rarely gets rebooted).  In the 32-bit Fedora VM on the same computer I ran the 32-bit version, and it ran for months without issues.  Since moving the data out to the host and loading the client, I can't get it to stay up for more than 24 hours.

I've got the node linked to Bitnodes so I know when it goes up or down, and it's been very consistent.  Data shows steady response times for anywhere from 12-16 hours, then it will either say it stalled or is down.  When I check the client, it shows the error "Error reading from database, shutting down".  When I ok that, it closes the client immediately.

Odd bit is I can immediately reopen the client and it loads and works (after syncing that is).  Granted, I never seem to catch it when it happens, so can't say if there is anything else that happens just before the error.  This Centos box is my home network server, so it's not monitored unless something isn't working.

Is there a problem in the 64-bit version?  Since I had a similar issue on Windows when trying this (64-bit was seriously not stable with the same error and worked fine after restarting it, 32-bit worked fine for weeks) I tried running the 32-bit version, but can't figure out what 32-bit packages I need to install to get it working.  The Bitcoin Core documentation is seriously lacking in what software requirements there are for Linux machines and the messages I get do not point to specific modules or libs.

So, any ideas on how to fix this?