Bitcoind crashes after logging off of SSH session

bigbeninlondon

hero member

Activity: 742

Merit: 500

Quote from: Foxpup on August 24, 2014, 07:19:17 AM

Quote from: bigbeninlondon on August 24, 2014, 06:27:01 AM

All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.

I agree; I was also a bit put off by the process as well at first. I guess I just saw it so many places I got used to it.

Foxpup

legendary

Activity: 4551

Merit: 3445

Vile Vixen and Miss Bitcointalk 2021-2023

Quote from: bigbeninlondon on August 24, 2014, 06:27:01 AM

All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.

Quote from: bigbeninlondon on August 24, 2014, 06:27:01 AM

What's the importance (if any) of leaving off the "group" in your chown command above?

It's not left off, it's left blank (note the colon). Leaving it blank sets it to the login group of the specified user, which may just be the user itself, or it may be an actual group (the groups command will tell you which). (And leaving the group off entirely (by omitting the colon) will just leave the group set to "root", which isn't what you want at all.)

bigbeninlondon

hero member

Activity: 742

Merit: 500

Quote from: Foxpup on August 23, 2014, 08:52:56 PM

Quote from: bigbeninlondon on August 23, 2014, 07:26:08 PM

Without it I get permission denied errors. I have to do everything in Ubuntu with sudo...

Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):

Code:

sudo chown -R --from root:root ben: /home/ben

It's either that or your path's messed up...

Yea that's probably how it went. Most of this is really a foray into more advanced linux topics (trying to build from source, running a python p2pool node, setting up init.d scripts, that kind of thing).

All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

What's the importance (if any) of leaving off the "group" in your chown command above?

Justin00

legendary

Activity: 910

Merit: 1000

★YoBit.Net★ 350+ Coins Exchange & Dice

Been using chown for ~15 years, Didn't know you could do --from (Never bothered to look, but still.. )

thanks

Foxpup

legendary

Activity: 4551

Merit: 3445

Vile Vixen and Miss Bitcointalk 2021-2023

Quote from: bigbeninlondon on August 23, 2014, 07:26:08 PM

Without it I get permission denied errors. I have to do everything in Ubuntu with sudo...

Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):

Code:

sudo chown -R --from root:root ben: /home/ben

It's either that or your path's messed up...

bigbeninlondon

hero member

Activity: 742

Merit: 500

Quote from: dserrano5 on August 23, 2014, 04:51:05 PM

Quote from: bigbeninlondon on August 23, 2014, 04:38:04 PM

Moved data directory, looking at 6 hours of uptime with no loss of connection.

Now it's a good time to drop that dangerous 'sudo' you have there.

Without it I get permission denied errors. I have to do everything in Ubuntu with sudo...

dserrano5

legendary

Activity: 1974

Merit: 1030

Quote from: bigbeninlondon on August 23, 2014, 04:38:04 PM

Moved data directory, looking at 6 hours of uptime with no loss of connection.

Now it's a good time to drop that dangerous 'sudo' you have there.

bigbeninlondon

hero member

Activity: 742

Merit: 500

Update:

Moved data directory, looking at 6 hours of uptime with no loss of connection. It appears that the ~ directory is doing something fishy. I had about 3 months of uptime, and it looks like a reboot may have sent a configuration off the deep end. Off to hunt it down. Will update again in 24 hours if its still up.

cr1776

legendary

Activity: 4270

Merit: 1313

Quote from: bigbeninlondon on August 23, 2014, 08:34:05 AM

Quote from: btchris on August 23, 2014, 07:54:01 AM

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

Forgot to mention though, this just started happening. I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption.

I've moved the directory and short exits from ssh seem to be working. I'm wondering if a restart has re-enabled home drive encryption.

I've been running a similar node too and just moved it to a new server. I didn't see that with or without screen (just as a datapoint -ubuntu 14.04).

Foxpup

legendary

Activity: 4551

Merit: 3445

Vile Vixen and Miss Bitcointalk 2021-2023

Quote from: bigbeninlondon on August 23, 2014, 08:25:28 AM

I am using -daemon to daemonize it. It's the last flag on the command.

D'oh. I need more caffeine, apparently.

bigbeninlondon

hero member

Activity: 742

Merit: 500

Quote from: btchris on August 23, 2014, 07:54:01 AM

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

Forgot to mention though, this just started happening. I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption.

I've moved the directory and short exits from ssh seem to be working. I'm wondering if a restart has re-enabled home drive encryption.

bigbeninlondon

hero member

Activity: 742

Merit: 500

Quote from: Foxpup on August 23, 2014, 07:49:04 AM

You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise.

I am using -daemon to daemonize it. It's the last flag on the command.

Quote from: btchris on August 23, 2014, 07:54:01 AM

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

I'll give that a try and see what happens.

btchris

hero member

Activity: 672

Merit: 504

a.k.a. gurnec on GitHub

As others have said, screen is your friend.

Ubuntu Server 13 also has byobu (installed by default I believe), just run byobu-enable, and then choose your activation key (press Ctrl-a, and choose an option, I prefer the first). Then instead of logging out, use Ctrl-a d (on either an SSH session or the console).

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

Foxpup

legendary

Activity: 4551

Merit: 3445

Vile Vixen and Miss Bitcointalk 2021-2023

You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise.

bigbeninlondon

hero member

Activity: 742

Merit: 500

I haven't really pursued that option just because what is a "daemon" actually for if it doesn't daemonize?

I plugged in a monitor and keyboard and logged in to the machine directly, ran the daemon and left it logged in. That's kept it from crashing so far this morning...

cr1776

legendary

Activity: 4270

Merit: 1313

Quote from: Justin00 on August 23, 2014, 04:45:34 AM

Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen"

Yes, running with screen should help.

Justin00

legendary

Activity: 910

Merit: 1000

★YoBit.Net★ 350+ Coins Exchange & Dice

Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen"

bigbeninlondon

hero member

Activity: 742

Merit: 500

Ok,

I've been around and around and I have no earthly idea what's going on here.

I built bitcoind ("version" : 99900) from git on a headless ubuntu 13 machine. Running the following command:

Code:

sudo bitcoind -datadir=/home/ben/.bitcoin/ -conf=/home/ben/.bitcoin/bitcoin.conf -daemon

Everything works great, as long as I remain connected to my ssh session. Logging off for a few seconds doesn't interrupt, but if I leave it disconnected for more than a few seconds, I get the following error:

Code:

2014-08-23 06:02:13 LevelDB read failure: IO error: /home/ben/.bitcoin/chainstate/000174.sst: No such file or directory
2014-08-23 06:02:13 IO error: /home/ben/.bitcoin/chainstate/000174.sst: No such file or directory
2014-08-23 06:02:13 *** System error: Database I/O error
2014-08-23 06:02:13 Error: System error: Database I/O error
2014-08-23 06:02:13 ERROR: ProcessBlock() : AcceptBlock FAILED
2014-08-23 06:02:13 Loaded 754 blocks from external file in 4783ms
2014-08-23 06:02:13 Reindexing block file blk00006.dat...
2014-08-23 06:02:13 addcon thread interrupt
2014-08-23 06:02:13 opencon thread interrupt
2014-08-23 06:02:13 msghand thread interrupt
2014-08-23 06:02:13 dumpaddr thread stop
2014-08-23 06:02:13 ProcessBlock: ORPHAN BLOCK 0, prev=00000000000007169e562472fb20f0e940fc2b17b8aa27be620c49f0da3abc14
2014-08-23 06:02:13 net thread interrupt
2014-08-23 06:02:13 Shutdown : In progress...
2014-08-23 06:02:13 StopNode()
2014-08-23 06:02:13 Shutdown : done

Different "sst" file every time I restart. Doesn't matter how far along I am in the chain.

For some reason, it's as if the chainstate sst files just "disappear" after I log off my ssh session. Restarting gives me:

Code:

Error opening block database.

Do you want to rebuild the block database now?

Running again with "-reindex" re-indexes just fine and starts it back up; that is until I exit my ssh session again.

I've rebuilt bitcoin from source, downloaded the blockchain about 40 times, moved the data directory, etc.

Any suggestions?

Topic: Bitcoind crashes after logging off of SSH session (Read 1778 times)