Topic: Segfault on hardened Linux systems (Read 3820 times)

I'm running bitcoin on hardened Gentoo.  Everything works short of generation.  If I understand the conversation so far, some optimizations fail on hardened systems, but if they are disabled, generation will likely work.  On the other hand, the integrity of the network as a whole is bolstered by legitimate clients working efficiently, so removing optimization will probably be a net loss.

How about a compile-time switch?  It's not uncommon for optimized code to get along poorly with hardening measures.  I'm not familiar with the code base, let alone the developers, so I couldn't intelligently guess about the tradeoffs involved, but it seems to me that it would make sense to include a toggle that defaults to "optimize" (current condition), but can be flipped to "just do it the slow ugly way".  That way I could contribute my CPU cycles (if somewhat inefficiently), and the vast majority of the rest of the world, who don't run extremely hardened systems, don't have to be drastically affected.  Ideally that could trickle down to a Gentoo USE flag.  

I'll be happy to help with testing, provide traces, etc.  My system is protected by ASLR, non-executable stacks, GCC's stack-smashing protection, and any other bit I could flip in the kernel or elsewhere to harden the system, excluding mandatory access control (so no selinux, grsecurity, etc).  If it runs on my rig, it should run anywhere.

If I disappear, my email username is aabugher, provider is gmail.

How about leaving the optimizations in there, unless/until they cause some problem? If an optimization causes a maintenance problem, then it can be removed.

If there's hostile action against bitcoin, it might be valuable to muster every last CPU cycle by encouraging everyone to turn on generation.

When the standard client was patched to fix the overflow bug, the "valid" block chain overtook the "sabotaged" one in less than a day. One of the reasons for that was the success of the pleading (in this forum) for everyone to install the new client as soon as possible, and to turn on generation.

gcc -v
Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-4.4.4-r2/work/gcc-4.4.4/configure --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/4.4.4 --includedir=/usr/lib/gcc/i686-pc-linux-gnu/4.4.4/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/4.4.4 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/4.4.4/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/4.4.4/info --with-gxx-include-dir=/usr/lib/gcc/i686-pc-linux-gnu/4.4.4/include/g++-v4 --host=i686-pc-linux-gnu --build=i686-pc-linux-gnu --disable-altivec --disable-fixed-point --without-ppl --without-cloog --disable-nls --with-system-zlib --disable-werror --enable-secureplt --disable-multilib --enable-libmudflap --disable-libssp --enable-esp --enable-libgomp --with-python-dir=/share/gcc-data/i686-pc-linux-gnu/4.4.4/python --enable-checking=release --disable-libgcj --with-arch=i686 --enable-languages=c,c++ --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --with-bugurl=http://bugs.gentoo.org/ --with-pkgversion='Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5'
Thread model: posix
gcc version 4.4.4 (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5)


Fine with me.

What version of gcc are you using?  After a little googling I found this thread about the same issue:

Unless somebody volunteers to fix/maintain this, I'm inclined to simply remove all of the "try to make the CPU miner go faster" optimizations from bitcoin.  CPU mining is, for most people, a waste of electricity.

Doesn't seem ok.

I haven't spend time on this any more. I'll do a check this evening.

BioMike:  any progress tracking this down?

I just committed a fix to the git integration tree CallCPUID code to declare ebx/edx clobbered...

@Hal, I'll be giving that a try this weekend.

@ArtForz, it falls over that when there is no optimalization  in place (gcc default), with optimalizations in place it falls over the other part.

Hrrrm, could this be caused by the asm in CallCPUID? CPUID clobbers ebx and edx, yet we don't seem to be saying so...

Maybe try putting

return false;

as the first line of Detect128BitSSE2(), see if that fixes it. If so, and you're really energetic, move the return down to just before, then just after the memcpy, to confirm that's where the problem is.

Doesn't seem to be SSE related (building without SSE support also causes it to segfault). Assembly isn't in that function as far as I can see (might be in one of the boost calls though).

I also noticed that the backtrace points to an other line then the one I saw earlier (still in the same function, only a few lines above, pointing to a memcpy), although nothing changed on that code. This where it points to now:


This seems to be caused by the changed optimalisation.

The CPU miner code has all sorts of now-mostly-worthless (because GPU mining is so much more energy-efficient than CPU mining) optimizations.  Maybe hardened Linux doesn't like the assembly code or SSE instructions?

Took some time. Removed all cflags except -ggdb. Same problem, it does however give an extra line in the backtrace (at place 0): memcpy@plt
No idea what to do with that.

Sure, although there is little optimization in place. And the bit there is , is fairly standard (-O2, -march=pentium4, -pipe, -ggdb).
Is "-O1, -ggdb" sufficiently unoptimized?

Traces on optimized code are unreliable. Can you build without optimization?

You are right. 'static' moves the variable from the stack the the BSS segment, which is always initialised with zeroes. I didn't notice.

lol, my bad

I don't think it's so weird, it's just ensuring it will only print that cpu info line for the first call of the function.

I think it's randomly initialized.

http://stackoverflow.com/questions/827393/default-value-for-bool-in-c

They're not talking about the static case tho.

Nonetheless this code looks really really weird certainly missing lines or context.

// Okay let's initialize a variable to a fixed or random value
// If it's true
   // Set it to true,
   // then do something

Reminds me of facepalming hard when reading stuff like

if (v == true) {
    return(true);
}

Not sure about C++, but in C, "static bool foo;" is guaranteed to initialize 'foo' to false.

fPrinted is uninitialized. Give it a value before testing on it!

Cross posting into its own topic, maybe more people have more knowledge about these type of things,
but might have missed it in the gentoo ebuild topic.

Building bitcoind on hardened linux seems to cause a segfault when setting the amount of cores to
use for generating bitcoins. Starting with -gen=0 doesn't trigger the segfault.

-----
Ok, I've changed one of my systems to a hardened system and was able to reproduce the bug
on version 0.3.17. Here is the backtrace. No idea what is going wrong.


The misbehaving line is:

in the following code:


fPrinted is false according to the backtrace, I don't know why this is causing the segfault?