Pages:
Author

Topic: debug.log runCommand error (Read 3366 times)

legendary
Activity: 1974
Merit: 1029
March 16, 2015, 02:03:33 PM
#44
Should I run 'apt-get autoremove' to remove the above packages that are no longer required?

Every piece of software that you have installed is a potential security hole.

The packages in that list were installed in the past as dependencies of some other software you installed. Later you removed this software but the dependencies remain there. Probably you aren't using those, so it's safe to remove them.


blocknotify is now working!!

Glad to hear that!


Code:
$ dpkg -L dh-apparmor
/.
/usr
/usr/bin
/usr/bin/dh_apparmor
[...]

dh-apparmor is a tool for debian developers ("dh" = "debian helper") and has nothing to do with the actual execution of apparmor.
legendary
Activity: 4228
Merit: 1313
March 16, 2015, 12:06:04 PM
#43

Ubuntu?  Debian?  version?


Ubuntu 12.04

I wonder if there is something different happening between 12.04 or 14.04.  Odd.

At least it is working.
jlp
sr. member
Activity: 266
Merit: 264
March 16, 2015, 11:11:55 AM
#42

Ubuntu?  Debian?  version?


Ubuntu 12.04
full member
Activity: 196
Merit: 103
March 16, 2015, 10:50:34 AM
#41
jlp, I am very glad to hear everything now works, as I suspected.

The remaining apparmor files listed on the system should give no issues.

As to whether your system is insecure without apparmor? Every program run as a user has the rights of that user, which has been normal linux behaviour for ages, so I wouldn'ẗ worry too much.

If you worry, you should reinstall apparmor, and then explicitly make an application profile or allow it the required access.

But if everything works now, I'd just leave it alone for now..

Great to see the issue solved!
legendary
Activity: 4228
Merit: 1313
March 16, 2015, 10:12:57 AM
#40
...
blocknotify is now working!!
...

Glad that is now working.

The odd thing to me is that apparmor is (at least on my three servers - one bitcoin related, two not) there by default (Ubuntu 14.04 on them).  But it causes no problems and I have no profile for bitcoind.

I am not sure if I would auto remove it.  Since it is part of the mainline, now that you know it was the problem, you might want to see if you can figure out why it was causing an issue and fix it that way.

Ubuntu?  Debian?  version?

e.g.
Quote
AppArmor is an established technology first seen in Immunix and later integrated into Ubuntu, Novell/SUSE, and Mandriva. Core AppArmor functionality is in the mainline Linux kernel from 2.6.36 onwards; work is ongoing by AppArmor, Ubuntu and other developers to merge additional AppArmor functionality into the mainline kernel.
https://wiki.ubuntu.com/AppArmor
jlp
sr. member
Activity: 266
Merit: 264
March 16, 2015, 09:22:01 AM
#39
dserrano5:

I did the following

Code:
$ sudo apt-get remove apparmor
[sudo] password for stockbet:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libxxf86dga1 x11-utils libxxf86vm1 xterm libgl1-mesa-dri libxcb-glx0 libgl1-mesa-glx
  libx11-xcb1 libglapi-mesa xbitmaps libxaw7 linux-headers-3.5.0-23-generic libxmu6
  linux-headers-3.5.0-23 libfontenc1 libxtst6 libutempter0 tk8.5 libxcb-shape0 libxv1
  libllvm3.0
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  apparmor
0 upgraded, 0 newly installed, 1 to remove and 55 not upgraded.
After this operation, 1,081 kB disk space will be freed.
Do you want to continue [Y/n]? Y
(Reading database ... 257432 files and directories currently installed.)
Removing apparmor ...
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

Should I run 'apt-get autoremove' to remove the above packages that are no longer required?

Code:
$ ps aux | grep apparmor
username  9244  0.0  0.0   9384   936 pts/0    S+   23:29   0:00 grep apparmor

As per http://stackoverflow.com/questions/9375711/more-elegant-ps-aux-grep-v-grep , grep is included in the results, and so I did the following as well, which returned nothing:

Code:
$ ps aux | grep '[a]pparmor'
$

I rebooted.

blocknotify is now working!!

Cryptowatch.com:

I had done the above already before I saw your posting.

I ran these commands:

Code:
$ sudo dpkg --get-selections | grep apparmor
apparmor deinstall
dh-apparmor install
$

$ apt-cache search apparmor
apparmor - User-space parser utility for AppArmor
apparmor-docs - Documentation for AppArmor
apparmor-notify - AppArmor notification system
apparmor-profiles - Profiles for AppArmor Security policies
apparmor-utils - Utilities for controlling AppArmor
dh-apparmor - AppArmor debhelper routines
libapache2-mod-apparmor - changehat AppArmor library as an Apache module
libapparmor-dev - AppArmor development libraries and header files
libapparmor-perl - AppArmor library Perl bindings
libapparmor1 - changehat AppArmor library
libpam-apparmor - changehat AppArmor library as a PAM module
python-libapparmor - AppArmor library Python bindings
$

To find the files installed with a package, I ran:

Code:
$ dpkg -L apparmor

From a glance, it seems that all of the files shown by the above command are still on my system, even though I had uninstalled apparmor.  There are some files in /etc/apparmor.d/local/ that were not listed by the above command.

All of the files shown below still exist on my system:

Code:
$ dpkg -L dh-apparmor
/.
/usr
/usr/bin
/usr/bin/dh_apparmor
/usr/share
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/dh_apparmor.1.gz
/usr/share/doc
/usr/share/doc/dh-apparmor
/usr/share/doc/dh-apparmor/changelog.Debian.gz
/usr/share/doc/dh-apparmor/copyright
/usr/share/debhelper
/usr/share/debhelper/autoscripts
/usr/share/debhelper/autoscripts/postrm-apparmor
/usr/share/debhelper/autoscripts/postinst-apparmor
$

I don't know if any of the above files matter anymore because blocknotify now works.

dserrano5 and Cryptowatch.com:

Thank you so much for your help.

What about AppArmor?  Without it, isn't my system insecure?

Is there any way to find out the offending AppArmor profile and to fix it?  I googled (startpage.com) for "default profile in apparmor" and cannot find much, other than to assume that apparmor runs the profiles in /etc/apparmor.d.  I looked at some of the profiles in /etc/apparmor.d and cannot figure out how any of them can interfere with bitcoind.

I didn't upgrade my operating system.  I didn't install AppArmor.  What do you think prompted AppArmor to cause the problem?

full member
Activity: 196
Merit: 103
March 15, 2015, 10:32:44 PM
#38
b. and e. (not quoted) together are just a paranoid way of making totally sure that apparmor isn't installed. I'd go with simply removing the packages apparmor-* and reboot (rebooting is the easiest way to ensure it's not loaded; inelegant, I know, but meh).

Yes, that's sound advice.

jlp, as for package:

-- https://en.wikipedia.org/wiki/Linux_package_formats

Packages on a linux systems is usually administrated by various tools, but the more usual ones are apt-get and dpkg.

Learn more:

-- http://www.cyberciti.biz/tips/linux-debian-package-management-cheat-sheet.html

So, when I suggest "a. Find out the name of the package(s) that contains apparmor on your system.", the way to go about solving that task is to understand what to do, then figure out how to do it.

If you don't know what a package is, you search for "What is a package in linux?" The first hit on duckduckgo.com is the wikipedia link a few lines above. Now you know what a package is, and you need to find all packages associated with apparmor.

If you search a bit more, for instance on stackoverflow.com, you will find postings like:

http://stackoverflow.com/questions/12740452/ubuntu-command-to-list-the-installed-software

From there you can learn you can run a command like
Code:
sudo dpkg --get-selections
to view all packages. Now you could either save the output to a text file by doing
Code:
sudo dpkg --get-selections > list.txt
and then look at list.txt for apparmor entries, or you could run a command like
Quote
sudo dpkg --get-selections | grep apparmor

It's also possible to user apt-cache:

-- http://www.howtogeek.com/howto/ubuntu/search-for-install-packages-from-the-ubuntu-command-line/?PageSpeed=noscript

Or you can go directly to Ubuntu, and search for packages:

-- http://packages.ubuntu.com/

When I search for apparmor there, I get this result:
-- http://packages.ubuntu.com/search?keywords=apparmor&searchon=names&suite=trusty§ion=all

As exact hit, I get apparmor.

This means it should be possible to uninstall it with
Code:
sudo apt-get remove apparmor
.

Then to ensure, the system is in a "fresh" state, the easiest thing is to just reboot. I assume that apparmor is the main package for apparmor daemon, and once it's removed, bitcoind should function normally for you again.

Linux can be a bit complicated, so you need to be patient, and spend time learning.

If you do not understand a sentence posing a problem then break the problem down in sub-parts, and make sure you understand every sub-part, if there's a word you do not understand, google it. Once you understand the problem, finding a solution is much simpler. Just trying stuff blindly, seldom works.

There's plenty of people around here wanting to help you, but it's also important that you demonstrate that you've done a real effort to try solve the problem. But don't worry, problem solving skills is something that can be improved.

Let us know how this turns out, I'm sure everyone rots for you to solve the issue with bitcoind not running the external scripts for wallet and blocknotify events.






legendary
Activity: 1974
Merit: 1029
March 15, 2015, 04:52:52 PM
#37
I'm not exactly sure I know what you mean by:


a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.


b. and e. (not quoted) together are just a paranoid way of making totally sure that apparmor isn't installed. I'd go with simply removing the packages apparmor-* and reboot (rebooting is the easiest way to ensure it's not loaded; inelegant, I know, but meh).
jlp
sr. member
Activity: 266
Merit: 264
March 15, 2015, 04:23:26 PM
#36
I'm not exactly sure I know what you mean by:


a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.


https://wiki.ubuntu.com/DebuggingApparmor  seems to shed some light on "package".  Based on this, I did the following:

Code:
$ cd /etc/apparmor.d
$ ls
abstractions  force-complain  tunables        usr.sbin.rsyslogd
cache       local       usr.sbin.mysqld  usr.sbin.tcpdump
disable       sbin.dhclient   usr.sbin.ntpd
$ dpkg -S sbin.dhclient
isc-dhcp-client: /etc/apparmor.d/sbin.dhclient
$ dpkg -S usr.sbin.mysqld
mysql-server-5.5: /etc/apparmor.d/usr.sbin.mysqld
$ dpkg -S usr.sbin.ntpd
ntp: /etc/apparmor.d/usr.sbin.ntpd
ntp: /etc/apparmor/init/network-interface-security/usr.sbin.ntpd
$ dpkg -S usr.sbin.rsyslogd
rsyslog: /etc/apparmor.d/usr.sbin.rsyslogd
$ dpkg -S usr.sbin.tcpdump
tcpdump: /etc/apparmor.d/usr.sbin.tcpdump

Note:  the following from /etc/apparmor.d are folders:

abstractions
cache
disable
force-complain
local
tunables

I looked inside each of the above folders and did not see "bitcoin".  The "abstractions" folder had approximately 73 files.  There were a few files in the other folders.  Did you want me to find the package for all these as well?

Yes, I'm running Ubuntu.  I don't know if AppArmor was added with a recent update.  I read somewhere that AppArmor comes with Ubuntu.

full member
Activity: 196
Merit: 103
March 15, 2015, 12:47:09 PM
#35
What part of my output is unexpected, as compared to your output?

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!

Some more exercises:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.
c. Remove apparmor with something like
Code:
apt-get remove apparmor
.
d. ensure it's not running with
Code:
ps aux | grep apparmor
e. verify that the files in b. is now gone.

Check if bitcoind now works as it should..

It might be that apparmour applied a default profile to applications not specifically listed with a profile. I'm about 100% sure that if you remove apparmor as dserrano5 says, everything will work just fine.

Are you running ubuntu? Could it be that apparmor was added with a recent update?

I also found this link, that might be worth reading:
http://ubuntuforums.org/showthread.php?t=1008906
legendary
Activity: 1974
Merit: 1029
March 15, 2015, 11:03:29 AM
#34
What part of my output is unexpected, as compared to your output?

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!
jlp
sr. member
Activity: 266
Merit: 264
March 15, 2015, 10:48:55 AM
#33

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block Smiley. It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.

Sorry for my slowness.  Is there anything else that I'm not reading correctly?  What part of my output is unexpected, as compared to your output?

Thanks for hanging in here to help me out.


legendary
Activity: 1974
Merit: 1029
March 14, 2015, 08:46:48 PM
#32

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):


Shortly after starting up bitcoind, I checked debug.log.  When I saw the runCommand error, I stopped bitcoind and ran grep.  Doesn't this explain why there is only one block in my mytrace files?

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block Smiley. It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.
jlp
sr. member
Activity: 266
Merit: 264
March 14, 2015, 08:40:41 PM
#31

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):


Shortly after starting up bitcoind, I checked debug.log.  When I saw the runCommand error, I stopped bitcoind and ran grep.  Doesn't this explain why there is only one block in my mytrace files?  If I had let bitcoind run for a while before running grep, there would've been several more invocations of blocknotify and therefore several more occurrences of the runCommand error.  Doesn't this explain it?

jlp
sr. member
Activity: 266
Merit: 264
March 14, 2015, 08:29:57 PM
#30
Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.

Thanks for your suggestion.

Apparmor's profiles are supposed to be in the following directory.  A profile does not exist for bitcoind:

Code:
$ ls /etc/apparmor.d/
abstractions  disable       local      tunables       usr.sbin.ntpd usr.sbin.tcpdump
cache       force-complain  sbin.dhclient  usr.sbin.mysqld  usr.sbin.rsyslogd

Nevertheless, I tried stopping apparmor.  As per http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/ , I did the following:

Code:
$ sudo /etc/init.d/apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."

But it didn't seem to stop when I checked the status:

Code:
$ sudo /etc/init.d/apparmor status
apparmor module is loaded.
6 profiles are loaded.
6 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/mysqld
   /usr/sbin/ntpd
   /usr/sbin/tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/mysqld (837)
   /usr/sbin/ntpd (1604)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

So, as per https://help.ubuntu.com/lts/serverguide/apparmor.html , I tried the following to stop AppArmor:

Code:
$ sudo service apparmor stop
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...
   /etc/rcS.d/S37apparmor

When I checked apparmor status, I got the same as before.

I tried to set bitcoind to complain mode, but it didn't work:

Code:
$ sudo aa-complain /usr/local/bin/bitcoind
sudo: aa-complain: command not found
$

As per https://help.ubuntu.com/community/AppArmor , I did the following to disable AppArmor:

Code:
$ sudo invoke-rc.d apparmor stop
[sudo] password for stockbet:
 * Clearing AppArmor profiles cache
   ...done.
All profile caches have been cleared, but no profiles have been unloaded.
Unloading profiles will leave already running processes permanently
unconfined, which can lead to unexpected situations.

To set a process to complain mode, use the command line tool
'aa-complain'. To really tear down all profiles, run the init script
with the 'teardown' option."
$ sudo update-rc.d -f apparmor remove
 Removing any system startup links for /etc/init.d/apparmor ...

When I checked apparmor status, I got the same as before.

I stopped and started bitcoind:

Code:
$ bitcoind stop
Bitcoin server stopping
$ bitcoind -daemon
Bitcoin server starting
$

…but I still got the runCommand error in .bitcoin/debug.log:

Code:
2015-03-15 01:18:12 runCommand error: system(/bin/true) returned -1

I have bitcoind 0.9.0.  Do I need upgrade it?  I read that version 0.10.0 causes frozen blocks.

legendary
Activity: 1974
Merit: 1029
March 14, 2015, 08:21:52 PM
#29
Is this what you are asking for?:

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):

Code:
$ grep bin.true footreis.*
footreis.13987:01:05:38.978635 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.13988:01:05:38.988687 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.2739:01:12:16.482455 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.2740:01:12:16.491455 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.30087:01:00:44.489005 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.30088:01:00:44.542551 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0
footreis.31270:01:03:26.618716 execve("/bin/sh", ["sh", "-c", "/bin/true"], [/* 20 vars */]) = 0
footreis.31271:01:03:26.627652 execve("/bin/true", ["/bin/true"], [/* 20 vars */]) = 0

Like I and others mentioned, look into apparmor.
jlp
sr. member
Activity: 266
Merit: 264
March 14, 2015, 07:12:31 PM
#28
Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

Is this what you are asking for?:

Code:
$ grep -C 2 exec mytrace.*
mytrace.5727:execve("/usr/local/bin/bitcoind", ["bitcoind"], [/* 13 vars */]) = 0
mytrace.5727-brk(0)                                  = 0x7fe2a93af000
mytrace.5727-access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
$
legendary
Activity: 4228
Merit: 1313
March 14, 2015, 06:58:29 PM
#27
Try looking into AppArmour and see if it could be the culprit.  I think we're running out of Bitcoin related things. :-)

As cryptowatch.com said, perhaps it is interfering with app permissions.



In this thread:
https://bitcointalksearch.org/topic/m.4154179

there was an error elsewhere in the config file that was messing things up.  Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?


Thanks for your suggestion.

Automatic had the following in his bitcoin.conf.  However, he stated that the notify commands randomly started working and randomly stopped working.

Code:
rpcuser=no.
rpcpassword=no.
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/home/bitcoin/block.sh %s
walletnotify=/home/bitcoin/wallet.sh %s
alertnotify=/home/bitcoin/alert.sh %s

I tried the following configs in my bitcoin.conf:

Code:
rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
maxconnections=1000
checklevel=4
keypool=10000
rpcallowip=127.0.0.1
server=1
blocknotify=/bin/true
walletnotify=/bin/true
alertnotify=/bin/true

and

Code:
rpcuser=XXXXXXXXXX
rpcpassword=XXXXXXXXXX
blocknotify=/bin/true
walletnotify=/bin/true

After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:

Code:
2015-03-13 18:45:17 runCommand error: system(/bin/true) returned -1

Any other suggestions that I can try?


full member
Activity: 196
Merit: 103
March 14, 2015, 04:08:32 PM
#26
Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.


legendary
Activity: 1974
Merit: 1029
March 14, 2015, 03:48:46 PM
#25
Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?
Pages:
Jump to: