Every piece of software that you have installed is a potential security hole.

The packages in that list were installed in the past as dependencies of some other software you installed. Later you removed this software but the dependencies remain there. Probably you aren't using those, so it's safe to remove them.



Glad to hear that!



dh-apparmor is a tool for debian developers ("dh" = "debian helper") and has nothing to do with the actual execution of apparmor.

I wonder if there is something different happening between 12.04 or 14.04.  Odd.

At least it is working.

Ubuntu 12.04

jlp, I am very glad to hear everything now works, as I suspected.

The remaining apparmor files listed on the system should give no issues.

As to whether your system is insecure without apparmor? Every program run as a user has the rights of that user, which has been normal linux behaviour for ages, so I wouldn'ẗ worry too much.

If you worry, you should reinstall apparmor, and then explicitly make an application profile or allow it the required access.

But if everything works now, I'd just leave it alone for now..

Great to see the issue solved!

Glad that is now working.

The odd thing to me is that apparmor is (at least on my three servers - one bitcoin related, two not) there by default (Ubuntu 14.04 on them).  But it causes no problems and I have no profile for bitcoind.

I am not sure if I would auto remove it.  Since it is part of the mainline, now that you know it was the problem, you might want to see if you can figure out why it was causing an issue and fix it that way.

Ubuntu?  Debian?  version?

e.g.

dserrano5:

I did the following


Should I run 'apt-get autoremove' to remove the above packages that are no longer required?


As per http://stackoverflow.com/questions/9375711/more-elegant-ps-aux-grep-v-grep , grep is included in the results, and so I did the following as well, which returned nothing:


I rebooted.

blocknotify is now working!!

Cryptowatch.com:

I had done the above already before I saw your posting.

I ran these commands:


To find the files installed with a package, I ran:


From a glance, it seems that all of the files shown by the above command are still on my system, even though I had uninstalled apparmor.  There are some files in /etc/apparmor.d/local/ that were not listed by the above command.

All of the files shown below still exist on my system:


I don't know if any of the above files matter anymore because blocknotify now works.

dserrano5 and Cryptowatch.com:

Thank you so much for your help.

What about AppArmor?  Without it, isn't my system insecure?

Is there any way to find out the offending AppArmor profile and to fix it?  I googled (startpage.com) for "default profile in apparmor" and cannot find much, other than to assume that apparmor runs the profiles in /etc/apparmor.d.  I looked at some of the profiles in /etc/apparmor.d and cannot figure out how any of them can interfere with bitcoind.

I didn't upgrade my operating system.  I didn't install AppArmor.  What do you think prompted AppArmor to cause the problem?

Yes, that's sound advice.

jlp, as for package:

-- https://en.wikipedia.org/wiki/Linux_package_formats

Packages on a linux systems is usually administrated by various tools, but the more usual ones are apt-get and dpkg.

Learn more:

-- http://www.cyberciti.biz/tips/linux-debian-package-management-cheat-sheet.html

So, when I suggest "a. Find out the name of the package(s) that contains apparmor on your system.", the way to go about solving that task is to understand what to do, then figure out how to do it.

If you don't know what a package is, you search for "What is a package in linux?" The first hit on duckduckgo.com is the wikipedia link a few lines above. Now you know what a package is, and you need to find all packages associated with apparmor.

If you search a bit more, for instance on stackoverflow.com, you will find postings like:

http://stackoverflow.com/questions/12740452/ubuntu-command-to-list-the-installed-software

From there you can learn you can run a command like to view all packages. Now you could either save the output to a text file by doing and then look at list.txt for apparmor entries, or you could run a command like
It's also possible to user apt-cache:

-- http://www.howtogeek.com/howto/ubuntu/search-for-install-packages-from-the-ubuntu-command-line/?PageSpeed=noscript

Or you can go directly to Ubuntu, and search for packages:

-- http://packages.ubuntu.com/

When I search for apparmor there, I get this result:
-- http://packages.ubuntu.com/search?keywords=apparmor&searchon=names&suite=trusty§ion=all

As exact hit, I get apparmor.

This means it should be possible to uninstall it with .

Then to ensure, the system is in a "fresh" state, the easiest thing is to just reboot. I assume that apparmor is the main package for apparmor daemon, and once it's removed, bitcoind should function normally for you again.

Linux can be a bit complicated, so you need to be patient, and spend time learning.

If you do not understand a sentence posing a problem then break the problem down in sub-parts, and make sure you understand every sub-part, if there's a word you do not understand, google it. Once you understand the problem, finding a solution is much simpler. Just trying stuff blindly, seldom works.

There's plenty of people around here wanting to help you, but it's also important that you demonstrate that you've done a real effort to try solve the problem. But don't worry, problem solving skills is something that can be improved.

Let us know how this turns out, I'm sure everyone rots for you to solve the issue with bitcoind not running the external scripts for wallet and blocknotify events.

I'm not exactly sure I know what you mean by:


https://wiki.ubuntu.com/DebuggingApparmor  seems to shed some light on "package".  Based on this, I did the following:


Note:  the following from /etc/apparmor.d are folders:

abstractions
cache
disable
force-complain
local
tunables

I looked inside each of the above folders and did not see "bitcoin".  The "abstractions" folder had approximately 73 files.  There were a few files in the other folders.  Did you want me to find the package for all these as well?

Yes, I'm running Ubuntu.  I don't know if AppArmor was added with a recent update.  I read somewhere that AppArmor comes with Ubuntu.

Some more exercises:

a. Find out the name of the package(s) that contains apparmor on your system.
b. List all files associated with those packages on your system. Make a note of those.
c. Remove apparmor with something like .
d. ensure it's not running with e. verify that the files in b. is now gone.

Check if bitcoind now works as it should..

It might be that apparmour applied a default profile to applications not specifically listed with a profile. I'm about 100% sure that if you remove apparmor as dserrano5 says, everything will work just fine.

Are you running ubuntu? Could it be that apparmor was added with a recent update?

I also found this link, that might be worth reading:
http://ubuntuforums.org/showthread.php?t=1008906

I was expecting you also had the execve("/bin/true") line, possibly returning -1. The fact that it isn't there means that bitcoind isn't trying to run /bin/true, ie it's finding an error at an earlier stage.

F*ck it, UNINSTALL APPARMOR and try again!

Sorry for my slowness.  Is there anything else that I'm not reading correctly?  What part of my output is unexpected, as compared to your output?

Thanks for hanging in here to help me out.

Yes of course. My parenthesis remark is only to explain why there's more than one pair execve(sh),execve(true). Of course, waiting for one block and actually being there to stop bitcoind will result in only one block . It's only, I was doing other stuff and forgot about bitcoind so several blocks were added to the chain in the meanwhile.

Shortly after starting up bitcoind, I checked debug.log.  When I saw the runCommand error, I stopped bitcoind and ran grep.  Doesn't this explain why there is only one block in my mytrace files?  If I had let bitcoind run for a while before running grep, there would've been several more invocations of blocknotify and therefore several more occurrences of the runCommand error.  Doesn't this explain it?

Thanks for your suggestion.

Apparmor's profiles are supposed to be in the following directory.  A profile does not exist for bitcoind:


Nevertheless, I tried stopping apparmor.  As per http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/ , I did the following:


But it didn't seem to stop when I checked the status:


So, as per https://help.ubuntu.com/lts/serverguide/apparmor.html , I tried the following to stop AppArmor:


When I checked apparmor status, I got the same as before.

I tried to set bitcoind to complain mode, but it didn't work:


As per https://help.ubuntu.com/community/AppArmor , I did the following to disable AppArmor:


When I checked apparmor status, I got the same as before.

I stopped and started bitcoind:


…but I still got the runCommand error in .bitcoin/debug.log:


I have bitcoind 0.9.0.  Do I need upgrade it?  I read that version 0.10.0 causes frozen blocks.

Yes, but the output keeps being unexpected. Look at mine (there are several blocks here, not only one):


Like I and others mentioned, look into apparmor.

Is this what you are asking for?:

Try looking into AppArmour and see if it could be the culprit.  I think we're running out of Bitcoin related things. :-)

As cryptowatch.com said, perhaps it is interfering with app permissions.

Apparmor was mentioned.

Some info:
http://en.wikipedia.org/wiki/AppArmor
http://manpages.ubuntu.com/manpages/precise/en/man7/apparmor.7.html

I'm unfamiliar with apparmor, but I think these suggestions might make sense:

If you have root access, try disable or remove apparmor entirely. If block and wallet-notifications now triggers correctly, this was the culprint, and you might decide to have it uninstalled, or you might decide to have it installed but alter it's configuration to suit your needs. From what I understand from the docs, it gives access permissions on application level, rather than user level.

As for how to complete these steps, it's an exercise for you. Good luck.

Hmm, nothing there. What if you 'grep execve' or maybe just 'grep exec'?

Here it is:


I am still running bitcoind version 0.9.0.  Will this be a factor?  Do I need to upgrade to 0.10.0?

Ahem:

• Stop bitcoind.
• Configure it again to run /bin/true.
• Run bitcoind with strace, eg. 'strace -o mytrace -s 99 -ff bitcoind &'. This assumes you usually run bitcoind by simply typing 'bitcoind'. If you run it some other way (for example './bitcoind' or '/path/to/bitcoind', substitute the "bitcoind" at the end accordingly. You may have to install strace first.
• Wait for the next block. Find the error in the log.
• Stop bitcoind.
• Run bitcoind as usual (without strace) so you contribute to the network.
• Examine the mytrace.* files you'll find in the current directory. A quick way is 'grep -C 2 bin/true mytrace*'. Post your findings/the output of the grep.

Thanks for your suggestion.

Automatic had the following in his bitcoin.conf.  However, he stated that the notify commands randomly started working and randomly stopped working.


I tried the following configs in my bitcoin.conf:


and


After restarting bitcoind each time, I'm still getting the same error in .bitcoin/debug.log:


Any other suggestions that I can try?

In this thread:
https://bitcointalksearch.org/topic/m.4154179

there was an error elsewhere in the config file that was messing things up.  Perhaps this is a symptom of a different problem that is manifesting itself in an odd way?

Thanks for your help.  Do you have any ideas on what I can try next?

That has nothing to do with the result of the program. The error means that bitcoin core is unable to run /bin/true and that's the problem you have to tackle.

I got the same output as you did:


I'm still getting the following in /home/username/.bitcoin/debug.log:

I just checked my system and I have it:

/etc/apparmo
/etc/appmor.d

However, I have never seen it anywhere before you mentioned it.  Should I try changing something in it?

Just to respond once more to this, the 0 should indicate that it returned success (like /bin/true), or no error, which is a good thing.

/bin/true   should return a 0
/bin/false  should return a non-zero, typically 1  (or -1)
e.g.
As dserrano5 said, it seems that "not even a simple /bin/true works."

Try what I have as code above.  Do you get something similar to the output above?


/bin/true and /bin/false are useful for testing your logic when needed since you always know the result.

Interesting. So you had this working for a year, then it suddenly broke "a few months ago" (as per the OP) to the point that not even a simple /bin/true works. Are you using apparmor o something similar?

Thanks for your suggestions.

I tried "blocknotify=/bin/true".  I got the following in /home/username/.bitcoin/debug.log:


What does this error mean?

I'm not exactly sure what you mean by "Then copy /bin/true to your home".  I assume that you simply want me to try "blocknotify=/home//true".  I haven't tried this nor your suggestion with /tmp/foo, as I assumed that I'll get the runCommand error again.  Let me know if I should anyways or if there isn't else I can try.  Thanks.

You can try "blocknotify=/bin/true". Nothing should happen, not even an error in your log .

Then copy /bin/true to your home. "blocknotify=/home//true". Nothing should happen again, no errors either.

Another thing worth trying is "blocknotify=/bin/touch /tmp/foo" to weed out problems with spaces and that. After the next block /tmp/foo should exist, and of course no errors in the log.

I got the following:


Am I correct to assume that the permissions for /tmp are fine?

Do I need to change something in ~/myscript.sh in order to enable it to write to /tmp ?

0 should mean it returned successfully.

If dserrano5's changes didn't create files in /tmp, that is odd unless there are permission problems there.

Thanks for your keen observation.  I indeed entered both commands on one line.  Here's what I got now:


What does the 0 mean?  Should I try putting back the following into bitcoin.conf?:

Try the curl command on one line, then the echo command on a second line as it was in grue's example.  

It appears from the above that you did it all on one line.

Thanks for your suggestions.  I created myscript.sh and changed bitcoin.conf as per your suggestions.  Then I did this:


When I noticed that "blocks" (from bitcoind getinfo) increased, I checked the /tmp folder.  But this folder only had cookies.txt, but log and err do not exist.

I checked /home/myusername/.bitcoin/debug.log and saw this:


Is it possible that there is something wrong in myscript.sh that is causing the runCommand error?

When you say "call curl directly with no wrapper", are you saying to call curl directly from bitcoin.conf without using myscript.sh?

Then "blocknotify=/home//myscript.sh" in bitcoin.conf. After the next block, take a look to /tmp/log and /tmp/err.

Next step: feed the 42 from the "blocknotify=" line, use $@ in the script. Next step: call curl directly with no wrapper.

Yes, I have SSH.  Sorry, I didn't understand what you meant.  I tried your suggestion and I can tell that it executed my PHP program, as I have code in it that sends me an email.  However, SSH outputted the following:

$ curl https://mydomain.com/blocknotify.php/?blockhash=1 echo $?
curl: (6) Couldn't resolve host 'echo'

Found

The document has moved https://0/">here.

$

I don't know what that output means.  Do you?  Does this mean everything went as planned?

as in, ssh into your server (I seriously hope you have ssh access)
type:
the echo on the last line is to check the exit status. I'm suspecting that the culprit is your script failing and thus returning -1 to bitcoind when it's executed.

Are you referring to this?:


If so, then yes, that program executes when I invoke it from a browser.

If you are suggesting that I add the following command in bitcoin.conf:


...then that means I need to create a shell file.  If you are suggesting that I have "echo $?" in the shell file, then I'm not sure how I would see the output.  Doesn't the echo output go to the screen?  If the shell program is running on the server, how do I see the output?

does it run successfully in in shell? as in, run the command, then do "echo $?" to check the return status.

I have used the following successfully in bitcoin.conf for about a year:


They seemed to stop working a few months ago.  The following is in debug.log:


I changed my code in bitcoin.conf to this:


But I'm still getting the same error:


Can anyone tell me what "runCommand error: ... returned -1" means and how to fix this?