The fix for the thread priority level on linux is available in the 0.3.1 release candidate here:
https://bitcointalksearch.org/topic/m.3198
Topic: Runaway CPU usage for 64bit BitCoin (Linux Client) (Read 14949 times)
Could you make it possible to reenable that feature with a startup switch or even compile flag? I like it a lot, and I don't have this issue.
I like it alot too, wished it work for me 
OK, the undocumented switch "-minimizetotray" which re-enables the option.
I uploaded the change to SVN.
I uploaded the change to SVN.
Could you make it possible to reenable that feature with a startup switch or even compile flag? I like it a lot, and I don't have this issue.
Ok, easy to replicate (well for me anyway)
64bit Client Linux
Having either the "minimize to tray instead of taskbar" or "minimize on close" options checked will cause the bug.
First thing you notice (as per the first screen-shot), the tasktray is being filled up with invisible icons "spaces". If you hover the mouse over them, they all say "Bitcoin (not connected" but the one that does have an Bit Coin icon, just says "Bitcoin" and that's what you can bring the window back up with.
If you just have "minimize on close" checked, it still sends it to task tray causing the bugs.
After a while, all your CPU will be consumed by the X server, even after you close Bitcoin, I think some process stay stuck in memory, not sure.
The 32bit Client for Linux doesn't have this exact issue. I will still make at least one invisible Bitcoin icon, but doesn't tear up the CPU later. So the 32 bit client sort-a has the bug, but not in a way to bring the system down after a short while.
64bit Client Linux
Having either the "minimize to tray instead of taskbar" or "minimize on close" options checked will cause the bug.
First thing you notice (as per the first screen-shot), the tasktray is being filled up with invisible icons "spaces". If you hover the mouse over them, they all say "Bitcoin (not connected" but the one that does have an Bit Coin icon, just says "Bitcoin" and that's what you can bring the window back up with.
If you just have "minimize on close" checked, it still sends it to task tray causing the bugs.
After a while, all your CPU will be consumed by the X server, even after you close Bitcoin, I think some process stay stuck in memory, not sure.
The 32bit Client for Linux doesn't have this exact issue. I will still make at least one invisible Bitcoin icon, but doesn't tear up the CPU later. So the 32 bit client sort-a has the bug, but not in a way to bring the system down after a short while.
After it initially tries incorrectly to set itself to the lowest priority, the generate thread only changes its priority again temporarily when it finds a block. When you've found a block, you should want it to hurry up and broadcast it as soon a possible before someone else finds one and makes yours invalid. The generate thread only changes to higher priority for less than a second every few days.
There should be a 0.3.1 release for this soon. There are a few other issues we need to look at fixing in 0.3.1 before making a release.
It only seems to happen ifThere should be a 0.3.1 release for this soon. There are a few other issues we need to look at fixing in 0.3.1 before making a release.
On a side note, I've tracked down the other GUI issue.
The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
That's interesting. I know the minimize to tray on Ubuntu is very clunky, but I didn't know it had a CPU peg problem too. Anyone else able to reproduce this problem? We had this feature disabled on Linux before, but then it seemed better to have the imperfect UI than to lose the feature entirely. I'm thinking we should disable it again on Linux.The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
A) you minimize to tray, later bring it back up, minimize again, bring it back up. It starts stacking tray icons out (but they are blank, but still labeled as BitCoin)
B) sometimes the program does it randomly
I'll turn it back on with mine to make a screen shot, doesn't take long for it to happen, hehe.
After it initially tries incorrectly to set itself to the lowest priority, the generate thread only changes its priority again temporarily when it finds a block. When you've found a block, you should want it to hurry up and broadcast it as soon a possible before someone else finds one and makes yours invalid. The generate thread only changes to higher priority for less than a second every few days.
There should be a 0.3.1 release for this soon. There are a few other issues we need to look at fixing in 0.3.1 before making a release.
There should be a 0.3.1 release for this soon. There are a few other issues we need to look at fixing in 0.3.1 before making a release.
On a side note, I've tracked down the other GUI issue.
The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
That's interesting. I know the minimize to tray on Ubuntu is very clunky, but I didn't know it had a CPU peg problem too. Anyone else able to reproduce this problem? We had this feature disabled on Linux before, but then it seemed better to have the imperfect UI than to lose the feature entirely. I'm thinking we should disable it again on Linux. The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
One fairly simple workaround is to run bitcoind as a user other than root. Since bitcoin does not require root permissions it will run just fine and it won't be able to set a negative niceness (only root can do that). That will ensure that the highest priority bitcoind can use is 0 (normal).
hardcoding to 19 should work, also you could just change PRIO_MIN to PRIO_MAX which would basically do the same thing, (technically PRIO_MAX is defined as 20 in the linux kernel source)
I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)
I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)
Good point. I just wanted to test if it made a difference (I suspect it will). But after reading the other topics, seems the fix was already checked into the SVN, just not in time for the release so there is no sense in me trying a fix that which is already fixed code wise, hehe.
I agree though, it just needs to run at the lowest prioirty at *all* times, I mean it's suppose to run off of idle CPU and a nice level of 2 isn't far from *shared* CPU cycles than *idle* CPU cycles in my opinion.
hardcoding to 19 should work, also you could just change PRIO_MIN to PRIO_MAX which would basically do the same thing, (technically PRIO_MAX is defined as 20 in the linux kernel source)
I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)
I agree that there authors saw some practicality in changing the priority depending on what the function is doing, but personally Id rather just have it not touch the priority at all, as the program will work just fine sitting at lowest priority all day.. I dont want it ever stealing any priority from any process regardless of what the thread is doing.. (maybe it would be a good suggestion to make this a command line parameter to disable/enable automatic priority adjustment - Im sure theres many who would prefer to keep the automatic priority scheduling in place, but also plenty that don't want any part of it)
from what I saw, there was an error in util.h basically the THREAD_PRIORITY_LOWEST was defined as PRIO_MIN.. the intention was to switch to the "lowest" priority, but in the linux kernel source in resourch.h PRIO_MIN is actually the lowest NUMBER (-20), meaning it actually changed to the HIGHEST priority.. and then after that function is done, it will change priority again (to either 0 or 2, not as bad as -20) but either way it is changing it, so either way no matter what you nice/renice the threads to, it will switch to diff priority levels automatically, with those particular threads that switch priorities never being the most unobtrusive at 19..
Yeah, I finally found it as well (I knew I shouldn't have peaked in the code, LOL). I'm going to try manually setting it to hard code 19 and see if that makes a difference after a compile. If so, it might be a more elegant solution that removing the function all together since it appears it's trying to balance out when to use more CPU depending on the system load.
from what I saw, there was an error in util.h basically the THREAD_PRIORITY_LOWEST was defined as PRIO_MIN.. the intention was to switch to the "lowest" priority, but in the linux kernel source in resourch.h PRIO_MIN is actually the lowest NUMBER (-20), meaning it actually changed to the HIGHEST priority.. and then after that function is done, it will change priority again (to either 0 or 2, not as bad as -20) but either way it is changing it, so either way no matter what you nice/renice the threads to, it will switch to diff priority levels automatically, with those particular threads that switch priorities never being the most unobtrusive at 19..
Its been stated in the latest source code, this min/max is fixed, so if you dl the latest source tree and compile the min/max reversal should be fixed... personally I didnt care of having any of the threads change priority on its own at all, so I just removed the function alltogether..
Its been stated in the latest source code, this min/max is fixed, so if you dl the latest source tree and compile the min/max reversal should be fixed... personally I didnt care of having any of the threads change priority on its own at all, so I just removed the function alltogether..
yep, that 2 is exactly what im talking about..
wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....
Ah I see now. Since you've already dove into the code and disabled the function, is there any reason why the function in question wouldn't just set everything to lowest priority by default? Why would it need to change it around? I'm afraid to dive in myself, otherwise I'll spend days fixing it, hehe. wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....
yep, that 2 is exactly what im talking about..
wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....
wait a little longer and another thread might reset too.. after my change all my threads all stay 19 forever....
knightmb - just curious, if you check not just the main process, but all the thread processes (ps -eflL | grep bitcoin - need the L, or else it will just show the main process), do you see the other thread processes change the priority after a little while ? When I change the priority or renice, it will just stick for the main process, but some of threads will change priority everytime it goes to the generate coin function..
Feedback from the ps -elfL | grep bitcoin command
Code:
[knightmb@KnightMB ~]$ ps -eflL | grep bitcoin
0 S knightmb 11200 6781 11200 0 10 99 19 - 114169 poll_s 17:17 ? 00:00:02 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11202 0 10 99 19 - 114169 hrtime 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11203 0 10 99 19 - 114169 hrtime 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11204 0 10 99 19 - 114169 hrtime 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11212 0 10 99 19 - 114169 sk_wai 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200 6781 11213 0 10 99 19 - 114169 - 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11214 0 10 99 19 - 114169 hrtime 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 S knightmb 11200 6781 11215 0 10 82 2 - 114169 hrtime 17:17 ? 00:00:00 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200 6781 11216 86 10 99 19 - 114169 - 17:17 ? 00:40:37 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
1 R knightmb 11200 6781 11219 85 10 99 19 - 114169 - 17:17 ? 00:40:11 /home/knightmb/bin/bitcoin-0.3.0/bin/64/bitcoin
0 R knightmb 13291 12282 13291 0 1 80 0 - 1842 - 18:04 pts/1 00:00:00 grep --color bitcoin
Anything in that look out of place? I did notice one that was nice to "2" instead of "19"
[edit] I see now from the other topic, I'm running the client, the other was the server daemon, might be a bug with just the daemon, not sure though.
knightmb - just curious, if you check not just the main process, but all the thread processes (ps -eflL | grep bitcoin - need the L, or else it will just show the main process), do you see the other thread processes change the priority after a little while ? When I change the priority or renice, it will just stick for the main process, but some of threads will change priority everytime it goes to the generate coin function..
there may be some confusion in thinking that nice is working in that if people dont use -L to list all threads, they just see the PID of the main process, and think that the nice/renice has worked, when in fact the 2 subthreads that are maxing the CPU are the ones that are changing the priority back..
I myself just ripped out the whole priority changing function, so that whatever priority I set sticks permanently..
my post about ripping out priority change function is in : https://bitcointalksearch.org/topic/bitcoin-auto-renice-ing-72
also shows the whole process tree.. in that code pasting, the main process would stay to whatever you nice/renice to, but 2 of the threads would change back, until i ripped out the function
also https://bitcointalksearch.org/topic/bitcoind-possibly-process-priority-bug-285 describes the bug with bitcoin changing the threads to use the highest priority instead of lowest
there may be some confusion in thinking that nice is working in that if people dont use -L to list all threads, they just see the PID of the main process, and think that the nice/renice has worked, when in fact the 2 subthreads that are maxing the CPU are the ones that are changing the priority back..
I myself just ripped out the whole priority changing function, so that whatever priority I set sticks permanently..
my post about ripping out priority change function is in : https://bitcointalksearch.org/topic/bitcoin-auto-renice-ing-72
also shows the whole process tree.. in that code pasting, the main process would stay to whatever you nice/renice to, but 2 of the threads would change back, until i ripped out the function
also https://bitcointalksearch.org/topic/bitcoind-possibly-process-priority-bug-285 describes the bug with bitcoin changing the threads to use the highest priority instead of lowest
On a side note, I've tracked down the other GUI issue.
The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
The "minimize to tray instead of taskbar" is what was eating up all the CPU on my system. After I turned this off, the issue was resolved with Runaway CPU.
This only seems to affect the 64 bit Client, as the 32 bit Clients I have don't seem to be affected by this.
I did notice on the 64 bit Client, what happens is, it spawns multiple "tray" icons until X server finally kills over, so I guess I should submit that as a bug to somewhere?
It automatically raises it, I don't think nice 20 ./bitcoin will help to be honest.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.
I wondered about that, but I've never noticed any issues on a dual-core 64bit system.
I don't have any problems on my quad core system here either!That may be the issue because you can't "Nice 20" anything, 19 is the max
So far, I've "Nice 19" 3 clients (one 64 bit and two 32 bit) for the last hour and they haven't changed the Nice level on me yet.
I figured since the process is started at that level, it would remain for all child processes spawned afterwards. I could be wrong though, I'll check again in another few hours to see.
It automatically raises it, I don't think nice 20 ./bitcoin will help to be honest.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.
I think some people have set higher priorities for themselves in /etc/security/limits.conf, and when -20 isn't available as a nice level, a lower priority is tried.
So, I think the dirty hack of editing /etc/security/limits.conf might fix it.
I wondered about that, but I've never noticed any issues on a dual-core 64bit system.
I don't have any problems on my quad core system here either! There has been a problem in settings the "nice" level on Linux.
A programmer who wasn't familiar with Unix/Linux used the highest priority (-20) instead of the lowest priority (20)!
So, we have a bug.
Yeah, that always gets me too. You would think a negative number would mean less and positive number means more, but it turns out "Nice" works the opposite of that, you want it to be more "nice" so you want a higher number, LOL.A programmer who wasn't familiar with Unix/Linux used the highest priority (-20) instead of the lowest priority (20)!
So, we have a bug.
I checked the 32bit Linux clients, they run a Nice level of 0, so I guess it's running equal time with everything else.
On the Windows clients, Bit Coin runs at "normal" priority as well.
It's easy to start the process in low priority via command line for both of them, so for now I can do that to manually control their priority
Jump to: