I have no ideas how to troubleshoot this any further.
48h later, the problem still persists. Occasionally I've seen the load on that card go even lower, down to 40-45%. It will occasionally recover by itself, but often needs a mine restart. And sometimes even that doesn't fix it. I tried several other things, like -q 2 but even that doesn't fix it.
I thought maybe it's overhead due to having X running on a headless box, bit of a waste if you ask me. I don't have much experience with Debian (I'm more of a Slackware guy) so I killed X as best I could but that brought the whole mine down unfortunately. I tried various kernels, including the latest svn of phatk (which I'm still running right now) with again no change in this behavior.
have you installed all the fixes? your symptoms sound very much like the screensaver is running, which can happen in some situations (never figured out why, but pretty much forced it dead in one of the updates).
edit - even if you have installed fixes, might also want to:
cp /opt/bamt/.xscreensaver /home/user/
and reboot. worth a shot.