On my boxes this is how I do mining.
When the box fires up I VNC into it and open a terminal and open a tmux session in that terminal. I then disconnect the tmux session. I now disconnect the VNC session. Next, I ssh into the box and resume my tmux session. This tmux session is now running "in" X. I've tried exporting the display and for some reason that is flaky but this method works all the time. Next I run my cgminer.sh script which has something like this in it... I set the resolution of the X server to a very low value so that less resources are dedicated to driving the X Window system.
You won't notice any lag in this method because you are never waiting on the graphics display.
export DISPLAY=:0
export GPU_USE_SYNC_OBJECTS=1
export GPU_MAX_ALLOC_PERCENT=100
xrandr -s 800x600 # this sets the resolution of your display to a low value
cgminer