As I use BAMT which is 32 Bit only 2GB of 8GB are available. When I find some spare time I'll setup a 64Bit system and see if that works better for scrypt-jane
Just throw in a 64 bit system.. 2 GB are never enough for scrypt-jane at higher N. You can try though TC of 4096 or 2048 to see which one is working. Raw intensity 1024/2048/4096 at lg 3 or higher. Then you just increase the TC until it will not start anymore. Then you go higher with R in increments of w (if w is 128 just add 128 each time).
You don't need much system RAM on Linux, 256-512MB should be enough if you are running some lite distribution such as Lubuntu. Windows drivers have a bug, it queries amount of system RAM instead of VRAM.
Below are my settings for 280X, about 135khs per card.
"vectors" : "1,1,1,1",
"worksize" : "256,256,256,256",
"kernel" : "scrypt-jane,scrypt-jane,scrypt-jane,scrypt-jane",
"lookup-gap" : "0,0,0,0",
"thread-concurrency" : "16384,16384,16384,16384",
"shaders" : "0,0,0,0",
"gpu-engine" : "0-1050,0-1050,0-1050,0-1050",
"gpu-fan" : "0-85,0-85,0-85,0-85",
"gpu-memclock" : "0,0,0,0",
"gpu-memdiff" : "0,0,0,0",
"gpu-powertune" : "0,0,0,0",
"gpu-vddc" : "0.000,0.000,0.000,0.000",
"temp-cutoff" : "95,95,95,95",
"temp-overheat" : "85,85,85,85",
"temp-target" : "75,75,75,75",
"gpu-reorder" : true,
"api-mcast-port" : "4028",
"api-port" : "4028",
"auto-fan" : true,
"expiry" : "120",
"gpu-dyninterval" : "7",
"gpu-platform" : "0",
"gpu-threads" : "2",
"gpu-reorder" : true,
"log" : "5",
"no-pool-disable" : true,
"queue" : "1",
"scan-time" : "60",
"scrypt" : true,
"scrypt-jane" : true,
"temp-hysteresis" : "3",
"shares" : "0",
"kernel-path" : "/usr/local/bin"
Are you getting no hardware errors with that ?