Could you have by accident installed 32bit windows? I know you said it's x64 but it sounds like you're pressing that 4gb limit (~3.25gb addressable)
I can confirm it is indeed x64 via the Control Panel's basic system information. I have the same distro installed on two other rigs, one of which is an Intel Bad Axe 2 seating 1 390x, 1 290x & 1 290 w/ 2 x 1 GB + 2 x 2GB where all 6GB are properly allocated and the other from vague recollection is an Asus P5b also on socket 775 with 2 x 2GB where only 3.25GB is usable as indicated in the motherboard's manual.
fo00ok
It works flawlessly with Virtual memory set to static 16GB size and all Genoil/Claymore versions