The x64 miner comes with AVX instructions (available since W7 SP1), that's what makes the difference, and a big one.
I did a double check to see what the requirements for AVX were, and from what I found it looks like it's an extension to x86 (so not required to be on x64). The windows optimized miner comes with an x86 AVX enhanced executable, but I can't vouch for it as I haven't tried that one, since all my CPU's that support AVX are running x64 Windows.
True, I didn't notice that the x86 version also came with AVX and wrongly assumed it was only for x64 processors.