The check works flawlessly on my configs, both linux and windows. It should work.
There is something wrong somewhere but where...
vanitygen -check works with a old version, this is the only compile error I got:
g++ -DWITHGPU -m64 -Wno-write-strings -O2 -I. -I/usr/local/cuda-8.0/include -o obj/hash/sha256_sse.o -c hash/sha256_sse.cpp
/usr/local/cuda-8.0/bin/nvcc -maxrregcount=0 --ptxas-options=-v --compile --compiler-options -fPIC -ccbin g++ -m64 -O2 -I/usr/local/cuda-8.0/include -gencode=arch=compute_50,code=sm_50 -o obj/GPU/GPUEngine.o -c GPU/GPUEngine.cu
ptxas info : 0 bytes gmem, 16872 bytes cmem[3]
ptxas info : Compiling entry function '_Z9comp_keysjPtPjPmS0_' for 'sm_50'
ptxas info : Function properties for _Z9comp_keysjPtPjPmS0_
16712 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 128 registers, 360 bytes cmem[0], 116 bytes cmem[2]
ptxas info : Function properties for _Z11_GetHash160PmS_Ph
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z15_GetHash160CompPmS_Ph
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z7_ModInvPm
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z9CheckHashjPtPmS0_ijPjS1_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
vanitygen -check deosn't works with the current version, this is the compile error I got:
/usr/local/cuda-8.0/bin/nvcc -maxrregcount=0 --ptxas-options=-v --compile --compiler-options -fPIC -ccbin g++ -m64 -O2 -I/usr/local/cuda-8.0/include -gencode=arch=compute_50,code=sm_50 -o obj/GPU/GPUEngine.o -c GPU/GPUEngine.cu
ptxas info : 0 bytes gmem, 33320 bytes cmem[3]
ptxas info : Compiling entry function '_Z9comp_keysjPtPjPmS0_' for 'sm_50'
ptxas info : Function properties for _Z9comp_keysjPtPjPmS0_
32936 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 126 registers, 360 bytes cmem[0], 116 bytes cmem[2]
ptxas info : Function properties for _Z10CheckPointPjiiPtjS_S_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z11_GetHash160PmS_Ph
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z13CheckHashCompPtPmS0_ijPjS1_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z15CheckHashUncompPtPmS0_ijPjS1_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z15_GetHash160CompPmS_Ph
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z7_ModInvPm
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Function properties for _Z9CheckHashjPtPmS0_ijPjS1_
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
As you can see, in the last there is a different error: Function properties for _Z10CheckPointPjiiPtjS_S_