What you are doing is interpreting the pubkey as a string. However a pubkey is not a string but rather a blob of binary data that is 33 bytes long. What you want to do is to convert your hex string into a vector of unsigned char first and then pass that to the CPubKey constructor.
It would be better for you to just initialize an array with your pubkey instead of performing a string conversion.
Edit:
Your code should look something like this:
unsigned char pk[] = {0x03, 0x96, 0xf8, 0x78, 0x1a, 0x49, 0x00, 0x37, 0x2a, 0x5d, 0x72, 0xd8, 0x47, 0x18, 0xd1, 0x46, 0x17, 0x0d, 0x59, 0x83, 0xe6, 0x7d, 0xff, 0x8b, 0x4a, 0x28, 0xfe, 0xf8, 0x06, 0x90, 0xc0, 0x97, 0x67};
std::vector vec(pk, pk+ sizeof(pk));
CPubKey pubkey(vec);
if (pubkey.IsValid()) {
cout << "valid" << endl;
} else {
cout << "invalid" << endl;
}