It is hypocritical and dishonest at best, and dangerous at worst. If no one is allowed to build on your code or use your code for anything, then you are going to have far fewer people looking at it, examining it, testing it, using it. As you say, few people can actually interrogate the code themselves, and most users rely on independent developers or power users examining the code of open source projects on their behalf. If you aren't actually allowed to do anything with the code, then there is far less incentive to spend your time going through it.
No one can prevent you from looking at the code and testing it for security vulnerabilities. It's public, go ahead. But you can't use it as a base to build your own software. Whether the code is open-source or not and someone finds bugs or vulnerabilities in it, you can only do one thing. You open an issue about it on GitHub and inform the team. It's the devs who need to patch it up, change it, or get rid of the faulty code. You might say, the software is open-source, I can do it myself. In that case we are going back to the verifiability dilemma. The most important thing is that the necessary code is public so you can go through it and change it according to your needs. In case of the Coldcard, it's equally public as Trezor or Passport. nvK doesn't know what is running on your local machine. 
