Plenty of people have made claims that gambling sites are "rigging" rolls and provably fair is "bull"... However, I'm not currently aware of any instance of a (properly and fairly implemented[1]) "Provably Fair" system having been shown to be rigged. The sites simply can't cheat the math... much like the players can't when they're playing against odds skewed in favour of the house (House Edge).
[1] There were accusations levelled at one particular site that involved having to click a button to show the server seed etc... which is a flawed design as it could potentially be used to allow the site to know if the user was checking the seed or not and therefore only "rig" rolls when the user hadn't click the button and could not check the fairness of the roll.
The whole point of "Provably Fair" is that YOU can prove it to YOURSELF.
As you just pointed out it is possible to have a flawed design where it doesn't really allow you to prove it. But that will always be spotted very quickly when some of the customers of these gambling sites are experts in cryptography.