and the players would have no idea what would others bet (In Nash equilibrium, the other player must know the strategy of the other).
The high level premise of your AI, is that it's going to try make players lose X% of the amount they bet. That itself is leaks a sufficient amount of information that it would make most forms of gambling, impractical (e.g. I know if have been unlucky, I will now be unnaturally lucky. If I have been lucky, I will now be unnaturally unlucky etc.)
It's possible to constrain the gambling game in such a way it won't be exploited, I guess (i.e. only fixed size bets, no hi/lo choice).
And this is already somewhat implemented on bustabit. Where the program itself will decide to maintain a profit for the website.
Hm. Not to be crass, but you should look at how existing stuff works before trying to design a different system. Bustabit and, in fact every single provably fair casino I'm aware of does *not* work like this, on the contrary, the next 10M games have been
defined and set in stone, and regardless of how much players have won or lost the exact set of games will continue to play in order. If the casino loses 200 BTC on night, it will *not* alter the game in any shape or form to "recover" the loss etc.