16 validators?
We don't have data that scales to thousands of validators yet, but there's really no reason you would need so many. Adding more validators doesn't significantly affect the transaction rate. But it does logarithmically increase the latency (the time between when you submit a transaction and when you know for sure that it's going to be part of the ledger).
I also want to mention that it is an internal benchmark setting. That is far away in comparison from a production setting.
The differences are not that significant. It really only depends significantly on two factors: how many digital signatures can all the CPU cores together verify and how fast can a single core process the inherently serial process of applying the transactions one at a time. Network latency affects transaction latency but not transaction throughput.
How do we know those 16 validators are not colluding?
The way you know something came from a validator is that it was signed by that validator's public key. So if the validators collude, you'll have the signed proof.
Also what will prevent validators from colluding in a real live setting? What is their incentive?
If they collude, there will be positive proof that they did so. Assuming the other participants are rational, they'll just stop listening to those validators.
If there's literally nobody who has any incentive to keep the network reliable, then it doesn't matter if the network is reliable or not. If there are people who have such an incentive, they'll figure out who else has that incentive and those will be the validators they'll use.
Remember, there's no way for a validator to say "I created these funds out of nowhere" or even "unvalidate this transaction the network previously validated". Fundamentally, validators just put transactions in an agreed order so that double spends can be identified. (If two transactions fully conflict, the one the validators place first wins.)
Validators could collude to do two key things:
1. They could reject transactions they don't like for some reason. Maybe a court orders them to. Maybe they just don't like someone. Maybe they want to demand specifoc people pay them high fees.
2. Enough validators refusing to agree can stall the network.
Both of these are essentially denial of service attacks, so colluding validators can DoS the network. But if everyone else doesn't like that, they can change validators. The trick is to find people who want a reliable network and make them the validators -- if nobody wants a reliable network, then it doesn't matter if you don't have one.
Also, validators vote to decide the future of the network through Ripple's amendment process. So people who want control over the network's destiny -- anyone who cares about the network's evolution -- will want to remain a validator. Violating trust will cause them to lose that control.
The cost of running a validator is negligible. Non-validator nodes do all the same work as validators, they just don't produce or send signed statements of what they saw, proposed, and agreed to.
