Well, from another viewpoint, the 2 biggest pool operators, slush (180 Ghash/s) and Tycho (160 Ghash/s), effectively control 49% of the global network hashrate (the last 3-day average was ~700 Ghash/s). Based on historical growth, in the near future they will assuredly indirectly control more than half of the global hashrate. If there is collusion between these 2, they can in theory start doing evil things.
Of course I trust them today, but a pool duopoly (or monopoly) is not something I am comfortable with in the long term.
I've thought about this as well, and discussed it with a friend.
Consider that slush and Tycho are making money by using their mining power legitimately. If they wanted to collude to attack the network, how long would it last? A day? Probably less. Someone would notice and the forums would explode. They would both lose their pools because users would obviously not want to mine for them anymore. And they would never be trusted again.
So, they would have to profit enough in a very short period of time to make up for the lost profits of simply mining legitimately for as long as they can.
I don't think there is much profit in attacking bitcoin. Of course there are other motivations that could be considered.
That said, would I like to see more large pools show up, reducing the percentage of the network that is in the hands of so few? Yes, of course, competition is good!