They can stall by not signing units or signing branches that the other witnesses have not seen.
If rogue witnesses gain a majority, they can damage (or rather kill) the network in two ways:
1. by stopping any activity and thus stalling the network like you say above
2. by posting nonserial units and thus creating ambiguity in total order that is not resolvable without a network reset (roughly equivalent to hard fork)
If they are a minority, such behavior is harmless and would rather lead to swift replacement of the misbehaving witness(es).
So if I understand it correctly, someone can try to populate his rogue witnesses, maybe slowly and patiently. If people are taken in and use his rogue witnesses, then eventually the system could be in trouble?
Yes, the system could be in trouble if the attacker manages to convince people that at least 7 his witnesses are all independent and every single witness has much more to lose (in the real world, not inside the system) than to win from attacking the system.