Well for example, let's say the server disappears entirely...
The user sends a recovery request to the other pool members, who verify the situation and then vote to recover your coins.
This contingency is actually the whole reason for having voting pools in the first place. The server would be unwise to do such things however, as he cannot steal the coins anyway (they are in the pool.) Plus, he'd instantly lose access to any future transaction fees, as well as losing access to all the rest of his legitimate share of the coins in the pool.
Your next question might be: what happens if the entire pool disappears? Answer: you're screwed.
This is why the ultimate distribution of risk must occur inside the wallet itself (for example, allocating your funds across multiple pools.)
We cannot eliminate the existence of risk, we can only devise better and better mechanisms for distributing it in a user-centric way. In fact, this philosophy underpins Bitcoin itself.
This concept of a provider-independent distribution of risk is borrowed from Tahoe-LAFS. Kudos to Zooko!