In the extreme, yes. But if the powerful miners leave after a few thousand shares, the less powerful miners may find it profitable to stay, because they will get a bigger share of the rest. The problem, for small miners, is that when you're up against people that have 1000x your computing power, they will find blocks before you even contributed a share, and you will gain nothing.
No they won't. The calculation they ought look at is "what is the expected value of a new share here?" this will be less at a pool with lots of shares owed to powerful miners who have left than at a fresh pool.
Perhaps I'm missing something though because I don't see on any margin any way to interpret someone leaving your pool as a positive. All that happens is that you get a larger share less often. If that is what you want then choose a pool with the least people, mine by yourself. If there is some optimal pool size then what is it? When is bigger not better?