If someone rejects a block that most of the network accepts, won't the rejecting nodes eventually give up on the rejecting branch and move to the longer branch that accepted it?
If you reject a block, then all blocks after it are also invalid from your point of view, no matter how many other people accept them. If this was not the case, then an attacker could create bitcoins out of thin air by getting more than 50% of the network's CPU power. This effect was demonstrated when the overflow bug was fixed: even though 0.3.10 nodes were in the minority, they rejected all blocks produced by old clients (which contained a fraudulent and impossible transaction for 184 billion bitcoins).
This is not quite the case for timestamp issues, but there is a similar effect. If your clock is off, you won't accept
any new blocks, as they'll all be too far in the future. You'll eventually get the blocks when they are no longer too far in the future, but you'll still be unable to generate because your view of the "latest valid block" is wrong.
The same problem would happen if someone double spends a coin, and nodes disagree on which one came in first. The majority will make a longer chain and the minority nodes will have no choice but to jump over.
You don't reject blocks for transaction timing/ordering issues.