It doesn't. We consider the median timestamp of the last 11 blocks, and not all of the miners.
Cheers
There is also the point about network adjusted time, which prevents the timestamp from drifting uncontrollably; nodes wouldn't accept your block if your timestamp deviates too far from the network adjusted time. Even if there were two miners, it doesn't mean that they can set their timestamp arbitrarily. The blockchain doesn't necessarily provides for an indisputable or concrete "timestamp" but rather it provides an environment where you are required to expend far more to re organize the blocks.
If I understood Maxwell's comment correctly, let's give the hypothetical that there were only two miners on the network, and that they published blocks two years into the future:
Nodes would then wait two years before updating their own records, after their own clocks matched the published blocks, and only if there weren't conflicting blocks published before then. So, there is incentive for miners to be reasonably accurate.
Would that be correct? I've not looked into the code enough yet to know myself.