Read the coinbase sig.
Alas, bitcoin rules (which means all mining) sets limits on the value it can have as stated above.
As I regularly state when the subject of the 'block timestamp' comes up, the stamp is not 'the time the block was mined'
It is either when the pool sent the work to the miner or what it was adjusted to by mining code (or the pool).
But adjusting the timestamp is no longer required with stratum, and all pools other than those setup by n00bs who don't know what they are doing, use stratum.
Stratum was designed for this reason, to put extra nonces in the coinbase sig, not use the timestamp.
This usually leads to the block timestamp being 'close' to when the block was mined - generally within X seconds of it, unless the miner or the pool is screwing with it.
That X seconds depends on how often the pool sends work to the miner, e.g. on my pool that's every 30 seconds, but other pools that may be more.
Some may screw with it, but in general for most blocks, the block timestamp will be within that X seconds of when the block was found.