.....
Am I missing something? (Hopefully!)
Both of these things can't be true at the same time, if more mining power is leaving than joining then mining gets less difficult and less expensive.
It depends on what portion of the mining payout is block reward and fees. If fees are a very small part then yes the halfing has the same effect as difficulty doubling - miners will make half of what they did before. This already happened once, and yet mining hashrate continued to grow - why? because bitcoin is still growing, more users == higher price == more incentive to mine.
One of the common failures when analysing bitcoin is to consider future events but assume that the state of the ecosystem will be the same as today. If Bitcoin keeps growing, when future halvings occur the price per coin will be much higher, and the transaction fees will be a bigger part of the mining reward - some day they will exceed the block reward.