There's no way to prevent this. That's how Bitcoin works. "Addresses" are actually just an abstraction; they don't really exist the way our brain thinks about them. An address is just part of the output.
Your problem is that you are receiving a huge number of UTXOs. When you try to use all those UTXOs to send a transaction, it takes significant block space, hence the high fees. Now that compressed keys are the norm, transaction size is generally calculated as follows:
So a transaction with 5 inputs and 1 output will be in the range of ~779 bytes, as compared to a 1-input-1-output transaction which usually amounts to ~191 bytes. Transactions compete for block space via the fee market. At 200 satoshis/byte, the first transaction costs 0.001558 BTC. The latter transaction costs 0.000382 BTC. It doesn't matter how how many addresses are involved in the larger transaction. All that matters is the number of inputs and outputs.