Yes, the sender decides the fee.
Exchanges have a fixed fee which is higher than the actual transaction fee (keep in mind, they usually batch transaction and therefore pay less per withdrawal transaction than when sending them all one by one) because of multiple reasons.
First, you are not just paying for a single transaction. Your deposit transaction has to be merged with others and moved to the cold wallet. The hot wallet has to be topped up from the cold wallet regularly and in the end the actual withdrawal transaction has a fee.
Further, you are paying for the "work" behind these transaction. You can't just calculate the transaction fee and say that's what it should cost. They have algorithms checking stuff, doing sanity checks, making sure there are no bugs, vulnerabilities etc.
This all costs money (development costs, maintenance, energy, servers, etc..) which is priced into the withdrawal fees.