but by using my ledger nano's xpub in my server to receive deposit, which mean there will be 10,000 addresses too, sending from here to a hot wallet for example will cost me huge fees too right?
*anyway is that how exchangers like binance do? channel all deposit to cold wallet first and sign for hot wallet later?
It's confusing when you're just starting out, but in reality you shouldn't focus on addresses... It doesn't matter if you derive 10.000 addresses from an xpub or just 50.
What's important is the amount of unspent outputs that are funding the addresses generated by your wallet. As soon as you want to spend those unspent output's you'll have to create a new transaction. The fee you'll have to pay to convince a miner to put your transaction in the block he/she is trying to solve depends on the size of the transaction (not the value).
Why? Because the size of the block is limited. A miner doesn't care if the value of the transaction is huge or very small, he just wants to optimize the sum of the fees of all transactions he can fit into the limited blockspace.
If you want your transaction confirmed, you'll have to make sure your fee (in satoshi per byte of transaction data) is within the top 1 Mb of transactions in the mempool... The fee needed for a fast confirmation fluctuates over time, so at times of low feerates, it makes sense to combine unspent outputs, so the fee you'll have to pay at times of high feerates is lower...
Since segwit, the blocks can be up to 4Mb in size, but the last 3Mb can only be filled with witness data, so don't worry about this untill you've mastered the rest of the protocol first (it'll only make things more confusing if you start reading about this first).
The size of your transaction increases with the number of unspent outputs you use an input and the number of new unspent outputs that'll be generated by your transaction.
So, in the end, things like hot wallet, cold wallet, number of addresses,... It doesn't matter... Your fee will depend on the type of wallet, the number of inputs and the number of outputs. It's a good idear not to create unnecessary transactions, and it might also be a good idear not to receive dust payments... If you're going to receive dust, try out the lightning network, switch to an altcoin or try to sell credit to your users in bigger chunks instead of accepting micro-payments.
Here's a neat tool to emulate the fee you'll have to pay when spending your unspent outputs:
https://coinb.in/#fees