The fee you need to pay depends on two things - the size of your transaction in bytes (which is not the same as the value of your transaction in bitcoin or fiat), and how many other transactions are waiting to be confirmed and what fee rates those transactions are paying.
Every time you send or receive bitcoin, you create an output. Each output is separate, even if they are on the same address. Each output contributes a number of bytes to a transaction size. So the more outputs you have, the larger your transaction. So if you have received to your Ledger wallet 0.01 BTC on 5 different occasions, your Ledger will display a balance of 0.05 BTC, but that is still split across your 5 outputs. If you want to spend that bitcoin, you have to pay for all five outputs. Conversely, you could receive 1 BTC in a single output, and so spending this single output of 1 BTC will be cheaper than spending your five outputs totaling 0.05 BTC.
The other thing to consider is all the other transactions currently waiting to be confirmed. You can get a good representation of this by using a site such as https://jochen-hoenicke.de/queue/#BTC%20(default%20mempool),8h,weight. If you pay the minimum fee of 1 sat/vbyte, then your transaction will be right down the bottom of the blue part, with all the other transaction paying a higher fee rate taking priority over yours. Conversely, if you were to pay something like 20 sats/vbyte at the moment, then your transaction would be right at the peak, and will almost certainly be included in the next block.