With the ledger HW.1 hardware wallet there is no such option, so you cannot set it manually. However I think it's worth whatever the fee is(in high variant) as long as we receive a fast confirmation like 10-30 minutes max.
Your not getting my point, yes i understand that a high fee gets a high confirmation, but what about 15 years from now when i want to move my long term storage and the fee for a slow confirmation is very expensive, set by who ever i would like to be able to set a custom fee, that is what i am saying, you may think its worth it, others may not.
You could always set up the Ledger in another wallet such as Electrum which supports custom fees. I don't have enough knowledge of the Ledger wallet to know if it supports custom fees, but personally I'd just use it with Electrum.