Foxpup has it. There is no single fee. The mandatory fees are calculated by the client based on the transaction inputs. The number of inputs (and thus tx size not to be confused with value) and the age of the coins determine the mandatory fee.
The reason you are getting hit with higher fees is your transactions are seen by the network as the most dangerous. They are very large (in terms of bytes) and very low value (in terms of BTC).
A rule of thumb is
One Bitcoin Day to avoid mandatory fees. If you receive 1 BTC and wait 1 day it will be old enough to avoid being charged the mandatory fee (of course you can still add an optional fee to increase processing). If you are spending 10 BTC input the time would be 1/10th of a day (~2.4 hours). If you are spending a 0.1 BTC input you need to wait 10 days for the input to be old enough to avoid fees.
Note: The the
One Bitcoin Day is only an approximation. In reality priority is calculated based on blocks so if you want to use the real formula it is:
priority = sum(input_value_in_base_units * input_age)/size_in_bytes
value is in satoshis (0.00000001 BTC)
age is in blocks
priority must be >= 57,600,000 to be considered high priority.
low priority transactions must pay a mandatory fee (spam prevention & DDOS protection)
mandatory fee is 0.0005 per KB (round up).
Transactions are exempt from fee calculation both of the following are true
a) the tx is less than 10KB
b) ALL (including change) outputs are >= 0.01 BTC
Or you can approximate as
One Bitcoin Day.