The problem is that you have many small amounts sent to your blockchain.info wallet.
The fee is based on the size of the transaction (in bytes) and not based on the value of the transaction. The size of a transaction is based on the number of transactions that sent the bitcoins to you. That can be confusing, so let me give you an example.
Let's say that you that you received 0,1 BTC in one transaction and now you want to send it out. You received the bitcoins in a single transaction, so your transaction is going to be small because it only has to reference a single prior transaction.
On the other hand, suppose you received your 0,1 BTC in 100 transactions of 0,001 BTC each. The transaction sending your 0,1 BTC has to reference 100 prior transactions, and it will be very large.
In more specific terms, each reference to a prior transaction (called an "input) adds about 180 bytes to the size of the transaction. Right now, earn.com estimates that you must pay 0,00000350 per byte to have your transaction processed in one hour. That means that it costs 0,00063000 BTC for each input. If you have 100 inputs, you will have to pay around 0,063 BTC for the transaction.
It also means that any input less than 0,00063 BTC is not spendable because it costs more to send it than it is worth.
I hope that explains everything.
If you are not in a hurry and you can set a custom fee, the solution is to set the fee to 10 satoshis/byte and hope that the transaction is processed sometime over the next few days. That should lower the cost to about $40.
Thanks for the explanation, you're right, that was the problem, I have received many small payments on the same address. I was thinking about setting the custom fee, but I'm afraid of my transaction getting stuck, never getting confirmed and I end losing my bitcoins.. Is that possible?