https://bitcoin.org/en/release/v0.9.0
What's considered insanely high tx fee ? is there a formula per tx or something ?
The code checks to see if the fee in the transaction is larger than 10,000 X nMinRelayTxFee.
The default setting of nMinRelayTxFee is 10 µBTC, but I believe it can be changed in bitcoin.conf
Therefore, with default settings, any transaction with a fee larger than 0.1 BTC will be rejected by sendrawtransaction.
(assuming I've got my math correct, the actual setting in the code for nMinRelayTxFee is 1000 satoshis).
If you really want to pay a fee larger than 0.1 BTC, I think there's an override parameter you can send along with sendrawtransaction.
In originall bitcoind client fee can be set by "settxfee" command. Whole API here https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list .
Transactions with fee under nMinRelayTxFee can be rejected by network but usually it just takes longer to process (and even it gets stuck it can be double spended anyway).