The blockchain would not get that big since these transactions would never be included in a block. For block inclusion, miners have a strong interest in setting their own anti-spam rules
bitcoind is already configurable however miners have never been very good at looking at true cost. Also Bitcoin is somewhat unique in that the miner gets the profit but the cost is shared by all users globally forever. This leads to a scenario where miners can make very poor decisions that affect all users. There is no free market, one can't boycot a particular miner so "bad" decisions are at least partially rewarded.
For example many miners would happily include spammy garbage tx as long as they pay "something". The true cost is externalized but the miner making the bad decision increases his/her bottom line.
bitcoind only needs to make it configurable.
I think this is a common misconception miners can already configure block inclusion rules using bitcoind.
the min fee (on low priority tx for nodes to RELAY the tx is only 0.0001 BTC (about 1 US cent at time of writing).
the min fee (on high priority tx) for nodes to RELAY the tx is 0 BTC.
miners are free to include any tx they wish in a block. A miner could even advise users sending spammy, bloated, free tx to send them directly to the miner to bypass min fees for RELAY. Of course no miners has an incentive to do this but if you wanted you could start mining and have users send you their garbage tx directly.
Do you see my point? I realize it would shift from "hardcoded minimum fee" to "hardcoded max bandwidth/memory consumption", but at least the latter seems more coherent with anti-DoS purposes. It would also need to be updated once in a while to reflect increases in available resources, but that happens less often than bitcoin's price movements IMHO.
I never said that FEES MUST BE USED TO FIGHT MALICIOUS USE. I simply said your initial proposal was lacking. Remember Bitcoin is like a car in motion. The developers are like mechanics forced to fix and upgrade the engine without cutting off the ignition first. When thinking of an alternative it is better to start with "how can this be abused" rather than thinking "how can this be used". The min mandatory fee is inelegent (in the software design use of the word) but it does work. It makes malicious attacks on the network non-economical.
You seem to be indicating that fee & priority should be combined. There should just be one priority and it can be based on a lot of things (coin age, UXTO set reduction, fee paid, etc). That likely is a start however some other components are necessary.
a) miners need a way to be able to see downstream fees (allows receiver to spend 0-confirm with a fee to get both tx included)
b) clients need to provide an accurate prediction for users otherwise the end result is lots of cheap users just spamming network w/ tx which never confirm (which helps nobody not even the user)
Gavin has indicated that the fee model needs to be improved however until now it hasn't been a high priority. The min-mandatory fee on low priority tx isn't perfect but we should be cautious in changing it. It does currently act as a safeguard and also prevents users from doing something foolish.