But soft forks can only add rules, by defining parts of the protocol that were previously "anything goes" to something more limited. All the old rules are still intact and everyone counting on them is still happy.
This is NOT true.
The old rules allow me to put anything I want in coinbase but now I must put the block height as the first item. (BIP34)
The old rules allow me to format the signature in many ways but now it's heavily limited. (BIP66)
The ability to "add rules" means the ability to "add bad rules". Adding bad rules is removing "good attributes" from Bitcoin. A softfork could: a) require a centralized service to sign all blocks; b) blacklist addresses; c) burn dormant bitcoin; d) reduce the max block size to 500 bytes; and more. Are you happy with any of these?
Alternatively, with the ability to remove rules, hardfork would allow us to remove "bad rules". If someone managed to introduce bad rules with softfork, the only hope for us is to hardfork and stay away from the bad chain.
--------------------------------------------------
As a Bitcoin expert you should know this fact: full nodes not upgrading after a softfork just PRETEND they are doing full validation. They are just slightly secure than an SPV client. Miners not upgrading after a softfork are just mining fool's gold.
Having said that, I still believe softfork (with super-majority miner support) is a safer practice than hardfork because the new fork will constantly attack the old fork, and it won't create 2 sets of trading tokens.