It is the biggest one. That and the fact that SHA256 is very well tested and known to be secure.
Having ASICs mine is one thing, but having a hashing algorithm that is insecure is a complete chaos.
ASICs happen for every algorithm, but there are currencies like Monero that hard fork every time they have a doubt that ASICs are developed.
This is exactly why it never forked and it won't for a very long time. First you need for the almost entire community to agree that fork needs to happen.
Then you need a vast majority of the community to agree to which algorithm we should change.
And after all of that being discussed for years (probably decades based on how much time we needed to simply increase a block size) we would already have some company create an ASIC for the new algorithm.
It is not a simple problem and it doesn't seem to be absolutely necessary.
ASICs do hurt decentralization, but it is not widely established how much they really hurt it.
Anyone can buy ASICs and multiple companies can develop them.
And we still have big mining centers and pools in other cryptocurrencies that don't have ASICs for their algorithms.
And we still have practically only two companies developing hardware used to mine these altcoins.
Centralization in mining is not just an ASIC problem, it is a bit more complicated than that.