I agree with you that centralized exchanges have to go. Even though I never have had any problems with them, I still feel that they're undermining the essence of decentralization. How can a coin be truly decentralized when it can only be traded (at a large scale) one an exchange that is owned by a corporation.
Lucky you, not having problems with centralized exchanges, personally, I have been scammed by Bittrex a couple of weeks ago and just like thousands of other people, in the excuse of KYC or something (never been officially declared).
I was very pleased when I read up in the concept of atomic swaps but seemingly you don't. What are the downsides of atomic swaps that we won't be able to fix in the future?
Sure, atomic swap technology is gradually succeeding and we have news in this field almost on a daily basis. It is great, we need it, and there is no downside here to have a protocol to do some fascinating distributed ledger transaction processing, but, unlike some proponents of this technology, I'm not sure it can solve the decentralized exchange problem by its own even once it is available, full fledged and open source.
The main challenge (as I have mentioned before) is scalability, and it is a snow ball situation:
Atomic swap, if going to be used directly as an infrastructure to facilitate peer-to-peer coin exchange in the very first steps imposes large storage and hardware requirements to both peers. As @d5000 and @687-2 have discussed about in previous posts here. Now it is just for beginning. Suppose in some point we have overcome this challenge and got to run our nodes, supporting AS. Then what? It will escalate transactions and we will have more scalability problem to overcome and so on. So, obviously we have a limit here, solving the problem raises the scale of the problem and no one will be satisfied ever.
Another major concern is that Atomic swap is just an infrastructure, it can not be projected directly to the problem domain. Trade is a complicated socioeconomic behavior, one can not address it with a device. In the exchange case you need a marketplace, an order book, a matching rule, ...