If you have a solid working knowledge on cryptography—public/privkey encryption in general—then you don't have to ask these thinga but here it goes anyway.
There is more to this then the knowledge of how encryption works. It's also important to understand the mechanics, how it's been implemented. Yes, it's obvious that an exchange, which offers a service would or could hold my keys which to me is not a good thing. If I alone have my keys, how can the keys be backed up, can that backup be imported by some other agent. The keys are only a small part of the issue. If a user is to pick local wallet software how many choices will they have to research? Just pick one and it could send my keys anywhere even though an open source entity would have a difficult time pulling that one off. It's an issue of trust and with software that always comes with a degree of unknown risk. Gold can be in my hand a private key to a bitcoin is a bit less tangible.
Knowing how the keys are used is rather obvious as you say but, when I use software which manages a private key whether that software is on my personal hardware or at a distant server, I don't know how it is being handled. As a consumer all I can do it try to find a highly regarded open source product where mass consensus is that it's safe in how my keys are protected. Since I as an ordinary user don't have time to plow through source code, I mush have some degree of trust in the process which presumes transparency.
Remember that I have just started digging into to all this and until I have a reasonable level assurance that the transparency is what people say it is, I will and should have reservations. This is a huge system and it will take time to determine that it is what people clam it to be, a new currency which will change everything. To a new comer, those are big claims which I have heard big claims about many things many times before. 1 in 1000 turns out to be the big game changer.