The crypto is not exceptionally slow, but the additional packets are. A full TLS handshake requires at least four additional packets. Also, some browsers will delay the connection until they've performed an OCSP check on the certificate, which can alone take up to a half second. All of this can add up to seconds of additional delay.
That's only for the first access. After that, the session can be cached. Also, there has been a lot of work (by Google, for example) in removing the extra roundtrip which is in newer browsers and webservers. See how fast gmail.com is *with* HTTPS?It really is a non-issue these days. Just use HTTPS. If it's noticably slower you're using old broken software (either browser or webserver).
And even with a slight delay, the added security is worth it.
