It doesn't matter when you are verifying a signed message (eg. proof of ownership), that would be a 1 time call to recover function. So that function being slow doesn't affect anything.
But when you are verifying transaction signatures, most of the times you are doing it hundreds of thousands of times (eg. a new block that contains 2000-3000 transactions or syncing your node and downloading ~600k blocks). In this case the slowness of that function is very important. And it is indeed quite slow:
2x square root (which a ModPow for secp256k1 curve with 256 bit numbers)
6x check point on curve
4x modular multiplicative inverse
6x point multiplication
2x point addition
In comparison the signature verification is:
1x modular multiplicative inverse
2x point multiplication
1x point addition
2x simple integer multiplication