What exactly does this file contain? If it's true that it contains unencrypted private keys, then it is quite dangerous to leave it lying around.
If you don't delete the file, it will remain there.
What happens is that when you encrypt your wallet, a new encrypted version of the wallet is written to wallet.dat.rewrite. Then the old one is closed, and the rewrite is renamed to wallet.dat. If for any reason that rename fails, the rewrite file stays there. This file does not contain unencrypted private keys, ever.