If it is non-HD wallet (HD symbol crossed out) all the private keys are randomly generated and not related. The only "risk" comes from accidentally using that compromised privkey/address again in the future.
In my opinion, the safer option is to move to a new wallet... but moving all the coins to a new private key/address is definitely better than doing nothing.
You might want to be careful renaming it... I've seen posts by several users on here that have previously done this and then been unable to remember/locate the file at a later date
On windows... likely to be something like: C:\Program Files\Bitcoin\bitcoin-qt.exe -rescan
or add rescan=1 to the bitcoin.conf file in your Bitcoin data directory (where your wallet.dat is)... if you don't have a bitcoin.conf, just make one using a text editor. Just remember that once you have done the rescan, you should remove rescan option from your bitcoin.conf or it will do it EVERY time you start the program!