Public -> Private is basically impossible, so nothing to worry about there, if someone gets your public key.
Private -> Private is obviously possible though. More than likely they would just loop through the keys.
As a side note, make sure whatever you're using to sign transactions is legit. If you sign your transactions using malicious code that specifies the nonce to use for signing, then yeah, they'll be able to obtain your private key after gathering some signatures from the blockchain.