Unfortunately, Bitcoin Core stores your public keys and funds unencrypted on the computer that runs it. If that computer is regularly connected to the Internet, it is at risk to hackers - making you a target once your balance is discovered.
That information: "you possessing N number of Bitcoins" will be the basis of hackers to make you a target, specially if it's a significant amount.
It's not necessarily mean that they'll use the xpub in Bitcoin Core directly to hack your Specter wallet.
Nevertheless, take note that the xpub is not entirely safe if you've been exporting individual private keys from Specter, even prvKeys of the unfunded addresses.
(AFAIK, there's no option to do that in Specter, just like in Bitcoin Core's descriptor wallets)
Because a child private key can be used together with its parent "extended public key" to compute its pair "extended private key".
But if you haven't exported any private key and exposed it to the internet, then this shouldn't be an issue in your case.