Its safe as long as you do not disclose a single private key.
I remember watching a podcast with Andreas Antonopoulos and he mentioned how sharing a private public key might not be that safe, so I have my doubts. Excuse me while I find the podcast.
Edit: I finally found it
https://www.youtube.com/watch?v=YPvsTe5bjGs
You should maybe listen to it again.
He clearly stated that a public key or pub key could be used in a trustless environment, the example he used was on a public server where there was no trust. So there is no risk in sharing such a key. The risk he spoke about was if somebody should make use of a brain wallet type passphrase instead of the seed as we currently use. He explained that it was vital that the seed be protected as that would give access to the main xpub key and thus to the whole wallet, sub wallets, sub accounts and because of that the risk was bigger as every single sub wallet and or account would be compromised just through the leak of the single seed.
Where a xpubkey becomes vulnerable is where one of the private keys within that wallet is known. So never ever give out a single private key to anybody as they will be able to gain access, not easily but it can be done, to the whole wallet in that way. Some may think that its okay to share a public and private key in a sub wallet as that would not affect other wallets/accounts but it could and will compromise any wallet/account derived from the same seed.
When a xpriv key or the master private key is compromised the wallet is compromised and you can lose everything inside. If a xpub or public master key is compromised and gets known then only privacy is compromised because the person who has it can then see every single transaction in that wallet. They can however not make any transactions on that wallet.
The sharing of a xpub key thus has more to do with the risks to privacy than that of the risk of losing funds. Once somebody has your xpub key they can track every single transaction in every wallet/account derived from the same seed. Only way to get past that is to setup a complete new HD wallet with a new seed.