there are two different things in a "client" that you should not confuse with each other.
1. the consensus rules.
these are a set of rules that make bitcoin. for example the block size is one of these rules. number of newly generated coins is another [1]. when you say a full node client it must enforce these rules to be considered a bitcoin client. if it is enforcing something else then it is not a bitcoin client and it won't be compatible with the rest of network and be on another chain (fork) as an altcoin. like Bitcoin-abc which is now on bitcoin-cash fork.
in short "compatibility with the rest of the network" that you ask about concerns this and this alone.
2. the rest of the things it does.
a full node is not just a full node to validate transactions, it also is a wallet too. so they may have a variety of options, for example better options for cold storage, a better GUI.
there aren't really that many alternatives to bitcoin core, most of them are forks of core with additions or modifications.
there is a topic here that discusses this:
https://bitcointalksearch.org/topic/how-many-bitcoin-core-alternatives-do-exist-4180898[1]
https://bitcoin.stackexchange.com/questions/52620/list-of-bitcoin-consensus-rulesIs there a practice among larger participants, like mining pools for instance to use alternative implementations for their own means?
usually miners go with the biggest project which happens to be bitcoin core because it is tested more, reviewed more and possibility of a bug is small. they may do modifications to satisfy certain needs but nothing rule breaking.
How do we know that an alternative implementation is not malicious?
they are (or must be) open source so you can check the code and see what it is doing or trust the others who are familiar with coding to do it.
Also would anyone be able to recommend a good read on how mining pools work. In particular I am curious how they share the workload towards the same objective, which is to find a hash bellow the threshold. I suppose they should have a way to communicate what combination of bits they have tried out to find the nonce, so others can move forward.
not familiar with the process myself but this may help:
https://www.investopedia.com/tech/how-do-mining-pools-work/