You see unlike BIP-39, Electrum heavily modifies the words in the given string before it passes them to the HMAC function. For example in the case of the test vector linked above all spaces are being removed then it is passed to the HMAC function as "msg". That makes the data part 35 bytes whereas it normally would have somewhere around 80+.
Basically my code was simply missing these 2 methods:
https://github.com/Autarkysoft/Denovo/blob/648419d8a3ccd590051ab390539b0b7147917d4e/Src/Autarkysoft.Bitcoin/ImprovementProposals/ElectrumMnemonic.cs#L234-L258