If you have stable connection it doesn't matter if you have 20 or 200 ping. If you lose the connection to server often then it's a problem.
Actually, that's not accurate - you're generally right that connection stability is probably the most important, but that doesn't mean latency isn't important also. Mining you're using TCP instead of ICMP (ping), but still it does give you a good idea of the best case round trip time for a packet - so when you see 20ms vs 200ms, that's 10x faster.
Why does this matter? If you're mining Bitcoin, then it probably doesn't because it's block time is 10 minutes - but if you're mining some coin with a very fast block time, let's say 10 seconds, then 200ms is 2% of the total block time. What this means is that it takes 2% longer for a block solve notification to make it to you than if you were on a faster (latency) connection. So you'll make 2% less mining revenue - if it's a single GTX 1070, then who cares - but if you're spinning up a larger mine then it definitely matters. Do I care about 10ms or 20ms? No, it's just noise, but at 200ms+ I would definitely be more proactive about finding a faster provider.
And just to be clear, in this example we're using ICMP RTT time, which is ROUND TRIP, so really the hit would be half of the RTT (or 100ms in this example). But I would still do my math using 200ms, because many ISP's prioritize ICMP traffic as it makes the network appear faster, and ICMP is connectionless vs TCP which incurs additional overhead. Is it 2x the overhead - almost never, but it's also almost never as good as ICMP, and I'd rather be more conservative in my calculations.
