Temperature and power use aren't the only limiting factors when overclocking.
There are also safe limits for core and memory clocks, for every unit. Off the top of my head, excessive memory clocking can lead to artifacting (I've seen some users with Samsung memory get close to +600-650 stable OC, but check with your own hardware), which is a sign that you are pushing your card too hard.
Whether or not is something you should check experimentally, although there are some trends with certain algorithms. For example, with equihash, I haven't noticed gains between +100 and +300 mem on my 1060s and 1080tis, but people have had mixed results.
in my early days of mining, I pushed my cards with +1800Mhz on memory (under Ubuntu), after a week one started failing at start. I plugged it into my windows desktop to see what what happening and like you said, display was full artifacts. It was still under warranty so I had it replaced, now I stay around +500Mhz which is probably much safer for the long term.
where to get ?
.
