One other question is are you using any secondary cooling methods such as a tabletop fan blowing onto the cards to help dissipate heat? You really can't rely on the card's own fans to dissipate the heat efficiently, especially when you are running more than one card on a rig. Then you have to deal with heat from one card affecting the other cards, and vice versa. Not sure what your setup is like. Do you have the cards set directly into the slots on the motherboard? If so, is your motherboard and cards inside a chassis of some sort? Or do you have the cards attached to risers and attached outside of a chassis? Either way, having secondary cooling is always a good idea. Hopefully you get it sorted out and not lose any of your cards. Best of luck!
not sure i agree, i find it depends on the brand of the card! i have rigs with 3 cards sitting next to each other and they run in the high 60's low 70's! My sapphire rig runs really hot, 1st card was hitting 84 so I put a cheap external fan on top and bam temps fell but the rest are fine with fans set to 95%! suppose i also run those rigs on the cooler master tests benches, that may help.
I agree that different cards can handle different amounts of heat. But that doesn't necessarily mean that running them up to or exceeding those temperatures is going to be fine in the long run. Air flow is very important for heat dissipation, and it sounds like you have a good setup with plenty of air flow. And you made my original point too when you said that you put a cheap external fan on top of your sapphire rig and the temps immediately fell. That was the point I was trying to make.
Putting additional cooling fans around the cards will not always lower the temps, it may just lower the fan speeds, depending on how you have CG/SG/BFminer set up.
For my 6 R9 290's they're:
"auto-fan" : true,
"gpu-fan" : "60-95",
"temp-cutoff" : "95",
"temp-overheat" : "92",
"temp-target" : "85",
"temp-hysteresis" : "2",
These are cards with the original stock cooler which is REALLY loud and the cards run very hot (by default the cards will run up to 95 degrees before dialing the speeds back but the fan will only be running at 57% as AMD didn't want them to be too loud for most people).
Hence the fans spin up and down to keep the temp @ 85 degrees, however if I used;
"auto-fan" : false,
"gpu-fan" : "80",
"temp-cutoff" : "95",
"temp-overheat" : "92",
"temp-target" : "85",
"temp-hysteresis" : "2",
The temps would increase and decrease depending on the ambient temp until they reached 92 degrees when the fans would spin up to 100%, if that's insufficient or the ambient rose too quickly and the temp reached 95 degrees, the cards would stop mining until the temp got back below 95.
Keeping a stable temperature, clock speed and voltage will greatly increase the lifespan of the cards over a changing temp/speed/voltage.
This is because
A changing temperature will mean the materials will be expanding and contracting constantly (specifically the solder) and therefore break themselves
A changing clock speed doesn't make too big a difference providing the cards aren't changing their own voltage but it still puts additional strain on the chips
A changing voltage will be changing the temp and putting more/less load on the voltage regulators increasing their work.
Every little bit of stability helps. You have to remember, these are consumer grade cards, designed to run around 70-90% load for 1-3 hours a day not flat out constantly.
Think of a car (most easily recognised analogy rather the most accurate), if you run a car flat out constantly for days and days (assuming you can supply fuel) it's not going to like it too much, but it's better than run constantly at changing speeds from max to 70% to max as you have to keep changing gear (putting strain on the gearbox, clutch, driveshafts).
Additional cooling to supply cool air to the GPU fans is a good thing though, it means the card's fans can spin slower to maintain the same temp which means they'll last longer.
A lower temp is also preferable and will use less electricity due to current leakage and resistance, but you have to live with these things in a house usually, so noise becomes an issue.
Try to run at the lowest temp for the max noise you can cope with while ensuring that you rig's safe from rain (and potentially fog).
EDIT: Also, make sure to have the external fan as close to the coolest air as possible while ensuring the cool air it's pushing towards the cards can actually reach.
Putting the fan as close as possible to the cards is rarely the best idea as it'll end up pulling some of the hot air from the cards back through recirculating it, the air doesn't just come from the back of the fan, what also happens is a low pressure area is created infront of the fan due to the fast moving air. The surrounding air will try to fill that area as fast as possible from wherever it can.. Think of when changing the binliner in the kitchen, when you put the bag in, it's all tight, so you blow in there to open it out. If you put your head right in there, the only air to get in will be from your lungs, blow in from further out and you end up with a lot of the surrounding air going in too
Video exampleAftermarket coolers and particularly water cooling or mineral oil cooling is great.
Aftermarket coolers are nearly always far better at cooling than the ones that come on cards from the factory (especially the R9 290(x) stock coolers which are terrible), giving better temps at far lower noise levels. ~ £30-£40 per card
Water cooling allows the radiator to be put in an optimal location, allows the cooling capacity to be shared around all the cards attached and can allow for easy expansion in the summer or when adding additional rigs. A significant drop in power consumption is also possible (~10% depending on GPU and cooling capacity of your setup). ~£800-1000 for 6 R9 290s (then £80-100 per card thereafter until an additional or larger radiator is required)
Mineral oil (submerged) cooling is excellent for cards doing mining as it cools the entire card equally and is relatively cheap. The big downside though is if you have to change anything its very messy. ~£300-£500 per rig
I'd be happy to give folks advice on cooling their rigs better on a case by case basis, it gets tricky when talking about specifics on a general basis as we're talking here about rigs of 3 x R9 270s (400W) upto 6 x R9 290x's. (2.5KW) and varying budgets, climates acceptable noise levels etc...
EDIT: This video is worth a watch if you're not experienced at fiddling with computer hardware and all the above was interesting and useful for you, give this video a try
Linus Tech Tips: Safe PC Temperatures as Fast As Possible