Topic: GPU chip degradation (Read 4968 times)

It can be either. Electromigration may cause a slow decline of maximum stable OC, particularly if the affected logic thats being degraded is in the critical path, or it may cause a sudden catastrophic failure that appears to come out of the blue

If you want to get a fairly quick understanding on electromigration and how atomic wire structure (chemical) changes as result of temperature and electrical current, there is a pdf document from some researchers on semiconductors at Arizona State University that is informative:

http://schroder.personal.asu.edu/Electromigration.pdf

In a nutshell:
Pretty neat that it mentions how hardware failure may take a very long time under "normal operating conditions," but is temperature dependent --> increasing temperature from 30 Celsius to 250 Celsius results in hardware lasting 30 seconds (example); in addition integrated circuits today use copper which does not electromigrate as fast as aluminum; the smaller the chip (40 nm, 32 nm, 28 nm, etc) the more prone it is to electromigration, because a higher amount of current is running through a thinner wire!

The chip slowly degrades and requires higher voltage and/or lower clocks to be stable.
Yeah, this particular card passed hour long Furmark and OCCT tests so it's still in a pretty good shape, except it's now mining at 935 MHz instead of 942.

I haven't had a catastrophic card failure yet, like some users here. Lowering GPU voltage does seem to prolong cards' lives wonderfully, this ASUS is the only card I could not undervolt.
All GPU fans are ok (running at 60% RPM max), though I wore out two case fans (they started rattling/buzzing). Those case fans were running at 100%

Do the effects of electromigration progress over time or is it just an instant thing where the card simply gives up after running great immediately before the total cut in the interconnectors?

If not, I believe it may be your own configuration issues with the drivers, OC, and etc.

Does the card still function fine at stock values?

you really like to piss away money + power on quad core cpus.

Yes. Open Mining stand built from a $6 wooden Walmart Shoe rack...lol

Here is an older pic of the setup for those 2 cards.....a bit different now (settings) but the stand is the same.

GPI #0 & #1 on the 6960MINER are the Asus DCUII cards.

What you say is true.
However, since I can't undervolt that card to achieve superior Mhash/W my next best bet is to push it as hard as possible to offset the (very significant in the case of this machine) system power usage: 8 TB of storage is quite power hungry, as is a four-core Phenom II.
Power usage rises linearly with core clock so I'm ok here, except for the temperature which isn't THAT bad. 52% fan speed is relatively quiet.

The idea of upgrading the server with mining GPUs was that the machine would pay for its energy footprint.
What I should really do, is change the case to something better ventilated, like the rv-02 I'm using for my dedicated rigs.

I take it you're running your DCIIs in a caseless rig?

It's unfortunate that you had to run the card THAT hard to get performance out of it that satisfied you.

My (2) Asus 1GB DCUII 6950's run locked shaders and with stock voltage @ 900Mhz, get 374 MH/s without breaking a sweat (40% Fan Speed, 67C).
I'm just not sure the extra 15 MH/s would be worth the clocks/voltage/shortened life to achieve....for me anyway..

Good luck with it all.

I know, P4, thanks for your concern.
No action needs to be taken, it's a highly overclocked 6950 with locked memory and voltage - of course it will get hot.

Did you know that the 69xx series was supposed to be manufactured in 32nm due to heat dissipation issues?
Obviously, the plan failed and AMD had to re-engineer those chips back into the 40nm process.

Actually, shame on me for purchasing a non-ref 6950 in the first place, but I wanted a quiet card since it's installed in my general purpose home server upgraded with mining capabilities.
It's right here in my study, I don't want to hear it howl when I'm in the bathroom  

The other card in this server is a 6750 (206 MHash/s) - an oddball combination of cards but case space limitations didn't allow for another large card and I got it very cheap.
Getting ablut 600 MHash with this machine which is not bad for a non-dedicated rig.

As I said, once the card can't pull 900 MHz I'll sell it provided it's still adequate for gaming.

In a word: electromigration.
You can slow the aging process by trying to reduce your temps. 77C is way above my comfort zone. Getting it down 10C will double your cards life expectancy.
If you have room for a triple slot cooler, I would consider buying one. Most of them are pretty universal, so if the card dies or becomes obsolete, just reuse it for another card.

FWIW, I just installed a Deepcool V6000 on an XFX 5870 @1 GHz with a dead fan.
I bought it for ~$30. Temps dropped from ~70C at deafening noise to 54C and you can not hear the thing with the supplied fans (fixed low speed, they dont connect to the videocard fan header and are therefore not variable, which I do find a problem, but it works well enough it seems).

Before I have had the same problem with one HD4670. I also thought that the card is gone, but I was wrong. A few weeks I used the card on lower and lower frequency until I replace the PSU with new one - the problem disappear.

If you can, try with another PSU.

WD-40 also likes to attract dust, which could worsen your fan's performance over the long run.

So much fail... wd40 is not a lubricant. It is a water displacer and has no lubrication properties. Spraying wd40 into a fan bearing will destroy them extremely fast.

That is a horrible thing to do. No wonder you have so many problems. 

The temps are as steady as ever, lately even a couple centigrades lower due to the winter conditions.
There are no clogged filters or malfunctioning fans. Fan speed is at expected 2800 RPM.

I might launch some older cgminer version using the old phatk kernel but honestly I don't think I care to.

Excerpt from my thermal log:

let's compare that with an entry from two months ago:

c0 as in Cayman0; c for clocks, t for temp, u for usage, f for fan. Exact RPM speed needed to be checked in cgminer.
This arrangement allows me to just grep the desired parameters from the weekly-rotated thermal log.
GPU parameters are logged every 6 minutes, every hour additional info such as CPU temperature, date and time is being gathered.

Are the temps same now as always? As that might also explain the sudden unstability.

Can't say I've personally ran into this degradation, I thought it only really happens with large-ish overvolting in new hardware. Not that I would be too surprised, chips are individuals 

My ASUS 6950 DCII 1GB seems to be affected at last⁽¹⁾.

The card has been hashing since early September. Since late October the core speed was 942 MHz resulting in 389 MHash/s.
945 MHz would drop the card on its knees in a matter of hours. 942 MHz was stable for months... until this week.
The card hung a few days ago and twice yesterday.
935 MHz has been stable for a day now...

I was fully aware that setting the "cruise speed" so close to unstable clocks left me with no buffer zone - I did that more as an experiment into degradation progression than anything else.

I never liked this particular card much for its forced 125 MHz memdiff and locked core voltage resulting in power use of about 150W and temps in the high 70s.
Still, the degradation rate has been slower than I expected it to be.

Once the card is incapable of pulling 900 MHz I'll give it a second life in some gamer's rig provided no other issues emerge.

Just a data point for anyone interested in the subject.


Notes:
(1) Unless its just the latest cgminer pushing it harder than previous versions. The hash speed hasn't changed so I don't think the miner is responsible for the loss of stability.
     I don't feel like digging deeper into the matter or trying different kernels, I'll just drop clocks until stability is regained.