Topic: I think one of my GPUs has the the dreaded electromigration. - page 2. (Read 3291 times)

Well 48 chips, 24 cards.

Yeah it may be the "weak link" just thought it was interesting as this card has been running 24/7 for over a year now.  I figure these topics will become more common place as the oldest GPUs hit 18 months old, 24 months old, etc.

48 cards, that's quite a lot and an increased chance you stumble upon one that is of lower quality (as we know every chip is different, same as with CPUs)

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

You had same cards/same settings/same environment when you used windows?
As for the memclock I had a card where it started to lock up at 280MHz after some time, but it did not occur when I increased the mem clock to 350.

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

On correction.  Linux has no problem undervolting.  I have on rig running @ 730/240 1.00V (vs 1.05V stock) as a test.  cgminer has no problem setting values allowed by voltage and 5870/5970 allow core voltage to be lowered significantly (I tested down to 0.6V and it ran fine). However it relies on ADL library and as such only will run values allowed by bios so no overvolting without bios flash to increase the allowable range. 

Would be nice if AMD exposed all functionality via drivers and allowed larger range of values.  It would make 3rd party hack utilities obsolete.

  One of my cards is seeming to have a dead core quite often but IDK if it's due to EM or simply a dud card.  Diamond's looking into it though so we'll see.  I wonder if EM is the issue of the second card in that system though as it randomly sees and doesn't see the cores.  Either way, sending both back. 

  Can't vouch for standard clock instability really so far.  I'll get back to you in a year

I mentioned to you in my PM that recently the quality of the 5970s I have been getting on flea-bay have been sub par. I am seeing ~40% "failure rate" of the 5970s in the past ~30 days in the ones I got. Most of the failures have to do with the card starting up just fine and running without problems for about two to three minutes, and then the temp gauges suddenly go from 60-65C to 120ishC and the card stops responding and crashes out the whole system.

When I set the clocks down to normal levels the card will run without issue, but it runs 10-15C hotter then the 5970s next to it that are overclocked to 800. They are all spaced out with enough room between the cards to have two dual slot cards between them. I also have box fans blowing in cool air into the rack they are mounted on so I know its not a heat problem.

I also have AX1200s for all my rigs and I still experienced this problem when I did the following:

1. Brand new motherboard of the same make/model.
2. Replaced riser.
3. Replaced PSU with brand new AX1200
4. Replaced outlet.
5. Replaced breaker.
6. Clean install.

It "feels" like people run into this issue and since they are out of warranty they throw the card on Ebay and sell it off. It could also be pure luck that I have gotten bad cards lately. 5870s all seem to the fan or fan controller that are burning out on me a couple weeks after getting them.

Edit: It happens to me any any mem clock. I don't have a good number on specific temps.

That is the important bit here : Linux does not properly support overvolting ( discard effects of EM aftewards ) / undervolting ( prevent EM in first place ) and reading VRM temps for 5870 / 5970.

That sucks and needs to be fixed.

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Can it play games at stock clocks?  Mining is easier on the card than playing games.

Oh god. 

At this point I am not sure I can even warranty it.  It runs perfectly stable at stock clock and I doubt any warranty guarantees stability at > than stock clock.

From what I hear, 5970's are being replaced by 6990s when there is a warranty claim.

Hmm that's certainly something I've been worrying about as time goes on.  What I'm more curious though is how the hardware vendors will handle warranty claims against cards that are nigh irreplaceable at this point.

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.