Topic: My BFL SC Single 60 went out in a puff of smoke - what next? (Read 5710 times)

Cool. Still want to return it, or sell it?

I have received an RMA number from BFL

Any news?

I had troubles with my single, wrote BFL for a RMA and within 2 days received an email asking for my original invoice # and email. I wrote back telling them I bought the unit second hand and did not have that information. Within another day received a RMA. I sent the unit in and received a single back in about 1 week. It appears they just removed my cover and placed it over a brand new unit. Really overall great service.

I also have another single that acts the same way of slowing down after about 12 hours. I just stop and restart the easy miner and it goes back up to 58/59GH. I think it's either software or memory problems with my desktop.

Hm, you're right. Does the power supply get warmer? When the hashrate is dropping, what do you see per chip by doing D, then M to switch to per chip stats? I'm wondering if chips are dropping, or the clock is dropping.

Post and let us know.

temps are stable while it is at full speed, then temps drop with hashrate, if it was thermal I would expect dropping hash with stable temp

I think software, was hoping it was a known issue

The software load on the LS is really sensitive to temps. Check to see if any of the fans are blocked or down, if you really want to I can take a look at it, drop it in the mail.

Speaking of which I bought a 23.5gh LS that I have boosted to 28gh. Can be done just need software and heat sinks in the right place.

Memory leak?

here is my 60 single's new trick, it will run about 12 hours at rated hash, then starts slowly dropping speed.  Used to be it took me a few days to notice because it happened slowly, now I will be under 40 in 24 hours and I am rebooting it daily.

anybody heard of this one before?

Any news?

OP is there any BFL warranty?  or had you manipulated the single in any way that voids a warranty?  I have often wondered about this, seeing how shady BFL was originally with their delays.

try the Nidec Servo (Scythe branded) gentle typhoon 1850s

finest fan I have ever dealt with

http://www.sidewindercomputers.com/scge120mmsic2.html

Can I just butt in and ask what spec the BFL 120mm fans are inside the single, I am running one outside the case currently in a cold loft and want to get new fans on it and the case back on (originals broken in transit apparently and I don't have them)

tried some noctua 1.44W 0.14A things the guy gave me when selling and they're hopelessly underpowered.

TIA, and good luck OP.

It had been running for more than a month

That could work. I haven't played with the LS yet, but I would say the key is to get a fan on top of the heat sink and one blowing from the side over those FET heatsinks. Running even a little vornado on mine is keeping things cool.



There's only one fan? Blah, I thought there would be two. Ok.

C

Incredible thread, thank you lightfoot for your insight and a question if I may.

Little singles out in the open with a big fan blowing at them should be fine, right?

my temps are also reporting around 70 degrees (which happens to be the ideal temp for my jupiter, so I thought those temps were ok.

ambient is 90 f   poor AC is going full bore just to keep the room at that temp.

The thing is, when I finally got these little singles, I took one look and stripped the cases and threw them and the power supplies away. they didn't even include a fan for the heat sink! one fan only and that is the intake fan, which is terribly constricted by that ridiculous face plate and I thought "I can do better"

So I stripped them down, set the intake fan on top of the heat sink blowing down, and that blower fan blowing across them.



And a question for the OP please. How long was it running before it blew?

yes, I have an IR therm, but I just had downtime the other day when I cut the output plate and blew the dust out...It will be a while before I want to shut it off again.

Thanks for reporting on this. Real data always helps, I have put three little heat sinks on the bottom of my three hot FETs as well as the top, and both are wicking away a pretty fair amount of heat.

The temp sensor for the board is in the center of the square of 8 hash engines, it does a very good job of watching the temp of those chips but does not do a good job of monitoring outliers like the FETs. Although come to think of it, the little SC box that the jally uses pulls the air from the front, right over the FETs, up through the heat sink, and out the top. Maybe that is why they did that; and putting the fan upside down will cool that center board sensor (which is really not the problem) but will not flow air over (and in the case of the SC60) or under the FETs.

Does anyone with an SC60 have access to a IR temp tool? I am wondering about the temp of those sinks, with the fans going down.

Looking forward to hearing about the repairs.

The case was off when it blew up, the fans above the ASIC clusters were blowing down through the heatsink, the end fans were both blowing inwards - i didn't consider the heatsinks underneath at the time (note, they are just stuck on to the base of the PCB with double sided tape, so fairly poor heat transfer to them anyhow), although when the case was on originally, it was hashing below 60GH and temperature in cgminer was 70/71 °C so I considered it better to run 10° lower temps in cgminer. The failure appeared to be on the upper suface though and there did not appear to be thermal damage on the underside.
As an update bfl did finally return my email with an RMA number but it was too late (and as mentioned before would take too long and was cost prohibitive)

One of them. If there was no crossflow on the bottom heat sinks then that would be problematic. Likewise if the fans just pointed inbound then only half the sink would get cooling, the case provides the tunnel for effective crossflow top and bottom.

Someone needs to send me a blown one of these, I'll post a whole thread on how *not* to nuke your Single. Or something.

yeah, I think we found the problem

Just to clarify, was the case open or completely off? Were the side fans on, how were they blowing?

C

I think the ambient temperature was around 24 degrees C, temperature of the miner was 60 to 65 through the day. Cases were off, fans blowing in and down.
Cheers

What was the ambient/room temp?

Do you know what was the temp reported by CGminer also?

Wish you luck for a fast fix and back to happy mining.

It was not air conditioned

The unit is on its way to cranky4u, hoping he can fix it. I decided not to return to BFL due to the time/cost for shipping and customs delays. My fingers are crossed. 

Mmmhm. I am not going to exceed 24-27gh on my jally single based on this. Can't wait to get another chip so I can see if the temps on the FETs go up linearally or geometrically with load.

The problem with the warranty though is that they should be able to replace your unit. And if it takes 2 weeks to do that, it's two weeks of not hashing at current difficulty, the result may be a 20% hit to your profits or more based on diff changes.

That could be fixed by BFL offering some sort of conditional cloud hashing during the time your unit is being fixed. Man that would be a public relations *WIN*! Say when you get an RMA you get put into a special hashing queue where your hashes are immediately escrowed while you send the unit in.

If you didn't mod it, and they fix it, they send it back and when you take delivery they stop the mining in their cloud and give the money to you. If you did mod it, they keep the hash money.

That is a no-lose solution, someone suggest it to BFL. In fact that would be the biggest PR win in this kind of community and it would not cost BFL much of anything to do.....

Just a side note, from what I am seeing the two 30gh sides of the 60gh unit may be independent. In other words, failing that burned side should not affect the right side's 1 volt supplies.

Or, remove those FETs, fire it up (with fuses in the power supply lines) and you should be able to restart hashing at 25-30gh on the remaining side. Half a loaf is better than none and all that. I'd say 75% probability of this working.

Good luck!
C

I milled the output plate of my 60 yesterday so it is just a thin frame around the outside to hold the case together.  It cut down the wooshing air noise and where before the input fan would run all the time now it throttles in and out, so it is definitely cooling more efficiently.

The thermal pads in the pics I have seen do look very thick, I may rework those in the future, also thinking about some thicker 92mm Noctua fans for on the heatsinks themselves might really cut the noise...but I hate the downtime to implement, lol.

just wondering about the cooling for your 60g. was it in an aircon room ?

I think there's probably a reason BFL now sell these a 50GH/s mining machines instead of 60GH/s ones.

https://products.butterflylabs.com/homepage/50-gh-s-bitcoin-miner.html

There is a lifetime warranty, I'd send it back.

There is far too much thermal paste or pads on those ASICs it looks like cake!

Thanks everyone - I have a solder/hot air rework station but I agree it may be better to have someone more experienced look at it before I break something else.
I have got in contact with Cranky4u who has generously offered to have a look at it for me next week. Fingers crossed.

Yup. Sometimes you're lucky and it just nuked the FET. However since the FET gates on low voltage stuff like this are rarely driven by optoisolators and isolated DC-DC's, it usually takes out the drive circuits as well. IGBTs in high voltage systems are typically driven that way, which firewalls the damage to the IGBT, the resistors between gate and collector, and the um... circuit that snoops collector to emitter voltage and sends the HOLY FUCK EC voltage is going through the roof! WE HAVE A SHORT CROWBAR GATE NOW! thing.

The latter is usually what saves that 50kw fire that results when one side of your IGBT shorts. The crowbar sees the voltage from C-E screaming up (which means high current draws), forces the gate down to ground and lights an optoisolator to light up the "you're fucked" light on the dash.

Then if you're a super-special person, you say "No I'm not", bypass the crowbar circuit, fire up, and send me a box that looks like someone used a flamethrower in it. Been there, dealt with that. :-)

Indeed. looking at the specs for these FETs I think they have a solder pad under them that is where the heat's supposed to go. Hot air is probably the best way (400c is what seems to work with my Aoyue unit) to float these off. Not hard, just need the right tools. But they're small, so a pair of soldering irons one on each end might work as well.

Never dull.

hope you're right lightfoot, I'm just going on my spotty success replacing the VRM FETS on videocards and motherboards; sometimes it works, sometimes it doesn't, and sometimes the FETs smoke instantly when you apply power.

I think OP should definitely get ahold of Cranky4u if he doesn't feel comfortable soldering it himself

So.... What to do? Note all of this is from a complete random stranger on the Internet dreaming about stuff, so disclaimer, disclaimer, don't do any of this in real life, you're responsible for your own universe, this is just me.

If *I* had a board like this....

Bit of knowledge about the BFL design: If the 60 single boards are like the little_single, then the power supplies for the FETs are connected together and connected to the 8 chips that run on the board. What I don't know is if the left side 8 chips are powered by the left two power converters (the FETs that went foom) independently of the right side 8 chips. They may all be on one bus, or two.

Update: Based on reviews of the SC30, it looks like the two sides run separate chip power converters. That is good and bad from a design stage, but it can explain how one can lose half their hashing power.

Anyway, what's the damage? Well, it could be limited to the FETs, I would say scrape off the wreckage, hope the Pads behind them are still good (test for shorts and opens to the other FET baks) and if there are no shorts, try firing it with a regulated 12 volt supply (without any mining software, just see if the chips come up with the remaining 3 supplies). If they do then either lobotomize 4 chips on that side to be safe by turning them off in software (making a 45gh unit tops) or if this thing has all the power supplied in parallel lobotomize six chips to keep the remaining three supplies from overloading (assuming they also have some damage). And make sure your power supply's 12 volt source has a *FUSE*, wire in a (12 watt*12=144 watt/12 volts=12 amp fuse *TOPS* since you blew things start with a much smaller fuse TOPS). If you blow the fuse without hashing, you have a board short, start digging.

That's the fast way. Sure it's 35-45gh instead of 60, but half a loaf is better than none, all that. Peter if you have SMD soldering skills and you were me, you might want to try taking off all six FETs, check for shorts on the 1 volt to ground and 12 volt to ground lines, put in the power supply fuse on your lines,  and see if anything works (you need to remove both sides because they might be back-feeding the oscillator)

What's the worst that could happen?

If you want to fix this whole power supply it's going to be harder. My experience is that this will blow the FETs and possibly the gate driver chips. I don't know if they have 1 gate driver powering all twelve chips on a side, or two sets of gate drivers. If the driver is blown and they have two drivers per side, the other side should still work and you can power 12 chips. If the sides are isolated and they have one set per side you can run 8. If the sides are not isolated and they use one honking driver to power everything then you're screwed till you replace the driver (a 2708 chip on the bigger controllers, look for something like that in the chip mainfest).

What's the worst that could happen?

If the FET pads are warped, you might not be able to get it going without bypassing the 1 volt lines or re-routing power from the 12 volt source supply. That's way beyond the source of my ability to help from the other side of the earth.

So anyway, a fast potential solution that I would use to get me something would be:

Remove the wreckage with a good pair of soldering irons. Don't go with insane amounts of heat; the carmelized board will conduct heat faster and melt more than normal.

Check for shorts

Try powering it up without hashing with a fuse in the supply.

Reprogram it to only run 8 chips instead of 16.

Try hashing.

Add a few more chips, do not exceed 11 of them if you want it to last.

Good luck. Time is of the essence since difficulty keeps going up and time waits for no man in Bitcoin world. Having 30gh running now is better than 60 in a month's time. Good luck and keep us posted.

And thank you for sharing this: It's given me a chance to think through a problem, which to me is always fun. If anyone else finds this useful and it makes them money do me a favor: Donate some of that to your local soup kitchen in the name of "Bob Dobbs". That way my rantings here make the world a bit better or something like that.

C

Sharing my experience with a shorted chip, I can say it was probably not that.

Specifically as I add chips to my jally, the first chip I added I accidentally shorted some pins so +1 was connected to gnd. Happens when you move the chip while placing it hot. Anyway, the symptom (documented in the forum) was that the unit would "fast flash" on startup and I thought I was fucked. Removing the chip allowed the unit to start normally.

What seems to happen is that if +1 gets hard shorted, the board detects it (probably in hardware for speed) and the oscillator that gates the FETs shuts down. This immediately will collapse the voltage on the +1 rail to zero, and the board does it's "fast flash" dance. No damage. I think BTW this is why some people who take the heat sink off their jally to reprogram it get the fast flash; they torque down the damn heat sink too much and crush the chips into the board. Short develops, jally don't work no more. Solution would be to pull the chips and replace them, board is probably ok. But that's jally 101, we're in single 60 land.

Anyway, in this case it looks like running temps on the FETs have been high for some time. FETs don't share loads equally as temps go up, that's why on the better/bigger/more expensive electric car controllers you use a big 300-600amp IGBT instead of a bank of 20 30 amp FETs in parallel. but IGBTs cost large money and need more expensive drivers around their 2708's, but I digress. Moral is when a Curtis golf cart controller is used to drive a small car the FETs heat unevenly and the hot one is the first to short. When you short 600 amps at 150 volts, hilarity ensues as the other FETs blow up around it. :-)

I would guess that Q4 and Q(burned to a crisp) were heating up together, with Qcrisp finally failing shorted. That would probably also fail Q12 and provide a dead short from +12 to ground and blow the fuse in the power supply. End of story, board down.

What caused the total warping though on Qtoast is hooking up a bigger 12 volt supply. Since it's shorted, all current from the supply would shoot through Q4/Q12, and that would basically be a resistive load. It would heat up until either the silicon melted (which it did, letting out a lot of smoke) or the power supply fuse blew. The board couldn't stop it because the FETs were shorted, so the gates could not be opened under programatic control. Foom.

I've seen this happen on a 300 amp IGBT; the main pack fuse opened, but in the brief meantime there was about 90,000 watts of heat being generated in that IGBT. This is why they have big big big heat sinks. And why you have DC rated fuses, if someone was stupid enough to put an AC rated fuse in or bypass the fuse you would have a 90kw air heater in your aluminum box. Hilarity would... ensue...

What to do? I think there's a way forward, I'll post that next. Peter took the time to post this intel, I'll post a possible fix. Note I take no responsibility for *ANYTHING*, this is all pure speculation that I would never expect anyone to try in any way, shape, or form.

C

3rd image bottom right "fun pass" fun for who? BFL because of all the money?

anyhow it looks f***d XD and a bit of my ability XD that said it dosnt look like its dmaged anything else other than the 2 chips so it dosnt look like the voltage got any further meaning they did there jobs and failed (probably failed a bit to well and made a nice mess)

Saw that too. Suicide single of death

Build date 9-11.... no conspiracy theorists jumping on that one yet? 

I am an Electrical Engineer located in Bendigo, Victoria, Australia. I would be happy to take a look at your SC single to see if it can be bought back from the dead...PM me if ur interested

yup, VRM

question now is, did they fail because the ASIC failed and presented them with a bad load?  or did they take anything else out when they went?

I took the heatsink off which reveals 2 failed components I'm guessing Q2 and Q4 have failed. I couldnt see the identifier on these components, however one in the next row is 014NE2LI which I think is a power MOSFET.
The image of the heatsink shows the magic smoke escape locations.....

Thanks for your input lightfoot - I will see if a local place can rework it next week. Maybe still some life left in it yet! 

Thank you for posting those. I hadn't seen the newer boards fully loaded, that explains a lot.

In a nutshell, that section is where the 12 volt to 1 volt FETs are. There are two sets per side, for a total of four sets per big board (the little_single and jalapeno have two sets). Each set consists of three FETs which gate the "high" half of the 1 volt supply, and three FETs which gate the "low" half. At any moment, half the FETs are on. The capacitors then smooth out the flow and the voltage sensor on the 1 volt side feeds the computer back information on how long to keep the FETs on/off to maintain a smooth 1 volt.

When a FET fails due to overload or whatever it usually shorts. This puts the high and low lines on the 1 volt bus at the same time which shorts out one of the low FETs and the fuses blow in the power supply. Putting a new supply on it just fed straight into the dead short and let out the smoke in a "poof".

I've worked on things like this except they were 400a 300 volt IGBTs that would short on electric car transmissions. Same results with a *LOT* more smoke before the 350 amp main fuse would blow with a BOOM. :-)

Chris

Thanks for having a look

heres the other pics

Hm, neat! Let me take a look at that...

Question: Can you take a picture of the whole half of the board? And the whole board front and back?

Edit: Fast thought: Based on where that is, I'd say it's one of the FET clusters (push/pull) for the 1 volt power rail for the board. The little sc/jalapeno has two of them, I wonder if the 60 gh SC has four or eight...
 
C

So I took the covers off to try and get a closer look. It appears what ever is underneath the small black heatsink failed.

Thanks. I'll try to get hold of them

Have you tried giving them a call during normal business hours? They have answered my calls in the past. With this possibly being a warranty issue, I can see them trying to take care of it.

good luck, you can try sell it if you want.. im sure there are people who buy broken asic...

thanks - it wasnt a loud crack, and it was white smoke that had a really acrid odour. I thought I saw something bubbling as well (maybe it was electrolyte from a cap?) - anyhow, I emailed BFL when it happened, but havent heard a peep from them, so I guess I will try and get the heatsinks off and have a closer look tomorrow. Ill try and get some pics

Been there, done that.  I was sitting next to mine fortunately when it blew.  Heard a pop (probably a cheap cap) and white smoke out the back as the fans spun down.

Good luck with RMA, took me 2 weeks just to get attention of somebody on the BFL forums.

There are a few people who are doing mods like lightfoot - maybe he would want the dead device for a fee.

puff of smoke. was it white smoke? and how loud was the pop?

I am not a electronic expert... but "crack" sounds like some bug exploded. Talking about the plausible cause... I would bet for static electricity while manipulating the machine.

I would remove (CAREFULLY) the heatsink and review the PCB, even with a magnifier in order to locate the exact place and component.

Or you can post a picture of the PCB both front and back and wait till someone take a look.

Hi All,
I had my SC Single running nicely at 64 GH until a few days ago when it stopped hashing. I thought it may have been the power supply, and tried a brand new Corsair GS800 PSU. Anyhow, once connected, I heard a "crack" sound, no fans spun and a fair amount of smoke exited from near the middle of the PCB from underneath the smaller black heatsinks (not the ASICS). I doubt BFL will be any help, and with shipping to Australia taking so long+expensive - anyhow, has anyone else experienced this, and any suggestions on what I can try to fix it?
Cheers