Author

Topic: Worst case ASIC heat scenario (Read 225 times)

newbie
Activity: 4
Merit: 0
December 24, 2022, 12:43:03 AM
#17
How would an ASIC miner fail if the cooling does not work?

Ignoring risks from cables and connectors overheating, which seems to be the main fire risk.

So assuming an ASIC hash board has access to plenty of 12v DC power and the cables are over-engineered and all software safety mechanisms fail:

How hot will it get before it stops drawing power?

Or will it melt itself and cause a short circuit?

Thank God we have Immersion Cooling liquid

Check how it works: https://www.youtube.com/watch?v=sBgkyj-ILFk

If You have any questions or looking for solutions just PM me.

Antminer S9 after almost 5 years in our liquid Wink

https://lh3.googleusercontent.com/pw/AL9nZEUeLXvEptaOlSgHCSXCLXlutVlcc91mdOHT-VecFxf_WDm_S4_zQjWlcpobG-7hSxtMRVj40kLu9YlWh1NoIsxyaoECEGAzQ1bicKLddwWU22OPUMuO4WK5BKrow26CQc39bXopQDh6JynAK1GS-ORyCA=w876-h657-no?authuser=0

https://lh3.googleusercontent.com/pw/AL9nZEUjds2Cyc4iJqTh_iYLQT1RKTH5UE5fjbnY8bWDAlTYneMtp6VhL3hF1pY440YbTO-JcZZ5mrXT2_tn_161e2mrWSVvK7yvDKhjhH5axUEfEYoHX_amdQFtH-LI-L6hP7UpS3TL65KO8SqN7k0uYeQn6Q=w493-h657-no?authuser=0

Edit: I have an issue attaching an image. [ img ] doesn't work

Paweł, FlameIT
legendary
Activity: 2394
Merit: 6581
be constructive or S.T.F.U
December 02, 2022, 07:34:08 PM
#16
That means it is a very bad idea to use an oversized power supply.

The opposite is true, the problem is almost never the miner itself but the PSU or the other electric components, the miner itself is self-controlled and the likely hood of it heating to the point of melting is just slim to nothing, it will shut down if one of the fans stops and/or temps go above a certain degree, it's probably more likely that the neighbor's cat breaks into your garage and drops a candle on the miner than every protection on the miner itself failing. Cheesy
 
By under-sizing PSUs you will risk overdrawing from it, it's a lot safer to run an oversized PSU than not, and the PSU is less likely to fail or catch fire when not being fully loaded.

Now in my humble long experience with miners, whether my own gears or places I used to manage, I can tell you without a doubt that at least 80% of fire incidents happen to "loose connections", most miners that catch fires are those that use wires instead of direct connection to PSU's busbar, and most miners that catch fire are used, tight connections are everything in electricity in general, the un-plug/plug process takes away from the ability of the connector for firmly contact each other, so you have a used miner that changed hands (God knows how many) the connectors were unplugged 5 times before, you unplug them 3 more times, now they are pretty loose and ready to catch fire.

Both the male (PSU side) and the female (hashboard side) will become less secure, so even if you bought a new PSU the female connectors on the hashboard are still bad, if you google search for burned miners/rigs, and you pay some attention to the burnt components assuming they weren't totally fried, you will always see that the melting starts at a connector, not near the fan, not in the middle of any wire.

The "loose contact" also applies to the power cord going into the miner, the copper pins on the PSU need to "click" into the power cord, with continuous use/heat and bad quality, those on your C19 or C13 or whatever will widen and won't click-tight on the PSU pins, that's also a fire hazard.

One way I test this is by gently pulling the miner from the power cord, you will feel if it's holding tight or not, a good cable and a good PSU will stick together like glue, you will need to apply a good amount of force to separate them if you can unplug it easily, then there is something wrong.


15% of fires happen because of sizing issues, PDUs/sockets or wires being sized smaller than they should, a breaker that can handle more than a cable can, maybe a sudden drop in voltage, the miner doesn't care it will still draw the same watts even on a lower voltage, which means more amps to travel in wire, resistance goes up, so does heat, the wire is getting really hot, the mcb/fuse thinks everything is fine so it does nothing, and then fireworks.

I had a main MCB catch fire due to an unbalanced load on 3 phase system, the natural wire was sized with the assumption that there would be a smaller load on it than the load on the live phases (which is how a good system should be), but when one of the phases is out, the neutral will carry the amp difference, so that's also under the section of sizing issues.

5% of fires happen for other reasons that are probably beyond my knowledge, Cheesy.

I'd say, as long as your MCBs are smaller than the max capacity of your wire given a large voltage drop, and your connections are pretty tight everywhere, you will be safe, keep in mind that everything needs to be as tight as possible, I can't stress on this enough, many professional electricians I know are obsessed about that aspect, in fact, some of them will come back after any installation to re-tighten everything because when the wires run warm for a while they might expand a bit, then they shrink a tiny bit and now they are no more tight, so after they do their expansion and shrinking that happens due to heat and sparks, they tighten them again.
legendary
Activity: 4354
Merit: 3614
what is this "brake pedal" you speak of?
December 02, 2022, 10:24:42 AM
#15
too lazy to find em now but there are many, many threads with pics of burnt to a crisp mining farms and equipment here at bct. just start looking. big fires, little fires, and a ton of melted (but not actually on fire.. at that moment) plugs and power cables.

But how many of those were due to heat vs bad electric setups. Too small a power cable, cheap PDUs or power strips, generic low cost power supplies that were a fire hazard running a PC at a few hundred watts never mind a 1200 watt miner.

I'm sure there are some, but most I would think have nothing to do with miner heat.


ah i misunderstood

i do remember some KNC stuff (neptune?) was pretty bad in the day though and had a (deserved?) reputation of going up in smoke even with proper wiring etc used.. basically crap engineering and design. board parts were underrated or something?

https://bitcointalksearch.org/topic/knc-neptune-fire-hazard-warning-739877
member
Activity: 203
Merit: 37
December 02, 2022, 08:53:21 AM
#14
The cooling of a miner like an S9 is already redundant because it uses two fans instead of one.
Snip

I might be wrong, I don't have access to an S9 to play with, but I would bet my bottom dollar that both fans are running at variable speed, controlled by some part of the mining hardware and software.

And most likely controlled by the same hardware part and software! That would not be truly redundant, because there is a single point of failure for both fans.

Yes, one fan could fail and there would still be some cooling going on, but if the software that controls the PWM signal crashes, both fans might fail at the same time.
member
Activity: 203
Merit: 37
December 02, 2022, 08:39:25 AM
#13
In any case I never read about a burning miner before on this forum or anywhere else, so you should also not be to paranoid.

'Too paranoid' is of course relative, but I currently believe that I will continue to live with constantly running miners spread through my house.

I think the risk of causing a fire is very real and I intend to tame my miners as much as I can to prevent that.

Hope is not a strategy!

Taming industrial equipment so it can be safely used at home is no mean feat.
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
December 02, 2022, 06:45:17 AM
#12

I'm basically trying to figure out how to add redundant cooling and redundant (?smoke triggered?) power-shutdown to some small miners (200 Watt each).



The cooling of a miner like an S9 is already redundant because it uses two fans instead of one. I did some testing and in many cases the second Fan is not really possible but it is good for redundancy. Maybe you can set up your miners in an area where there is not so much flammable material, since only cutting the power may not be enough if the miner is already smoking/burning.
In any case I never read about a burning miner before on this forum or anywhere else, so you should also not be to paranoid.

too lazy to find em now but there are many, many threads with pics of burnt to a crisp mining farms and equipment here at bct. just start looking. big fires, little fires, and a ton of melted (but not actually on fire.. at that moment) plugs and power cables.

But how many of those were due to heat vs bad electric setups. Too small a power cable, cheap PDUs or power strips, generic low cost power supplies that were a fire hazard running a PC at a few hundred watts never mind a 1200 watt miner.

I'm sure there are some, but most I would think have nothing to do with miner heat.

-Dave
legendary
Activity: 4354
Merit: 3614
what is this "brake pedal" you speak of?
December 02, 2022, 06:29:57 AM
#11

I'm basically trying to figure out how to add redundant cooling and redundant (?smoke triggered?) power-shutdown to some small miners (200 Watt each).



The cooling of a miner like an S9 is already redundant because it uses two fans instead of one. I did some testing and in many cases the second Fan is not really possible but it is good for redundancy. Maybe you can set up your miners in an area where there is not so much flammable material, since only cutting the power may not be enough if the miner is already smoking/burning.
In any case I never read about a burning miner before on this forum or anywhere else, so you should also not be to paranoid.

too lazy to find em now but there are many, many threads with pics of burnt to a crisp mining farms and equipment here at bct. just start looking. big fires, little fires, and a ton of melted (but not actually on fire.. at that moment) plugs and power cables.
hero member
Activity: 1022
Merit: 642
Magic
December 02, 2022, 05:13:31 AM
#10

I'm basically trying to figure out how to add redundant cooling and redundant (?smoke triggered?) power-shutdown to some small miners (200 Watt each).



The cooling of a miner like an S9 is already redundant because it uses two fans instead of one. I did some testing and in many cases the second Fan is not really possible but it is good for redundancy. Maybe you can set up your miners in an area where there is not so much flammable material, since only cutting the power may not be enough if the miner is already smoking/burning.
In any case I never read about a burning miner before on this forum or anywhere else, so you should also not be to paranoid.
member
Activity: 203
Merit: 37
December 01, 2022, 04:33:00 PM
#9
Heat from the miners is usually the least of fire problems. The main issue is power supplies pushed to their limit and overheating / shorting, the electrical connectors pushed passed their limits and the back end circuits not being properly setup.

Miners for the most part will shut down if they get to hot. But the wall outlet that everything is plugged into will keep pumping power though even though it's burning hot to the touch.

-Dave
I agree completely with that.

But I only use small miners with negligible risk of frying the AC outlets, unless the miner or it's power supply is already shorting.

I'm basically trying to figure out how to add redundant cooling and redundant (?smoke triggered?) power-shutdown to some small miners (200 Watt each).

I am worrying about the 3D-printed material that holds the additional fans and the plastic of the additional fans themselves. That would be fuel for a fire. The miners (without my additions) don't have much flammable materials on and in them. Maybe they would just let out a lot of magic smoke but not set the house on fire, but the added PLA and fan plastics could potentially turn it into a house fire.

member
Activity: 203
Merit: 37
December 01, 2022, 04:14:14 PM
#8
If the software fails it will obviously heat up until something else fails. Means that it will either melt until it gets an error or it will start to burn. I had my Brains OS+ software already failing one time for some reason, but since the cooling was still intact nothing happened. I just noticed that it was running at a temperature that it should already shut of itself.
I guess the lesson from this is to always have a separate cooling system that does not fail when the main cooling (which is presumably controlled by the ASIC board) does fail.

At least 2 independent cooling methods.

And maybe some sort of smoke detector that will shut down the AC power to the whole thing if any smoke is detected.
member
Activity: 203
Merit: 37
December 01, 2022, 04:06:32 PM
#7

After I was back, my water was 75 oC hot and temperature measured on board was 90. Thankfully, S9 shutted down itself after going above 90oC. Thanks to immersion nothing wrong happend AT ALL to the device or house Wink The tank and water were just a bit hot Wink


I don't think that proves that immersion cooling saved the day.

As you say, the miner shut itself down, as it should.

To test this, you could submerge a heating element in the fluid and put the max power which the power supply can produce through the heating element.That would simulate the miner NOT shutting down, but shorting out completely and using all the available power to generate heat.

Maybe you would have ended up with a water heater explosion? See Mythbusters experiments: https://youtu.be/rGWmONHipVo

Another question is at what temperature will your cooling fluid start to smoke and when will it catch fire like a cooking oil fire?

My guess is that submersion reduces the fire risk frequency, but if you were to ever get a fire, then it would be an instant monster that cannot be extinguished with water.
https://www.youtube.com/watch?v=3LWYXJvU7yM
hero member
Activity: 1022
Merit: 642
Magic
December 01, 2022, 07:07:20 AM
#6
If the software fails it will obviously heat up until something else fails. Means that it will either melt until it gets an error or it will start to burn. I had my Brains OS+ software already failing one time for some reason, but since the cooling was still intact nothing happened. I just noticed that it was running at a temperature that it should already shut of itself.
legendary
Activity: 3500
Merit: 6320
Crypto Swap Exchange
December 01, 2022, 07:06:48 AM
#5
Heat from the miners is usually the least of fire problems. The main issue is power supplies pushed to their limit and overheating / shorting, the electrical connectors pushed passed their limits and the back end circuits not being properly setup.

Miners for the most part will shut down if they get to hot. But the wall outlet that everything is plugged into will keep pumping power though even though it's burning hot to the touch.

-Dave
newbie
Activity: 4
Merit: 0
December 01, 2022, 07:01:10 AM
#4
How would an ASIC miner fail if the cooling does not work?

Ignoring risks from cables and connectors overheating, which seems to be the main fire risk.

So assuming an ASIC hash board has access to plenty of 12v DC power and the cables are over-engineered and all software safety mechanisms fail:

How hot will it get before it stops drawing power?

Or will it melt itself and cause a short circuit?

Let me tell You my expirience I had 4 years ago when I started my journy with company: FlameIT - Immersion Cooling. Movie on main page is S9 the story is about Wink

YT: https://youtu.be/biC2pBTikOs

I have manage to buy Antminer S9 at super low price. I have paid less then PSU was worth alone (really nice Dell 2kW server PSU). That was gold hit Wink Anyway S9 was already not profitable at that time.

The story is. I have built my first immmersion coolling tank, put S9 inside and started my experiments. Everything was working fine, I was nicely heating water (summer) for my house + I had to open one radiator in basement to dissipate extra heat. Then I have closed that radiator and left house for a weekend forgetting to open it again Smiley

After I was back, my water was 75 oC hot and temperature measured on board was 90. Thankfully, S9 shutted down itself after going above 90oC. Thanks to immersion nothing wrong happend AT ALL to the device or house Wink The tank and water were just a bit hot Wink


How other miners look in our best available synthetic fluid? Check that:

https://www.youtube.com/watch?v=sBgkyj-ILFk

Cheers! Paweł 'felixd' Wojciechowski, FlameIT
member
Activity: 203
Merit: 37
November 30, 2022, 07:11:17 PM
#3
Yes
Case in point:
Scary stuff then!

That means it is a very bad idea to use an oversized power supply. Maybe the power supply needs to be matched so it cannot produce much more than the ASICs can use in normal operation.

And everything around the miner should be fire retardant.
legendary
Activity: 3822
Merit: 2703
Evil beware: We have waffles!
November 30, 2022, 05:38:14 PM
#2
Quote
Or will it melt itself and cause a short circuit?
Assuming that all software protection fails or the problem happens before the monitors can pick it up -- Yes
Case in point:
member
Activity: 203
Merit: 37
November 30, 2022, 04:39:40 PM
#1
How would an ASIC miner fail if the cooling does not work?

Ignoring risks from cables and connectors overheating, which seems to be the main fire risk.

So assuming an ASIC hash board has access to plenty of 12v DC power and the cables are over-engineered and all software safety mechanisms fail:

How hot will it get before it stops drawing power?

Or will it melt itself and cause a short circuit?
Jump to: