200 GPU farm got rained on, had to move it.

kotarius

sr. member

Activity: 340

Merit: 250

Quote from: MaxDZ8 on August 20, 2015, 01:49:20 AM

It is ridiculous you relied on pressure to keep the rain out and then pushed directly on the hardware.

As long as rain isn't being pushed up from the ground, no chance it's going to enter the building.

No, an awning does not offer the same level of protection.

In particular, installing drainage is pretty much nonsense as it doesn't catch airborn water droplets unless you install a filter which will reduce fan efficiency.

I've never heard of such issues when the 90deg turn is used (it depends on how fast you suck in BTW).

The exhaust fan was blowing out, not sucking in. Even with the fan off, regular rain would not get in unless the storm was severe and catastrophic.

VirosaGITS

legendary

Activity: 1302

Merit: 1068

Quote from: Gladimor on August 20, 2015, 12:29:11 PM

Quote from: kotarius on August 20, 2015, 01:30:42 AM

Most of the GPUs are downstairs now, looks like about 20 or 30 are broken.

If this upstairs GPU farm is launched again, the following will be done:

1. Installation of awning and drainage on window to prevent rain from entering.

2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged.

Do you still have RMA and warranty on those cards?

This gave me a good laugh. Yeah i'm sure warranty cover damage of hardware if left under the rain. Wink

I don't think Ugt will let it happen again.

Gladimor

legendary

Activity: 1050

Merit: 1000

Quote from: kotarius on August 20, 2015, 01:30:42 AM

Most of the GPUs are downstairs now, looks like about 20 or 30 are broken.

If this upstairs GPU farm is launched again, the following will be done:

1. Installation of awning and drainage on window to prevent rain from entering.

2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged.

Do you still have RMA and warranty on those cards?

Cryptozillah

hero member

Activity: 687

Merit: 502

Damn. I hope some hardware is still ok !

MaxDZ8

hero member

Activity: 672

Merit: 500

Owch!

Seriously, just ask a mason to install a huge ventilation pipe 90deg turn on the outside. It is ridiculous you relied on pressure to keep the rain out and then pushed directly on the hardware.
As long as rain isn't being pushed up from the ground, no chance it's going to enter the building. No, an awning does not offer the same level of protection.

In particular, installing drainage is pretty much nonsense as it doesn't catch airborn water droplets unless you install a filter which will reduce fan efficiency. Plus drainage requires maintenance (albeit little, and that's probably something you're not interested in). I've never heard of such issues when the 90deg turn is used (it depends on how fast you suck in BTW).

Idea (2) is actually fairly good. Not sure 30 min is sufficient in case of catastrophic failure of the first layer of protections but I like the idea.

kotarius

sr. member

Activity: 340

Merit: 250

Most of the GPUs are downstairs now, looks like about 20 or 30 are broken.

If this upstairs GPU farm is launched again, the following will be done:

1. Installation of awning and drainage on window to prevent rain from entering.

2. Exhaust fan will be connected to the same 3-phase power supplying farm. Upon power loss and subsequent restore, APC power strip will be instructed to turn fan on immediately, and the rest of the machines only after 30 minutes. This way, if any rain gets in, it will be dried out long before hardware is damaged.

VirosaGITS

legendary

Activity: 1302

Merit: 1068

Well thats catastrophic, my condolences... xD

I recommend you invest in at least a 50$ Back-UPS. Or maybe a good 150$ one.
Seem pretty cheap to protect 20k$ of hardware.

Or/and just a tarp. You know you put a tarp outside on the exhaust and if the fan stop it fall down and will prevent crap from enter. During outward pressure making it in, the tarp will kind of get sucked in and seal the hole. 2$ fix. Ugly but cheap and effective.

noideacoin

newbie

Activity: 52

Merit: 0

How much did you lose from it? How many were salvageable?

Mikestang

legendary

Activity: 1274

Merit: 1000

Quote from: kotarius on August 19, 2015, 01:41:49 AM

There was a freak thunderstorm that shut down 1-phase power, exhaust fan stopped spinning and stopped producing outward pressure on the rain.

Ouch, that looks like a lot of hardware! I'm surprised you didn't have backup power on-site. I'm also surprised your setup only relies on positive air pressure to keep it weather-tight. I guess hind sight is 20-20, any plans to upgrade the facility so this doesn't happen again?

djm34

legendary

Activity: 1400

Merit: 1050

Quote from: jjjordan on August 19, 2015, 09:48:15 AM

Quote from: madmartyk on August 19, 2015, 08:25:43 AM

You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!!

You only need a pound of rice. Just scatter it around and wait for the Chinese to come and repair...

was about to tell to use silica gel instead... but yeah rice is better Grin

jjjordan

sr. member

Activity: 271

Merit: 251

Quote from: madmartyk on August 19, 2015, 08:25:43 AM

You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!!

You only need a pound of rice. Just scatter it around and wait for the Chinese to come and repair...

madmartyk

legendary

Activity: 2702

Merit: 1030

Yes I am a pirate, 300 years too late!

You need a truckload of rice to suck all the wet out of them!!! Man, that SUCKS!!

aa

hero member

Activity: 544

Merit: 500

Litecoin is right coin

Mmmm deep fried GPU.

kotarius

sr. member

Activity: 340

Merit: 250

There was a freak thunderstorm that shut down 1-phase power, exhaust fan stopped spinning and stopped producing outward pressure on the rain.

Rain got in and affected PSU, MOBO, and GPU, resulting in cascading failure. PSUs fried, motherboards shorted, graphics cards VRM burst. It was impossible to troubleshoot, hard to know which hardware component cause failure.

Luckily, empty GPU rigs (just barebones, no GPU) existed elsewhere in the building (downstairs). A decision was made to move the GPUs from upstairs to downstairs, into known working hardware.

Here is a picture of all the GPUs extracted:

Topic: 200 GPU farm got rained on, had to move it. (Read 1994 times)