Topic: A journey of extreme watercooling: Cooling a rack of GPU servers without AC. - page 5. (Read 27312 times)

Not a thread jack at all.  If it is that exact one (RM41300-fs81) I am interested, PM me. 

Chenbro makes other RM41300 models but they only have 7 expansion slots.

Not to tread-jack to much. But I have 2 of these (more in storage) for sale for WAAAAAY less than that price.

Controlling a farm from single server.

Most farms are run as a set of independent rigs.  With watercooling and a shared outer loop that presents some challenges and risks.

The farm needs to shutdown in the event of a LOCA (to borrow reactor terminology http://en.wikipedia.org/wiki/Loss-of-coolant_accident).   Now including software on the rig which shuts down the rig when its temp rises is possible but not a solid failsafe.  Rising rig temps can cause processes to hang which could keep rig online cooking itself without cooling.  Having each rig monitor sensors creates a lot of duplication.

So my plan is to have a daemon on each rig monitoring GPU temps and if they rise above a failsafe to start a shutdown of the rig.

Backing that up a non-mining server "watchdog1" will provide analog monitoring of the entire farm.  It will also hold copies of all rig config files, collect stats from cgminer API, host p2pool & bitcoind daemons (and possibly act as a PXE server).

I plan to have:
4 outside temperature sensors (water before radiator, water after radiator, air intake, air exhaust)
1 ambient air sensor
1 flow sensor
1 reservoir level sensor (contact only)
1 pump current sensor

Now when watchdog detects something "abnormal" it could send a "emergency power off command" to each rig but we run into the same risk if a rig fails to shutdown.  So instead I am building a power control board which will allow the watchdog to power on, power off or power cycle all the rigs.


Multiple "dumb" quad 5970 water cooled rigs (initially 6).
&
"Watchdog" supervisor to keep them safe.

Yup that is the one. 

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

Will do.  I have a crimping tool from some home remodeling work already.  I will likely make a test loop involving just pump, radiator, and heat exchanger and leave it running.  The reason I would like to use PEX is I could take an off the shelf radiant heating manifold and use that for splitting the run to each rig.  

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger



Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer. 

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

webmon:

Is that a customized version of cgminer?  The frequency & voltage columns.  I like that layout.    I think current layout in cgminer needs some work.  The [g]pu menu takes up too much vertical space especially w/ 8 GPUs.

gota love the 5970's..  this one has been up since the 4th, restarted it then for a cgminer upgrade.

That is a little beyond my abilities in terms of earth moving and engineering but a pretty awesome concept.  If I had more land I might look into that for my entire house.

@DAT
You can reduce your cooling cost by using the earth as a heat sink and temperatures will always be in the high 50's or low 60's.
http://earthairtubes.com/

Your cost of operating this is the cost of running a fan. sweet!

If you've got reference cards exhausting heat through the inlet side, something is horribly wrong.  There shouldn't be any.


Er...  that's because it IS the exhaust side (on reference boards...  6990 is a different story). 


Sounds like you're using non-reference cards?


That's exactly what you should be doing.  It looks like your fans are blowing at the gpu interface side though?


My setup deals with the exhaust heat perfectly.  My hottest card as I type is 74 degrees and the fan is sitting at 40 percent.  This is because it's in the garage.  The non-reference cards in the house are running around 64.0C at only 1166RPM.

The exhaust heat from all of the cards gets pushed out into the hallway and back out the window of an adjacent room.  Since it goes through the hallway, some of the heat gets recirculated throughout the house.  I haven't used the furnace at all this winter (and it gets very cold here).  This type of setup requires that nobody opens a window somewhere else in the house though...  otherwise, the air will go in a different direction.

My shelves are sitting on sliders, so when summer comes around, I just flip the rack around and exhaust the heat out the window.  It gets in the high 90's here and I was cooling the whole rack with a 230 watt evaporative cooler last summer.  The rigs in the garage will be a problem though if I don't move them before summer hits.  I'm building a small shop though to house them all.


Starting...   now?   

Yeah, you've certainly got a sturdier rack built.  If we both had an earthquake, you'd win.  We don't really get any here though. 
I'm going for cheap & effective, and my time is a factor.  Setting up a new rig takes me about 20 minutes from unboxing to mining.


  It wasn't meant to be a tread hijack.  I just saw someone with blowing fans at the exhaust side of their cards and couldn't help myself.  I had a non-ref card die a while ago and used a reference card in its place for about a week while waiting for a replacement.  Because the card had to face the wrong direction in relation to the others, that GPU fan had to REALLY work hard to remove the air.  Flipping the card around would have *easily* cut the fan speed in half.

I'm very happy to see the design replicated!

I kept thinking how almost every detail was spot on, it would have been a huge coincidence if you never saw my rigs. Why is that vortex fan green? As far as I know the company only makes them in gray.


I actually thought a lot about this. Just through touch I could feel that each end of the card was exhausting heat, even more so for the power cable side (away from the interface). I, like you, always thought of the interface side to be the exhaust side. I even took a smoke machine and introduced some smoke at the fan intake to see where it went. It was about an equal distribution to both sides. Couple that with the fact that I wanted access to the interface side, along with the USB and other components on the motherboard, I decided to suck the heat away from the motherboard/gpu interface.

From the photos, you can see that I sandwiched the GPUs as closely as I could. I did this to maximize the airflow across the GPUs and increase the rate at which fresh air was introduced to the fans.


Your setup looks very clean, but doesn't deal with exhaust heat at all. Isepick and I are down south and heat will become a major problem in a few short months. You also use many fans. I think using one big powerful fan to deal with all the heat is the way to go.


Starting now.

Looks pretty clean. I think my cases are a tiny bit sturdier than yours though. I have also removed every extra fan out of my setup by undervolting a few rigs. Maybe I'll make a video.

Now back to our regularly scheduled thread topic.

Then both of you need to clean up your rigs.  Sheesh.     Okay, I don't know about Brian's, but what is going on in that picture?  There are fans blowing at the *exhaust* side of quite a few cards!  The fans are just fighting each other..

This is what halfway clean rigs look like.  Pink rope is a requirement:

   

ROFL. I understand, although when the door to the room is closed you can't really hear it so much, and when it is open its more of a white noise type of deal. And I have to credit Mr. DeLoach for the design, as I probably wouldn't have thought of it on my own . Anyhow, good luck, I am looking forward to seeing where you end up with all this.

An update.  

Rig has been solid for 36 hours now.  I have increased the clocks and get ~ 3.125 GH/s on 1120W 120VAC.  When moving it to 240V PDU wattage dropped to 1090W (2.86 MH/W).   Backing out the system non-GPU idle of 180W puts GPU load at ~230W per card giving GPU efficiency of 3.40 MH/W.

If we assume ~10% PSU inefficiency that's 980W DC putting PSU load at 78%.  Since it is a SeaSonic I am comfortable with it.  The SeaSonic 1250 hasn't disappointed.  Exhaust is warm but not excessively even with the cramped airflow of a rackmounted case.

I have been running cgminer with auto-gpu enabled and a target temp of 62C.   The dynamic is kinda interesting.  The cards are rock solid but as one card goes over 62C it gets the downclocked by cgminer.   The reduced thermal load cools all the GPUs and puts one of them under the target so cgminer raises the clock and and the process repeats.  Kinda a "musical clocks".  The load remains ~1.1KW and hashrate ~3.1GH/s the entire time with different GPUs taking the lead.

The data is showing (which I already knew) that this radiator is simply not designed for a heat load this large.  The fact that I have low RPM fans isn't helping.  I won't be able to push clocks higher without better thermal dispersion but based on the solid performance I think 3.25 GH/s is possible at stock voltage.  I will try using some Scythe Ultra Kaze temporarily to see if I can get the last little bit out of the radiator but we might be reaching the limit of what 4x120mm radiator can dissipate..

Somewhat disapointing is that one GPU won't go above 800 MHz even with low temps.  It just goes instantly sick.  Sadly raising voltage in Linux isn't possible without BIOS flash.  Hopefully cores like that are a minority.