Topic: A journey of extreme watercooling: Cooling a rack of GPU servers without AC. (Read 27312 times)

Did you buy the waterblocks yet for the rest of your GPUs? How is your project progressing?  This is awesome build!

Sweet looking setup bro.

Just wicked,  a watercoolers wet dream.   

An update.  Real job has been brutal last two weeks leaving me less time to work on my project.

I did assemble the rack though (got it off craigslist for $100 like 9 months ago).  The front inside rail is bent in a roughly 4" section which is why the PDU and switch is put mid rack.  



Pretty empty so far but hopefully more by next week.

I got the manifold mounted on plywood (which is going to mount to the side of the rack).

Yeah at least now.  As price/difficulty rises I can move to "normal volt" and eventually undervolting.

Right now the rigs get ~3 MH/W and my electrical rates are $0.10 per kWh. 

That puts my mining cost (electrical only) at:

MH per bitcoin =  (1498294.36282  * 2^32) / (50 * 1000^2) = 128702505.8
kWh per Bitcoin = 128702505.8 / (60 * 60  * 3 * 1000) = 11.9
energy cost per Bitcoin = 11.9 * 0.10 = ~$1.20

Gross margin is = $5.00 - $1.20 = $3.80

So while overvolting will reduce the margin I will produce more coins.  It is only once you start getting close to break even when overvolting becomes unprofitable. 

Overvolting is just something planned.  Once I measure power draw and stable clocks I can get some exact # of what my margins are at various over & undervolts.

TL/DR version:
My thinking is to overvolt and increase gross revenue.  As the rest of network becomes more efficient I can lower voltage to say ahead of the efficiency curve.  Eventually (say 12 to 18 months from now) I may need to be running at much lower hashrate and voltage into order to remain profitable.  I can dynamically balance efficiency vs gross hashing power based on the network's price/difficulty ratio.

You sure it is worth to overvolt considering the extra power draw?

looks nice btw

An update:
Had a minor leak at the radiator where the inlet pipe was knocked out of round.     Removed existing fitting, sweated on a 90 deg connector (which I wanted to use anyways, if you got to resweat the pipe at least make it an upgrade).  Was water tight for 12 hours+.

My Primoflex Pro LRT tubing (ID: 7/16 OD: 5/8) arrived.  A perfect fit for 1/2" PEX connectors.  Nice and flexible.  I updated the fitting on the rig to 1/2", attached the tubing put the inline no leak quick disconnect connectors and ran the loop to the manifold.

Had a minor leak when cutting water on.  The 1/2" barbs I bought don't fit on the "metal side" of the waterblock (they fit fine on the other end).    They threads are too long and they bottom out before compressing the o ring completely.  Luckily I had a pair of 1/2" barbs hanging around.   BTW large amounts of shop towels underneath all fittings made initial water turn on less scary.  Used hair dryer to remove any water (I didn't see any but I am paranoid).  Leak tested it for 12 hours.  Looks good.

Powered up the rig this morning.


Clocks are 830/150 at stock voltage.  (Most of cards were stable @ 870 but I decided to start low).
I am planning on eventually overvolting them slightly but that requires a bios flash.  I think 900/150 @ 1.125V should be possible which should give me 3.2 GH/s per rig.

Ambient is about 33C.  I have the garage closed with no AC to simulate hotter ambient temps of this summer.  I don't have my radiator fan yet so this is using a 14" desktop fan pushing minimal air through radiator.

Looking good so far.

Whats next:
Hopefully this weekend I can get the second rig connected to the loop.  Also want to mount the manifold to project plywood and start planning the PEX runs.  I also need to go grab the server rack I bought off a friend.   The next phase involves 20 danger den waterblocks (sound a bulk discount of $99 per block), 5 more server chassis, 50ft of tubing, and some fittings.

Yeah the tubing is the Lowes chepo vinyl stuff.  I got better tubing but no sense in cutting it up for test loops.  It was hard to keep it from getting kinked but I intentionally kinked some of them to increase resistance to simulate a server loop.  Not very scientific but luckily this pump seems have more than enough juice.

Most watercoolers measure resistance in terms of ft of head loss.  It varies by waterblock and finding exact specs can be tough but resistance is usually on the order of 1 to 2 ft of head loss per block at 1 gpm.  If the blocks are in series then the head loss if cumulative.  If the blocks are in parallel the head loss for the loop is the most restrictive.

There isn't a lot of information because really modern blocks aren't that restrictive and modern pumps have more than enough head.  This can make it tough to find good stats.  I have found reviews on pumps tend to have better stats on loop resistance than review on the blocks themselves.

Restriction (head loss) increases with flow rate but flow rates > 1 gpm really don't improve cooling much (1 gpm can transfer ~500 watts of energy w/ only 1 C rise in water temp).  The loop will reach an equilibrium between flow and resistance.  The goal is just to make sure that is >1 gpm.  I am planning for ~1.5 gpm flow through each loop.  6 loops = ~9 gpm across the main line.  The reason we want >1 gpm is to ensure turbulent (not laminar) flow through the block.  Older blocks required much higher flow to be turbulent but modern blocks have improved on that too.  

Sadly watercoolers today have it too easy.  Blocks are less restrictive, they avoid laminar flow even at low flow rates, pumps have higher flow and can handle higher pressure, larger tubing means less loop resistance, etc.  Compared to 10 years ago it is kinda hard to build a bad "normal" loop.  The bad news for us "extreme" builders is that means less stats on flow & pressure because for normal users taking any decent pump, any decent block, and any decent tubing likely will work just fine.

Beautiful. Those test loops look a little kinked, hope you will be using unkinkable hose for the final run. BTW, how much flow restriction should I expect from a waterblock? I'm sure normal pressure shouldn't be more than 1-2 bar, but when sizing everything up, the pump expert wants to know how much actual restriction there is, and I can't find anything relevant in the tech specs.

Some photos

The manifold.  5 test loops.  The balancing valves make it easy to keep flow even through all loops even if I create artificial restriction in one of the loops.  No rigs attached yet but hopefully I can get the first rig hooked up tomorrow and maybe second partial rig by this weekend.  If all goes well then I need to order 20 more waterblocks and for on the next 4-6 rigs.


The pump assembly.  Used two unions and the assembly to the left is the bubble eliminator MK II.  The larger pipe size slows the water velocity down which makes it easier to separate air.  Air flows up the T line which is where additional coolant is added. 


The radiator.  Also 4 of my existing aircooled 3x5970s rigs.  Will be nice to have some peace and quiet in the garage again.


The entire loop.  I am currently using 3/4" tubing but hopefully this weekend I can mount manifold properly to a sheet of plywood (which will mount to side of server rack).  Once everything has been tested in their final position I intend to replace the tubing with 1" PVC.

it probably will work but I am not positive.  That is why I used MSI 890FXA-GD70 motherboard.  It has 4 16x slots 2 apart. 

Likely you may need to do some trial and error to find something which works with a single slot.

I was thinking of adding a fourth card on one of my water cooled rigs. There would be no slot spacing between 2 of the cards though. I was thinking of using male to male adapter. You know if that would work? http://www.amazon.com/Enzotech-Rotary-Adapter-BRMM-G14-Black/dp/B007AIXE1M/ref=sr_1_36?s=electronics&ie=UTF8&qid=1332286492&sr=1-36

So an update.

The manifold:
I found a manifold for my rigs.  It is a radiant heating manifold.  I was at local plumbing supply house and after explaining what I needed  one of the employees found a returned open box manifold.  This is a $250+ part but was marked down to $120 due to some damaged.  Got them to drop it to $70 with no returns unless not water tight.    I will post some images tonight but here is a stock photos.  The blue caps at the top are balancing valves and the red tubes at the bottom are flow meters. 



One of the balancing valves is jammed, and two of the crimp rings are warped, one of the temp guages doesn't go above 80C.  It also has some cosmetic damages and tooling marks.  No idea what idiot tried to use this and why the store accepted the return but I got a lot of nice features for very little cost. 

Now it is designed for connecting PEX piping and the adapters for each zone are proprietary which makes it a challenge to repurpose.  Through trial and error I found that 1/2" ID, 5/8" OD tubing is very close to the nominal dimensions of 1/2" PEX and remains water tight even under pressure using the compression fittings which work with this manifold. I ordered some 7/16" ID tubing which I think will provide an even tighter seal.  Another option is changing the fittings on each zone to ones designed for 5/8" PEX which is a good fit for 1/2" ID, 3/4" OD tubing.  Since the tubing will be outside the server (connects quick disconnects on front of server to manifold) having a thicker sidewall may be better.

In testing I got 5 loops (no rigs connected) running @ > 1.6 gpm (the max on flow meters).    The balancing valves will make is very easy to ensure each rig gets at least 1 gpm to ensure turbulent flow.  Honestly despite being damaged this is a total score.

The pump and loop:
Hooked the pump up to using some PVC Unions.  At this point I am using 3/4" ID tubing (fits tightly over 1" PEX barbs) for the mainline because PEX isn't exactly flexible or easy to work with when testing.  Purging air from the system was a challenge.  I made a PVC "T" to allow a line for adding coolant and purging air. It only worked marginally.   It took hours to purge air from the loops and I don't think I got it all.  Went back to Lowes and picked up a 2" PVC T with some 1" threaded adapters.  The larger diameter means water will slow down as it enters the T allows the air to separate easier.  

The heat exchanger:


The good news.  It is massive and well built with a ton of surface area both the fins and the internal tubing.  The surface area of copper tubing is important because it is how heat is transferred from the water to the fins.  You can have a giant heat sink but if it only has a small amount of tubing it isn't going to be very efficient.

To put it into perspective your average 3x120mm water cooling radiator has 1/8" copper tubing usually 4 parallel lines which make one pass.  
Surface area of the copper tubing on 2x120mm radiator  = 2 * Pi * (1/8) * 14 * 2 * 4 = 88 in^2

This heat exchanger using 8 parallel lines of 3/8" copper tubing and makes 3 round trip passes.  
Surface area of copper tubing on this water to air heat exchanger = 2 * Pi * (3/8) * 20 *6 * 8 = 2261 in^2  

The bad news.  I bought it from an ebay seller and some fins were damaged.  I can fix them but I need a "rake".  Worse for some reason rather than capping the 1" copper tubes with plastic or rubber plugs they sweated bronze caps (using what looks like two gallons of solder).  In the process it looks like they knocked the tubes out of round.  I had a nightmare of a time sweating new fittings on.   I ended up cutting the caps off and having to bend the tubes back round.  It look a lot longer than I had planned.  Should have spent a little more because I certainly paid for any discount in manhours.

Disclaimer:
I hope these posts have shown that there is nothing "standard" about cooling a rack of servers.  Everything requires adjustment and modification as you go.  Luckily I own a lot of tools.  If I didn't I likely would have had to spent $1000 in tools and supplies (wrenches, channel locks, pipe cutters, soldering torch, mapp gas, tube cutters, pipe dope, thread sealant, etc).

What's next:
Will get some photos tonight.  Hopefully (work dependent) in the next couple days I can get the time to hook the test rig to the new cooling loop.  I have enough waterblocks to test a second 3x5970 rig maybe this weekend.  If everything goes good I need to pull the trigger on 20x 5970s waterblocks.

That was my original idea the major problem is that you now need n+1 pumps, radiators, and n heat exchangers.  Add in the additional cost of more fittings and the cost is simply not viable.

Each rig can be seperated from the main loop without affecting other rigs.  If the main loop is down then heat exchanger or not all the rigs will need to be idle anyways. 

That's a interesting idea, wonder how effective it would be. I'm a little skeptical about running everything in one loop. Repairs and downtime could be a major problem. Maybe one large loop that cools everything with each pc having its own smaller loop. Using heat exchangers to connect them.

Yeah I thought about that.  I even tried looking for smaller commercial ones as inspiration.  Seems they don't really exist for anything <20KW.  Even the 20KW units look pretty compact about the size of say 3 or 4 outdoor AC units side by side. 

The other option would be to bury 100m or so of tubing below the front line and have it act as a giant geothermal heatsink.

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

Something like that except I never run pumps without priming and I did have the common sense to not test it around the rigs.

Actually everything went fine in first two tests.  Then I figured I could test @ 10ft of head by putting the pump downstairs with a 5 gallon bucket, run discharge hose up the stairs to bathtub on the second floor.  Great plan but how to start and stop the pump.  So I give my wife instructions and warning about not letting intake run dry.  Everything is going good but I start thinking man this is a lot of water is the bucket almost empty.  I turn my head to yell down to her to make sure the bucket doesn't run dry (it didn't) and when I did I dropped the discharge hose.   Of course then it starts snaking around the bathroom and me trying to direct it towards the bathtub catch 20gpm of water to the face.

Short story is the pump "should" be more than powerful enough. 

Lol, I bet the sequence of events was something similar to the following:

Hell yes, the pump is here! *rips open the box like a kid at christmas*
Lets see if it works - *plugs in for only 3 seconds with no primer water, cause 3 seconds won't kill it, right?*
Sweet, it works. *runs off to the garage to attach some hose to the intake, and maybe a little bit to the output*
*fills up tub of water, primes pump and plugs in again*
Oh shit son, water everywhere hopefully none got on my rigs shitshitshit
*stupid grin while cleaning up the mess*

About 90W.  In a "normal" water cooling system that would be excessive.  Given I have a heat load of ~6KW, 90 more isn't much.  One way to look at it is I am replacing 24 blower fans @ 8W ea = 192W with a 90W pump and 40W fan.

The pump did arrive today.  UPS ground and it got here in 2 days.     It is impressive.  Everything about it screams build quality and performance.  Did some testing and ended up soaking myself.  Even at 10ft of head it moves a ton of water.   Hopefully I can get the first rig hooked up to the main loop this weekend and then order parts for the rest of the rigs.    No time to waste we already have 80F weather.