Topic: A journey of extreme watercooling: Cooling a rack of GPU servers without AC. - page 2. (Read 27312 times)

less than 1/2 of 1 GPU equivelant.   These things are used a lot for salt-water fish tanks.. they are very efficient pumps. 

Sigg

I'm curious as to how much heat that pump is going to add to the loop. 

I was/am using BAMT for the 3x5970 air cooled rigs  but it is x86 only which means it won't work w/ 8 GPU for the watercooled conversion.  For the first watercooled rig I am using a modified version of LinuxCoin.   I may consider a custom distro at some point.

The farm is on p2pool (well part of it is right now), but p2pool is not installed on each rig.  The "watchdog server" runs bitcoind and p2ool.  My workstation runs a backup copy of bitcoind & p2pool.  Each rig has first p2pool instance as primary, the second p2pool instance as secondary and bitminter as tertiary mining pools using failover.  Something that is a low priority but I would like to look into, is using a static distro which is reconfigured on each boot.  There would be no USB drives.  The rigs would simply boot from LAN and pull the distro from a PXE server (watchdog again).   Once booted the rig would pull a cgminer config file from the server.

My goal is to make the actual rigs as "dumb" as possible.  Turning them into 3GH/s hashing appliances with control/config/monitoring done by the watchdog.  It looks like my pump might arrive today.

So this thread is mostly about the hardware, but I'm curious what you use for the software.

I know you've mentioned that part of your farm is mining at p2pool.  Do you use BAMT? Linuxcoin? Your own installs?

Since my last post i discovered, that my sempron was unlocked to Atholon II and it maight be problem of the all GPU crush. seccond rig is stable after changing intensity to d (jumps 3-6) - aprox 20mhash loss :/ but stable for last 24h.

rig was simple to get on with ssh, but all gpus was down and i had to reeboot it. xorg 99.9% of cpu bug
thats why i was digging in the bios and i figured the unloced core.

thats haw i become 2x2.5Ghs 100% Watercooled rigs owner

I`m using 12V laing`s with modifed electrics - pulls 20-30W (depands on Voltage) and makes low noise and preaty nice flow.
My temps are aprox 51-55 with just 3x120Fans on old 3x120x2 (2 radiators puted toghether).
My system was build from scraches. I paid really low prize for it, for example: 1 block cost me 60$, pump 60$, 2 new blocks costs me 180$ each, block for 5870 costed 30$. 2 Radiators and other accesiores i had for a long time working in my old systems.

OK, you found the other brand that is a clone of the Iwaki.  I knew from my saltwater aquarium days there were 2 Iwaki clones, one was Pan World and the other I couldn't think of for the life of me so I didn't list it.  I would consider a BlueLine a contender also.  They all share excellent the same engineering specs, with different people actually making them.  Fake edit: I see you bought a used PanWorld 150.  Badass!  As long as the price isn't outrageous it is still cool that you got an overkill pump.  And yes, rjk, VFD would be the ultimate luxury on a setup like this.  Totally unnecessary, but we are nerds, so every improvement is cool!

Should put one of these in the system: http://www.marinedepot.com/Digital_Aquatics_ReefKeeper_Lite_Controller_Multi_Item_Monitors_Controllers_for_Saltwater_Aquariums-Digital_Aquatics-DA1131-FITEMOMI-vi.html . I mean, wouldn't it be cool to monitor the pH, temp, and *salinity* of your *freshwater* cooling loop? Im just kidding man, seriously, just kidding.  I want to see this project grow.

Man it would be cool to have a 3 phase pump on a PID loop with a VFD. Then you could add and remove rigs with no need to worry about balancing anything out (for the most part). The pump would just slow down as more resistance is added (closing off valves to swap a rig, or whatever) or speed up if you add more rigs to the loop.

Finally pulled the trigger on a Pan World 150 PS.  Found one at a good price (cheaper than other places are selling 100 PS)



Looks like 12 gpm @ 14 feet of head.  Should handle the resistence of 1" PEX mainline just fine (saving me the cost and tedious work of assembling 70 ft of CPVC).  I am hoping it gets here my Friday but it might not arrive till next week.   Come on Fedex lottery.

The 100PX is interesting.  In searches for Pan World the brand "BlueLine" keeps coming up.  Looks very similar to Iwaki.  

HD 40
http://www.aquacave.com/Blueline-40-HD-ExternalAquarium-Water-Pump-by-Pan-World-P488.aspx

HD 55
http://www.aquacave.com/Blueline-55-HD-External-Aquarium-Water-Pump-by-Pan-World-P490C762.aspx

Lots of merchant indicate Blueline is produced by Pan World but Pan world website has no reference no Blueline.



If these specs are to be believed then BlueLine might be the pump I need.

Even if we take the specs at face value it is 8 gpm at 4 ft of head.  No curve or max head is provided but I imagine that is the sweet spot.  Likely sub 4gpm at 8ft of head and 0gpm at <15 ft.  I don't even see that being in the same league of the kind of pump necessary to cool 8 servers in parallel w/ 8 GPU each in series, through 70ft of tubing to a massive radiator with 64 ft of 3/8" copper tubing. 

Oops, sorry, the 100PX is considered comparable to the MD40 - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html .  I was looking at flow curves instead of the feature briefs.

Does this satisfy you need to have a massive ass pump? - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html  hehehe, but seriously the price is way better for the same quality.

Even though this doesn't meet your head requirements, I would really like to see what it would do in your actual finalized setup - http://www.marinedepot.com/Aquarium_Systems_Premium_Maxi_Jet_3000_Utility_Pump_775_gph_Up_to_500_Gallons_Per_Hour_Submersible_Water_Pumps-Marineland-AS1311-FIWPSBUF-AS1319-vi.html .  You could sell it to the aquarium nerds if it doesn't work =P  Take into account also that your plumbing on the pump gets reduced to just a pipe that dumps into the sump, and then a hose straight off the outlet of the pump...instead of all the fittings required to get an external hooked up properly, plus union valves to make it easier to service, etc.

Do a search for "Panworld pump" .. they're the "inexpensive iwaki's" ..  I've got 2 of them running on my saltwater tank that have been running nonstop for 7 years now.

Sigg.

Have you considered running two pumps in series? You would gain additional flow, and redundancy in case of pump failure.

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

I am not sure what you are trying to say and I am not sure you do either? 

The flow rate at given resistance is what we are discussing.  I am not sure what you are advocating differently?

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

You should be looking at flow rate instead of static head pressure. Static head doesn't matter in a closed loop.

Look at the pressure drop (friction head) of each of your components to pick a proper pump.

EDIT: Also if you want a nice pump, get this. http://www.pexsupply.com/Grundfos-59896773-UPS15-55SFC-Stainless-Steel-3-Speed-Flanged-Circulator-Pump-1-12-HP-115-V

Have three of them for my radiant floor heating, one for each zone. They have no problem pushing through hundreds of feet of pex without skipping a beat.

I am shooting for 55C to 60C temps.  Remember though this is nearly 7KW in heatload and outside temp peaks at 35C.  To keep 7KW of heatload at 25C over ambient requires more cfm, flow, and surface area than you may think.  I did some heat flow calculaitons and to avoid needing 2000 cfm+ requires pretty significant gpm on the liquid side.  Sure cooling in the winter is easier but we need to make it through August in Virginia Beach first.

On edit (found this looking for info on flow rates):
http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphad/watercool9125-F2A.htm

Looks like IBM recommends 22gpm per server rack and 33 psi (~14 ft of head) of pressure.  Granted no exactly apples to apples (they are also using water chiller to lower input temp of the coolant to <20C) but thought it was interesting comparison.

I do believe I need 10gpm @ 15ft of hea.  For expansion and some margin of safety in flow rates (due to balancing) I would prefer 20gpm or as high as economically possible @ 15 feet of head.   Remember each rig is in parallel so 10gpm across the mainline is only 1.5gpm across the server if the minimum of 6 servers are used.  If that is expanded to 10 servers (~32 GH/s) for a full rack it is only 10 gpm mainline = 1 gpm per server.

I already used a flow/pressure calculator:
The server has 4.2ft of head loss (@ 2gpm) measured by me.
remember 8 GPUs worth of blocks in serial is rather restrictive this isn't your "normal" loop.

As I don't have a manifold yet I am reserving 1.0 ft of head loss for the manifold (and various fittings).
The heat exchanger has 2.2ft of head loss at 20gpm (thankfully it has 1" inlets and 8x 3/8" copper branches).
The mainline as planned is 70ft of 1" PEX.  At 20gpm that is ~8ft of head loss.

4.2 + 1.0 + 2.2 + 8 ~= 15ft.

So options:
Options 1) Could use a pair of radiators in parallel to reduce the flow across each and the corresponding head loss.  At 10 gpm the head loss is only ~ 1 ft. Still I doubt that saves much money when you consider the additional cost of 2nd radiator and fan assembly.  

Option 2) Use a less restrictive mainline.   1" PEX is just so cheap and easy to use.  Going to 1.25" CVPC is an option but a lot more work installing.  

Option 3) I could just accept lower flow across each rig but I worry that if I go that route flow isn't going to be perfectly balanced and one rig ends up getting too low of a flow.  That would require either putting in expensive balancing valves on the manifold or buying a larger pump.

Given those options I would rather not skimp on the pump and end up needing to waste time and money putting a second radiator in parallel, dropping in balancing valves, or re plumbing a larger mainline (1" PEX = $0.80 per foot.  1.5" PEX = $3 per foot ).  Going "cheap" on the pump may save $100 or so but it may end up costing three times that in extra work and modifications.

On evaporative cooler:
I don't think any home grown evaporative cooler is going to handle 7KW of heatload.  Regardless the summers in southern VA approach 90% humidity so I think a forced fan heat exchanger is going to be superior for year round operation.

I don't mind looking at alternative brands I just haven't found anything that gives me what I need for less.  No doubt it likely exists so if you have a pump model in mind I am all ears.  I have found some industrial pumps but they tend to be lower efficiency and dump as much as 500w into the loop which make them useless.

Your thoughts?