Topic: (Updated w/ pics) Watercooled Rack of Servers - 50% completed - page 5. (Read 10593 times)
Looking good so far. Do you have any pics of the other end of your watercooling setup?
Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.
Yeah the bottom one is done the way I want it. I took the 3.5" plate and drilled 2 holes. One half of the quick disconnect has a panel mount adapter so it mount directly to the plate. The plate is then monted in the drive cage. This makes the internal tubing loop fixed. Less chance of moving around, stressing the line, or causing a leak. Right now if you yanked on the tubing for the bottom rig the worst you could do is cause a leak outside the rig.
However the steel is pretty tough. It wore out my cheapo bit after the first plate. Need to get a stronger bit to drill out the rest.
Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.
That thing looks so sexy. I want to see it full of shinny electronics.
So the original thread is here (but the plan changed a lot and the thread got sidetracked at points w/ flames & trolls)
https://bitcointalksearch.org/topic/a-journey-of-extreme-watercooling-cooling-a-rack-of-gpu-servers-without-ac-66606
The Purpose (also know as "are you fraking crazy FPGA are going to take over the world ... ")
I ALREADY have a farm of six 3x5970 rigs plus another 6x 5970s cards looking for a home. The hardware has long since been paid off and I have some measure of pride knowing the rigs I built are just about as efficient as is possible with GPU tech (although some optimizes 7970 rigs might edge them out). The goal is to extend that investment as long as possible. Kinda a GPU "last stand" against the encroaching FPGA hoards
My goal was to find a way to:
a) survive the summer without the 1/3rd increase in electrical cost
b) avoid paying ~$3K to the upgrade AC system (farm was expanded in the fall so current AC won't be able to handle it)
c) have fun building a big insane system.
d) see if it would be possible to capture some of that heat for preheating hot water heater and/or heating the house.
So I had this crazy idea. Can you put 24+ HD 5970s in a standard server rack and using a really powerful pump move all that heat (18,000 BTUs) outside?
Well the farm is 50% converted.
The Rack
From top to bottom is APC PDU, management server, gigabit switch, second APC PDU, and the first three water cooled rigs.
Closeup of the rigs
No I am not cooling it with milk. I guess the flash made it look like that. Just using plain distilled water.
Connectivity & Power
I need some better cable management but power on left, connectivity on the right is a good start. I hope to eventually replace all the flash drives with PXE.
Manifold
First three rigs are hooked to the three left most ports. Isolation valves on top, flow control valves on bottom.
The connected loop on the right ensures even if I absent mindedly disconnect all 3 rigs the pumps will still get about 1gpm flow (important to keep the pump cool). The photo is kinda hard to tell but this is mounted on the right hand side of the rack when looking from the POV of the first photo.
http://i.minus.com/iBLM12pskhT2g.png
The pump
A real monster but quiet and has a nearly silent fan in the back which reduces the amount of heat which ends up in the loop. A pair of unions simplify hooking it up to the loop. The pump and fan are the only component which can't run on 240 which necessitates running 120V extension cord.
The radiator
Has about 16x the pipe surface area as your "normal" 3x120mm radiator and about 6x the fin surface area.
I am still using the poly tubing but will be replacing it with PEX when I get the time. Fan is just a "normal" floor fan. Probably 1800 cfm. I may replace it with a car radiator fan or maybe just a outdoor rated workspace fan.
The rack contains:
3x quad 5970s rigs
2x APC 9571 PDU (240V, 30A, 80% derate) for 11.5 KW usable power
1x Cisco 24 port gigabit switch
1x 1U watchdog server (runs bitcoind, p2pool, remote power relay card, and monitoring software)
Current config:
12 5970s (24 GPUs) ~ 9.2 GH/s
When complete:
24 5970s (48 GPUs) ~ 18-20 GH/s
Theoretical max:
40 5970s (80 GPUs) ~ 30-32 GH/s
https://bitcointalksearch.org/topic/a-journey-of-extreme-watercooling-cooling-a-rack-of-gpu-servers-without-ac-66606
The Purpose (also know as "are you fraking crazy FPGA are going to take over the world ... ")
I ALREADY have a farm of six 3x5970 rigs plus another 6x 5970s cards looking for a home. The hardware has long since been paid off and I have some measure of pride knowing the rigs I built are just about as efficient as is possible with GPU tech (although some optimizes 7970 rigs might edge them out). The goal is to extend that investment as long as possible. Kinda a GPU "last stand" against the encroaching FPGA hoards
My goal was to find a way to:
a) survive the summer without the 1/3rd increase in electrical cost
b) avoid paying ~$3K to the upgrade AC system (farm was expanded in the fall so current AC won't be able to handle it)
c) have fun building a big insane system.
d) see if it would be possible to capture some of that heat for preheating hot water heater and/or heating the house.
So I had this crazy idea. Can you put 24+ HD 5970s in a standard server rack and using a really powerful pump move all that heat (18,000 BTUs) outside?
Well the farm is 50% converted.
The Rack
From top to bottom is APC PDU, management server, gigabit switch, second APC PDU, and the first three water cooled rigs.
Closeup of the rigs
No I am not cooling it with milk. I guess the flash made it look like that. Just using plain distilled water.
Connectivity & Power
I need some better cable management but power on left, connectivity on the right is a good start. I hope to eventually replace all the flash drives with PXE.
Manifold
First three rigs are hooked to the three left most ports. Isolation valves on top, flow control valves on bottom.
The connected loop on the right ensures even if I absent mindedly disconnect all 3 rigs the pumps will still get about 1gpm flow (important to keep the pump cool). The photo is kinda hard to tell but this is mounted on the right hand side of the rack when looking from the POV of the first photo.
http://i.minus.com/iBLM12pskhT2g.png
The pump
A real monster but quiet and has a nearly silent fan in the back which reduces the amount of heat which ends up in the loop. A pair of unions simplify hooking it up to the loop. The pump and fan are the only component which can't run on 240 which necessitates running 120V extension cord.
The radiator
Has about 16x the pipe surface area as your "normal" 3x120mm radiator and about 6x the fin surface area.
I am still using the poly tubing but will be replacing it with PEX when I get the time. Fan is just a "normal" floor fan. Probably 1800 cfm. I may replace it with a car radiator fan or maybe just a outdoor rated workspace fan.The rack contains:
3x quad 5970s rigs
2x APC 9571 PDU (240V, 30A, 80% derate) for 11.5 KW usable power
1x Cisco 24 port gigabit switch
1x 1U watchdog server (runs bitcoind, p2pool, remote power relay card, and monitoring software)
Current config:
12 5970s (24 GPUs) ~ 9.2 GH/s
When complete:
24 5970s (48 GPUs) ~ 18-20 GH/s
Theoretical max:
40 5970s (80 GPUs) ~ 30-32 GH/s
Jump to: