Topic: A journey of extreme watercooling: Cooling a rack of GPU servers without AC. - page 3. (Read 27312 times)

Oh cool, you are still deciding on the pump.

I'm going to be fairly blunt about this.  What makes you think you need to pump ~15ft of head at 450+ GPH?  Have you calculated your total head based on real height plus all your fittings and doo-dads?  There are calculators out there, btw.

Based on my experience with even shitty aquarium pumps, they are all underrated on what they will flow at their "supposed" maximum head height.  I think you would be surprised what a ~50-75W submersed pump can do even "choked" down to 1/2" tubing.  Don't get caught up too much in the system designs for water cooling in the overclocking computer world.  They tend to grossly oversize EVERYTHING in search of dropping that last 1 degree.  I was doing computer watercooling before you could just go out and buy a kit in a box.  Some of the setups the enthusiasts started making in the early 2000's up to now are downright hilarious on how oversized everything is.  Some of the more commercial application are more realistic, with way smaller tubing, pumps, and radiators.  I think if you shoot for 50-60C temps at the GPU itself, instead of 2C above ambient like some guys, you would be surprised how cheap your initial investment gets on the equipment, host fast you get your ROI, and how long everything would last.

If you live in a dry environment you could even look into evap cooling to dump the heat instead of a radiator.  You know, how nuclear power plants are cooled?  A while back people designed PVC "water bongs" to blow air up over water that was showering down into a sump.  I made what I called a "bucket bong" that did the same thing in a 5 gallon bucket, just a shower head and an 80mm fan.  The advantage?  Your water coming out of the "bong" was 1-2C *colder* than ambient.  The disadvantage?  You have a bit of humidity to worry about in the room you dump the heat in.  You could either plumb it outside, or just dump some of the air outside with another window fan again, lol.

Entire rigs are crashed?  Can't SSH into them and see what happened?  That is weird.  I would turn on logging in cgminer and try to see what happened.  If you need help on logging in cgminer use the cgminer thread.  As a test I would take every card to stock and with a higher memclock of say 280.   Get them running 48 hours+ before you start playing around with overclocking.

My 4x5970 rackmount chassis has been rock solid.  Significantly overclocked it hasn't had any downtime since I started this thread.  No crashes, no failures, no noise just a continual supply of new shares.  The longer that test rigs runs solid, cool, and quiet the more convinced I am of this project despite other setbacks (like radiator arriving damaged, and trying to find a cheap manifold, etc).

Not to late.  Haven't pulled the trigger on the pump.  Was doing work on the heat exchanger and sweating npt connections this weekend.

Show me a pump with
a) similar curve as a Iwaki MD-40
b) has 1" NPT intake and discharge
c) has similar heat dump
d) has significantly lower price

and I am interested.  I have looked any haven't found it.

Everything I have found either has horrible head (most aquarium pumps) or is horribly inefficient (most sump dumps) dropping 300W to 500W of heat into the loop or is more expensive than the Iwaki.

If you found a good pump that can get similar flow as the Iwaki MD-40 at 10ft to 15ft of head let me know.

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

I`m having strange problem here with system 6GPU+
made 2 systems yesterday, and bouth got it.
GPU`s are shutting down randomly....
I was testing few OS`s: xubuntu, linuxcoin, W7(WTF? - i was desperat)
 

1 rig - 3x5970+5870  - OCZ ZX 1250 - all GPU down after 7h - i was sleeping
2 rig - 3x5870+2x5970 - ENERMAX 1250 + sommething 550W - 3 GPU down after 7H

Both rigs has 890FXA-GD70 and GPUs are watercooled.
Cheked watteg with KILL-O-WAT but only rig no.1 - takes 980W from the wall - 5870 - 950, 150 and 5970 - 750,150 - so not a big deal. temps - max 54. Cheked vrm tmp under W7 - each GPU - higher by 10*C so max ~65...cool and nice, so why down ??
any ideas ??

I didn't feel like elaborating further, minerals could be a real problem though if you had to keep filling your radiator because of a leak or something.

Except for that you're missing a step to distill water. Yes you boil it, but what you are collecting is the steam from what you are boiling, which then condenses back into water. You are not just using the water you have boiled directly. And yes, it is a good idea to use distilled water in a car just for that reason. A lot of times non distilled water has minerals in it that will leave buildup/corrode the cooling system components.

Yes.  Remember PSU are rated on DC load not AC.  It is ~1100W AC.  At ~90% efficiency that is <1000W DC.  No problem for a SeaSonic 1250W unit.  Now lesser brands might choke up but any top tier supplier should be able to provide 100% power continually.

Your 1250W PSU is handling this 4 x 5970 smoothly!?

wow, i didn't know distilled water is just boiled! OMFG!! j/k

I've even built my own water blocks.
The commercial products i've seen mostly use 10-15mm hosing. Many of the water blocks has been very simple, hose in, hose out, just a big water space inside the block, and some had ribs to have bigger surface area.


Good quality water pump is a must, i've never said you should skimp on water pump. Not once. Not even hinted.

No matter how you put it, 800-1100watts is still a small heat load relatively, and one definitively shouldn't over complicate things. Keep It Simple, Stupid! or K.I.S.S.

I can see the point in adding the silver coil, or even "dead water" or other chemical to kill bacteria, but i don't see a point in over complicating simple things. None of the commercial single system water cooling package i've had the displeasure to work with had nothing like this. One had so bad additive it kept on accumulating in corners and i had to spend significant amount of time running hot water through it to get that stuff out.

Use copper blocks, use brass fittings if you go with standard hydraulics fittings (ie. NPT, BSP) hell maybe even AN-fittings (aluminium btw!) even tho that's total über overkill, or maybe just use basic hose barbs (1.5-3$ each) and clamps. Go with a high quality pump with a backup pump (2 parallel) so no need to monitor constantly. Use basic standard rubber hose, if you want to be cheap even garden hose will do the trick (yuck).
Add a little bit of glycol for protection of the metals.
and do *NOT* omit periodic maintenance (ie. once a year)

tho adding dead water and silver coil costs next to nothing vs. the other gear. So why not add them.

I must admit tho, i hate water cooling generally, makes maintenance a bitch in regular systems, but for servers (or bitcoin mining cluster) it makes tons of sense.

You should take a look at how industrial water cooling systems work ... They are remarkably simple by design. Even those cooling tens of thousands of servers.

I started fooling around with water cooling when Thunderbirds were a new thing (or maybe even earlier ...), didn't like it then for single systems, don't like it now. Tho a new corsair's set made it remarkably simple, infact so simple i might consider that for myself too, after i've seen how a system like that works later down the road (maintenance wise): http://www.corsair.com/cpu-cooling-kits/hydro-series-water-cooling-cpu-cooler/hydro-series-h100-extreme-performance-liquid-cpu-cooler.html
Just installed H100 for a customer

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

My heat exchangers arrived today.    My wife wants a night out so next update will need to wait till tomorrow.

Whew, there is a lot of FUD in this thread but it appears you at least have a solid plan. Mostly curious if you'll try to harvest the waste heat and how you'll go about doing it. I know you mentioned pre-heating your waterheater, going to stick around to see what you do. I myself built out a small farm of 8 watercooled 5970s. Hopefully you don't buy up all the DangerDen 5970 waterblocks!

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

I'm not interested in race car water cooling. You don't need glycol to keep bacteria out of the loop. Dead water is just a biocide additive that keeps bacteria out of the loop similar to silver. You can find it about anywhere that sells water cooling parts. http://www.frozencpu.com/products/11744/ex-liq-154/IandH_Dead-Water_Copper_Sulfate_Biocidal_PC_Coolant_Additive.html#blank

Yuck

Simple answer is to not used mixed metals and then you can simply use pure distilled water (plus silver coil as a natural biocide).  Too easy.

If you must ignore the simple answer then remember that glycol has a lower heat transfer than water.  75% glycol is just stupid.  

More glycol means:
a) lower heat transfer (and higher pump resistance) - undesirable.
a) more corrosion resistance - unless you intend to not change loop for a decade even 5% should be fine.
b) higher boiling point - nope doesn't really apply here.  If you loop is getting to >100C you got problems.
c) lower freezing point - unless the radiator is outside it is a non-issue.  The container will tell you the freeze protection at various strengths.

so use the lowest amount that meets your needs.  

I didn't mean to imply it was impossible however I intend to run a pure distilled water loop for maximum efficiency.  Watercooling is complicated enough adding mixed metals is adding extra optional complication.  For 99% of users it is simpler/easier to just skip the aluminum parts go all similar metals and bypass the entire issue. 


I have no interest in using antifreeze.
1) It is a pain.
2) It has to be regularly changed
3) Corrosion will occur if it isn't changed or concentration isn't sufficient.
4) It has much lower heat transfer capabilities compared to distilled water which lowers the efficiency of entire system requiring stronger pumps, larger heat exchangers, and faster/louder fans.
5) I will be using a heat exchanger to heat up potable water for human consumption which will require the more expensive non-toxic propylene glycol and corrosion resistance is significantly reduced.

So hopefully to not derail this futher:
a) yes you "can" use anti-freeze.
b) no I don't recommend it because today it is too easy to build a water cooled system with no mixed metals.

Dude, obviously you do not have a clue about the underlying physics going on.
FYI, i've even made blocks myself BEFORE watercooling became anywhere near so popular that you could buy anything watercooling related. Back then i were already using compressors to drive -34C coolant to the CPU. Did you know that you can get by just fine with low pressure, low volume pump (cheapest 12V you can find) if your block is properly massive and TDP is below 100W?

Btw, "stop-leak" and other stuff you put in to stop leaks is the stupidest idea ever.

Do you know what is the coolant of choice of extreme high output racing engines which actually partake in sanctioned races?
Now i'm talking about engines like 2L Inline 4 boosted to some 4bar, Cummins 5-8L diesel boosted to 7+bar?
Nope, not distilled water.

TAP WATER, with maybe some water wetter.
And these machines need to be capable of removing heat in the multiple MEGAWATT range.
Are you telling me water is not sufficient to remove your couple hundred watts of heat?

As for the dirt accumulation, i've only seen it once, and it was with off the shelf system with which came the "water additive", very small amount to be used, i was shocked to see that accumulation and i somehow doubt that would happen if proper glycol was being used. I suspect the additive that came with it was just some coloring.

The thing is with water if you add a little bit of heat, but not much, and circulate it constantly, it WILL have bacterial growth, that's why you need to add enough glycol.

Still, WTF is dead water? Other than a sea water phenomenon known to few sailors, which might just be a myth as likely.

Ya, that sounds like a great idea! Maybe I will just put some stop-leak in my pc if a seal goes bad, or maybe even a little quicksilver. What kind of redneck trailor park crap are you talking about? Have you even looked inside of a cpu block? There is a reason why everyone is not using 50/50 antifreeze mix in there pc's. What you said is a clear example of what not to do.. Look it up.

Yet cars have been utilizing both metals mixed in the same system, in the same cooling system for decades upon decades. Those work wonderfully, and the coolant channels in cylinder heads are WAY smaller than your CPU block has, often a diameter of just 5mm in many parts!

So now you are using something which is a strange physical phenomenon known by some sailors to happen in sea?

Coolant, proper glycol, is to be used aswell to minimize the build up caused by bacteria etc.
Some glycols themselves will build up, and still causes dirt to apper in the coolant, hence, the coolant ought to be swapped annually, maybe accompanied by a flushing the system.

Aluminium does oxidize, but it doesn't matter at all in a cooling system, especially if you use red glycol which has properties to protect from that.

Might be a worthwhile read: http://www.gewater.com/handbook/cooling_water_systems/ch_32_closed.jsp
Speaks in length about corrosion in cooling systems.

All you really need is a bottle of cheap glycol and tap water, and annually change the coolant.