Topic: Submersing a rig (Read 5768 times)

Grats on the oil cooler


My practical experience is on the multimillion dollar stuff and on the operational/maintenance side rather than design but the basic principles still apply. 

Your basic 'rule of thumb' is that you will achieve a 6 Deg C split between supply and return providing you are within your systems heat rejection capabilities. This temp split is a good indicator on the amount of heat rejection your system is doing.

For example:
If you are at 6 deg split and temp is still rising you have exceeded your capacity.
If you are at 6 and temp is falling you are at within your capacity.
If you are less than 6 and temp is stable the split temp you are at is a good indicator of your overall capacity at that heat load.
If you are less than 6 and temp is rising something is wrong.

Hey all! Interesting thread.

I saw an immersed rig at a LAN party recently. The guy had put water blocks on the CPU and GPUs and pumps that would handle the more viscous oil. The rig was fully immersed, the pumps then pushed oil into the water blocks and then out to a radiator and then just back into the tank, ready to circulate again and be picked up by the pumps again. Worked really well, and kept every component at about 2 degrees above ambient, even though it was significantly overclocked.

He said his next version was going to be built into a chest freezer. Madness?!?

I think you are fine.  Remember water has 20x the thermal conductivity as air.  Granted it works slightly different than a flat plate heat exchanger but @ roughly 2gpm and a 20C delta (cold loop 20C colder than hot loop) a flat plate heat exchanger can dissipate over 5KW of thermal load.


Measuring temp in the loop isn't really practical.  Water has such high thermal conductivity that the whole loop roughly reaches equilibrium.  Every point in the loop (inside GPU water block, right before radiator, right after, inside radiation, inside tube) are all within 2-3C of each other.

So just look at GPU temps and you are fine.  From that you can assume every point in the loop is within 2-3C of that.  Granted oil has lower thermal conductivity but it is closer to water than air.  The reason why the "outlet" of a air cooled GPU is much higher than the inlet is because air has such poor thermal conductivity and you simply can't push enough air to keep the temp stable (it would require tens of thousands of cfm).

As far as "what temp".  In watercooling temps 20C over ambient air are considered the minimum.  If you aren't achieving that likely your system is undersized or you have some other flaw (cooling fluid bypassing hottest part of GPU, non-turbulent flow, etc).  <10C over ambient is decent.  <5C over ambient is really only useful goal if pushing CPU (which handle heat worse than GPU) to extreme overclocks.  <2C over ambient is possible but really provides just bragging rights.  You will always have some delta over ambient simply because the system has some inefficiency.  At best the radiator can only cool the oil to the temp of the cool side (air for you right now) and depending on flow rate, and size of loop the temp in GPU will rise some over that.

Yeah that certainly is an option especially if you have a clear tank.  Still I have found using opaque tubing makes the environment inhospitable.  That combined w/ tubing like Medical grade Tygon Silver (which has silver ions) is a semi-permanent bacteriacide.  Adding silver kill coil in the tank is really just being over cautious but it is cheap insurance.

Algae needs light and opaque tubing removes that.  No algae & no bacteria = no food for larger micro organisms.  Still if I ever had a problem I would look into a UV-C light aimed at the tank (or simply taping up the tank so it is opaque) before I put any "super goop" in my system.

IBM makes water cooled servers I wonder if they run straight distilled water or what they add.  I would imagine for something as large as super computer making a mesh of silver that could be installed/replaced in the loop would improve surface area contact with the water.

Ive never done any water cooling, but if you are worried about algae, you could use an UV-C bulb.  Its perfect to avoid green water.

PCIe extenders arrived yesterday and my "Michael Schumacher" oil radiator arrived this morning.
I should have read the specs more carefully, it clearly said 18.5cm so I only have myself to blame, but holding it in my hand, that is MUCH smaller than I thought by looking at the photograph.  Its exactly the size 2 90mm fans next to each other, so smaller (but thicker) than your typical highend PC watercooling radiator.  Somehow I pictured it at least twice that size.

Still, with its relative thickness and extremely fine grating and Im sure its more than capable of dissipating the heat of a few gpus, at least when submerged in cool water. Its actually kinda neat, and I cant believe how light it is. It came with a certificate of authenticity too, so thats at least an extra 100 MH/s .

I also picked up the cheapest 1200L/H aquarium pump I could find (15 euro). I dont expect it to last, Im sure it will overheat and fail,  I just want to see what happens regarding flow rate and temperatures when it must pump oil so I get a rough idea of sizing the actual pump. 1200 L/H turns out to be a serious amount of water for such a tiny pump, wow,  but then it doesnt have to cope with thicker oil yet, nor any resistance from long tubes and the radiator, so we'll see what that gives.

Now Im waiting for some epoxy raisin to arrive to coat the foam, I need to pick up some tubing and with some luck Ill have a test setup running this weekend, although it will be "dry" for now, cooling the oil with just fans. Its freezing -11C outside, so my pond pumps are shut down. Should the oil heat up too much (which I highly doubt) I can always submerge the radiator in a bucket of water for testing.

question: have any watercooling guys here measured water temperatures at the inlet and outlet of their radiator? Can you give me a ball park guestimate?

No problem.  Don't feel bad they sell a lot of that crud each year.

Sometimes it ends badly:

http://www.xtremesystems.org/forums/showthread.php?226632-Confirmed-Feser-1-Blue-Premix-GTZ-do-NOT-want&p=3837299#post3837299



The rig owner is a long time watercooler and folding @ home user.  He believes folding 24/7 cause thermal dissociation of the coolant compounds. 

Hmm.  running chips extremely hot continually for months at a time.  That reminds me of some similar activity.

Distilled water + kill coil + good quality (Tygon) opaque tubing.

But it guarantees exact volume by European Standards!

You are right DAT⁽¹⁾. On close inspection it appears I have taken that liquid for more than it actually is.
Thanks for kindly bringing this issue to my attention.

As much as I agree with doing extensive bleed testing of the assembled system, I'm afraid newbs will keep doing the newbish thing and run the machine ASAP :/
The same with security - some subjects which seem too complicated (or just non-intuitive) will constantly be overlooked.

Notes:
(1) As always, dang it

You probably could but you lose a lot of cooling capacity, need a much beefier pump and all the components essentially have no resale value once contaminated with oil.  The main advantage of submersion cooling with oil is that you don't need expensive water blocks (5970 full coverage blocks are $120 each).  To combine expensive watercooling gear w/ expensive oil seems ... well expensive.

Honestly if built right with good quality compression fitting leaks during operation are rare.  
1) Buy high quality tubing.  Not the ultra gamer UV super cool junk.  Get tubing used for industrial purposes (where people buy stuff that works) like Tygon.
2) Buy good compression fittings.
3) Don't use any "goop".  Distilled water + opaque tubing + silver kill coil = clear water for a long time.
4) Change out the water every 12 months.
4) Once built, LEAVE SYSTEM POWERED OFF, jumpstart the PSU to run just the pumps for 24 hours and bleed the system.

If you change components, go through a powered off leak test for 24 hours again.  I have had 3 leaks.   I have damaged no systems by watercooling.  Over the years I have blown HDD, video cards, power supplies, one AMD CPU that had a fan which siezed but never from water cooling.  

No because those chemical coolants are mostly distilled water and thus have the same "issue".  There are coolants which remain non-conductive but they tend to run $100+ a gallon and they tend not to come in "ultra leet" packaging.  

Not one of those "super goop" coolants makes any claim that they are less conductive than distilled water or will remain non-conductive when exposed to contaminants.  Reread the product description.  They don't provide better cooling, they don't provide better anti-fungal protection, they don't provide corrosion resistance on an all copper system, and they don't provide electrical protection.  Pure snake oil in a bottle marked up a couple thousand percent. 

I haven't seen one post EVER where any of that "super goop" helped a system but I have seen many horror stories of the stuff breaking down, leaving deposits, and "gooping" up components.

Water works.  It is used to cool super computers.  Plain distilled water.

How about mineral oil? Its not like those watercooling tanks take 10 gallon, so cost isnt a big concern. Of course it would reduce flow rate a bit, depending how viscose the oil is, but Im unsure if that would be a problem. At most upgrade the pump. Im sure someone has tried this, anyone seen results?

DAT, I targeted that piece of advice at a guy apparently having issues with his WC installation. Distilled water rocks but if you have even a minor leakage, it will pick up dirt and the conductivity rises fast. In this context, going with chemical coolants might not be that bad an idea, huh?
I'm not advocating that stuff over good-old water for general use, neither am I not trying to change your opinion.
I'm trying to find an optimal solution to a specific problem so that a newb operator doesn't burn his rig down.
You're right though - if the installation finally proves watertight and runs without problems, you can easily replace the chemicals with water.

CFCs? Only CFCs I know are gasses (at room temperature) and many of them powerful solvents. Aside from the fact they are illegal in most places, sounds like a terrible idea. Almost as bad as "pure" water that will ionize within the hour and short everything out.


If you dont mind more than doubling power consumption, coping with condensation and be able to move the heat to where you want it, yeah sure.

Garbage.  Distilled water is non-conductive (or more correctly has very low electrical conductivity).  Lots of horror stories of all that wonder goop breaking down inside waterblocks and destroying radiators.

Distilled water + silver kill coil + opqaue tubing (I swear by Tygon Silver medical grade tubing).  You don't need anything else. 

Of course distilled water is boring so companies sell ULRTA SUPER QUANTUM WATER NOW W/ 82% MORE MOLECULAR COHESION AND LEET NANOPARTICLES!!!!

I believe there are some CFC's that would work. Actually, so would 'pure' water.

However, wouldn't it be less risky to figure out the thermodynamics of the system and appropriately set your Freezer to the correct settings. 

Don't do it! This is madness!

I've seen some rigs in mineral oil before. 

Looks pretty sick when it's submersed in an aquarium.

Good for you P4

Buy the nonconductive coolant then.

BWAHAHAHA.. I won the auction on Michael Schumachers Benetton F1 oil radiator 

http://cgi.benl.ebay.be/ws/eBayISAPI.dll?ViewItem&item=350525626493&ssPageName=ADME:L:COSI:BE:1123

21 GBP.

ROFL. Im sure that will increase my hashrate by at least 10%!

I hope its not rusty or leaky, whoever sold it probably didnt expect anyone to actually want to use it LOL