Topic: Two Phase Open Bath Immersion Cooling Thread (Read 13037 times)

Ladies and gentlemen, Bitcoin mining and passive 2-phase immersion cooling finally made it to the cover page of Electronics Cooling magazine. About time

Bitcoin 2-Phase Immersion Cooling and the Implications for High Performance Computing

Page 24, full march issue here:
http://item-media.uberflip.com/i/269080

Full article (PDF):
Bitcoin 2-Phase Immersion Cooling and the Implications for High Performance Computing

I think you can trace much of this technology back to the 1980's when Cray Research used "Flourinert" from 3M in it's Cray-2 supercomputer. The stuff was pretty pricey at the time, but it did cool the system very effectively. It was by no means simple to regulate and make work.

One small item that I recall. There was a very real hazard that involved an electrical burn in contact with the Flourinert. The byproduct of such an event was Phosgene gas as I recall. I don't know if Novec has similar properties or not. I do know that an electrical burn inside the Cray-2 tank was BIG deal to the people in the machine room with the system.

Just Google "Cray-2" for some very pretty pictures of an immersed computer.

Nice idea for a prototype.  What is your plan for the full scale version?  I mean what size, shape, ASIC configuration are you targeting.  Using custom 3D printing is a good idea to reduce excess fluid usage.   What materials are you considering.  I was thinking of that but my concern would be an interaction between the 3D printing composite and the working fluid.  Too bad there are no 3D printers which can "print" stainless steel.   As an alternative I was thinking about maybe taking the 3D design and slicing it into layers and then cutting each layer with a CNC.  The layers could then be stacked inside the chassis.

On edit: I see you are looking to cool a MB and GPU combo not ASICs.  On the pressure issue. You may want to weigh your prototype with a high accuracy scale over time.  Under pressure Novec escapes most seals pretty well so you likely are encountering fluid loss.  Weighing the entire system over time is a good way to measure that rate of loss.  Improving the relative performance of the condensor and keeping the pressure differential as close to 0 as possible will reduce fluid loss to a minimal amount.  It may not be as much of an issue with your small prototype and low energy load but with a larger heat load and larger seal surface area unless you design it to be hermetic the fluid loss rate might be unacceptable when under pressure.

Agreed.  An example would be to cool the imaging system on a drone.  Every pound counts so the cost, complexity, and maintenance of a spray cooling system is worth it to cut a couple pounds on the final weight.

For hobbyist coolers thinking beyond bath cooling is getting the cart before the horse.  Bath cooling has enough challenges as it is.  Yes alternatives can support higher energy densities but you lose the "simplicity" of the passive nature of bath cooling. Throwing in flow cooling (pumping fluid past the component) or spray cooling (directing jet of fluid at the component surface) is just layering complexity on top of complexity.

Just popping in to say that you typically use nozzle sprays where space/weight is an issue. You can significantly reduce the working fluid required.

Hi all,

Just made a little setup for testing with 3M Novec 7100.
Please watch:
www.youtube.com/watch?v=Ut9zuh7Di4g

The fluid cools two stacked USB AsicMiners in a small olive jar.

I skipped adding thermocouple temperature measureing... it doesn't say much about the temp inside the asic components anyway. And I'm 'practical' And I do believe 3M that gas occurs at 61°C. Or at least... I did. However... now I doubt. Because there is a bit of pressure built up inside the jar. This causes the boiling point to increase. I guesstimate it's around 63-68°C now. But now I wish I added a thermocouple to know for sure.

However.. it's running for 2 days now. No problems so far.

I also have Novec 7000. That is next.

My final goal is to immerse an whole motherboard with 2 or 3 gpu boards. Also SSD drive and PSU will be cooled this way. The case is going to be mostly 3D printed, to minimize liquid usage. And total setup weight. It will include epoxy-poured windows. I already used special epoxy to create stunning optical lenses. Anyway... idea is also to add an underpressure mechanism. This will allow me to change the boiling point of the liquid at free will. And Im also thinking to 3D print custom designed cooling ribs for the condensor (shapeways allow 3D printing of sterling silver!!). But that needs some experimentation and calculations first. Sounds all cool, no? Any interest in this community to support me financially? I know kickstarter... but who needs that if you already have a bitcoin donation address?   I estimate I need 4k euro to realize all this.

http://farm3.staticflickr.com/2867/12572669655_6be0a9f82b_c.jpg
http://farm3.staticflickr.com/2848/12573161034_452e3d420a_c.jpg
http://farm4.staticflickr.com/3754/12572904303_df81d2186a_c.jpg
http://farm6.staticflickr.com/5522/12572928193_8b88b19e26_c.jpg
http://farm4.staticflickr.com/3688/12572951313_8bde0eb888_c.jpg

Not the greenest approach but it would decrease costs a lot.

It will be interesting to see how it works out.

All of my comments about cost would be out the window if you could achieve significant (20%+) overclocking I guess.

I hear your point on the evaporation, it will be interesting to see how that plays out. The health effects of the stuff are not known at this time either.

For the cooler, I'm pretty sure no chiller is necessary, since it only needs to be cooler than the boiling point of the fluid itself, for example if the fluorenert boils at 56 degrees then any loop with a radiator and pump (assuming you don't live in a very hot dessert) would be able to cool it below these levels so it would condense back.

My plan was to use cold tap water flowing through a copper coil, which in this part of the country is always cold throughout the year. No added infrastructure cost, no radiator would even be necessary.

Every set up like this I have seen has problems with the flourinert creeping.

Just because the guy selling to you says losses are zero doesn't make them zero.  With an unpressurized vessel there will always be some vapor phase material exiting the chamber.  I wouldn't care to be breathing that stuff for any period of time.

Cost is the bottom line.  This is an amusing engineering exercise, but you still have to provide chilled water / glycol to the system.  And chillers at less than 100 kW scales are as expensive as AC systems of the same capacity.  So the only benefit is higher density for higher cost.  Unless you happen to live in downtown Tokyo it isn't going to be a good trade.

If you have a look at the 3M site they have a lot of interesting pics including the answer to your question of water solubility. Basically its like the reverse of water / oil, in this case the water floats on top of the flourinert, they don't mix at all. So the way to combat water would be to have desiccant granules in a packet hanging in the case should take care of any water problems.

According to 3M losses to evaporation should be next to nil, but I suppose it means you need to accurately size the condenser radiator for the heat source its cooling, otherwise yes, likely to have some losses.

http://www.mgchemicals.com/downloads/3m/3m-FAQ.pdf

Little bit confused about talk of oil. Are you referring to mineral oil cooling now or compressor oil in a cooling loop? Flourinert product is not an oil, its not hydrocarbon based. The advantage of the flourinert over mineral oil is it can cool more much hardware per sq in than any other known method at the moment, plus its completely inert. It is desirable to have it change phase because it cools more efficiently and there is no need for a pump, at least not inside where the hardware is immersed. The downside of course is its cost.

What is the solubility of water in these fluids? 

The cooling fins are going to condense a non trivial amount of water along with Flourinert so you will either need to separate the water out of the tank, or actively extract it from solution.

Also with a non-pressurised 2 phase system, you will lose an appreciable amount of fluid each day.  That will dramatically increase your operating costs and worse, make every nearby surface an oily mess.

You would be better off immersing the cooling radiator, using a circulation pump and sticking with a single phase temperature zone.  Even with those conditions you will have trouble with losses of oil and mess due to surface tension driven creep of the oil on cables and pipes entering the tank.

Great thread, keep it going! I plan to play with this myself, definitely looks to be the way forward to cool these high power asics. But it will be nice to see how DeathandTaxes makes out first before blowing all my coins on Novec.

Here's a nice video thought I'd share from 3M. http://www.youtube.com/watch?v=ivVoANqFBuY
Anyone else building a test system yet?
The overclockers have been playing with Novec too! http://www.overclock.net/t/1209583/3m-novec-7000-group

AM partnered up with Allied Control in this matter. (electronics-cooling.com + report here). Thanks for the sources.

I used to ask 3M for a 'rendering' tank. The representation tells me about his clients briefly, including bitcoin (The ASICMiner may have implemented a tank like this).

The question is, Container needed can be in the lowest cost, not on Novec solution so only few people ask for it. For serious consideration. I will consider the Novec 7100 for the case. I will need to study about PCI-E connection for further run as x1 will do the most jobs.

Nice write up OP! This is definitely the future of high-density computing, traditional cooling methods like air and watercooling can't even compare to the efficiency of these systems. Novec is a pretty amazing substance.

I saw this today from the company that did the install of ASICMiner's immersion mine-

http://www.allied-control.com/immersion-2

Cool videos of immersion cooling rigs (cannot beat how awesome it looks visually)-

GPU Rig

http://www.youtube.com/watch?v=EN3m1bJvS4Q

Server Clusters

http://www.youtube.com/watch?v=NymeDU96pac

http://www.youtube.com/user/petuma1

Desktop PCs

http://www.youtube.com/watch?v=Z_X_hgtlJpA

http://www.youtube.com/watch?v=CIbnl3Pj15w

(apple g5) http://www.youtube.com/watch?v=HKcWUaC3s1c

OBI Tank construction

http://www.youtube.com/watch?v=ZrPq4ltZa0g

FPGAs

http://www.youtube.com/watch?v=QsRQrNSNISg

I guess my point is that with a heatsink and spraying, maybe you don't need the FC-72 because the liquid only needs to touch the heatsink and you don't have to worry about shorting things out. But yeah I can see the requirement on precision and complexity. It may not be worth it for a small scale shop.

Well with spray cooling you don't even need a heatsink.   The advantage is that it can handle extremely high heat loads 90W/cm2 is possible and with subcooling that can reach 300W/cm2 or more.   So pretty much insane power densities that would cause the component to melt with air cooling.

The disadvantage is that spray cooling requires some pretty incredible precision and if anything fails the chip will die in a matter of seconds (if that).  IMHO immersion cooling is already pretty complex as DIY project anything beyond that is starting to get into a serious engineering challenge.

I've actually been thinking about this. It may be possible to design a closed heat sink on top of the IC chips that encloses the metal fins, spray nozzle, and pipes to the pumps/condensers. That way you may be able to choose a fluid that causes less corrosion.

Flow cooling (what you described) is also an option and it can handle higher thermal loads.  Another option is subcooling where the evaporated fluid is cooled below the boiling point (not just to the boiling point).   Some high end defense components use a third concept called spray cooling where a nozzle sprays fluid directly onto the chip.  

The advantage of two phase (semi) open bath cooling is the simplicity of the system.  You have a tank, the heat source (chips) boil some of the fluid, it rises as a gas, hits a colder condenser, changes back into a liquid and "rains" back into the tank.  You still need a method of keeping the condenser "cold" and that usually involves circulating water or glycol but the actual immersion tank itself is passive with no moving parts.  No fans, or pumps, pressure regulators, or spray nozzles.  The fluid level should be roughly constant so using a float switch one could just cut power to everything in the tank if the fluid drops below a threshold.

So for a DIY the relative simplicity of two phase makes it more attractive than other solutions which are capable of higher thermal loads.

I would strongly recommend not buying Fluorinert on eBay.  The chance of getting diluted, contaminated, or used fluid is simply too high.  There is no cheap way to test the purity of Fluorinert so most likely they way you find out is "poof" you instantly destroy thousands or tens of thousands of dollars worth of gear.