Topic: Two Phase Open Bath Immersion Cooling Thread (Read 12992 times)

Ladies and gentlemen, Bitcoin mining and passive 2-phase immersion cooling finally made it to the cover page of Electronics Cooling magazine. About time

Bitcoin 2-Phase Immersion Cooling and the Implications for High Performance Computing

Page 24, full march issue here:
http://item-media.uberflip.com/i/269080

Full article (PDF):
Bitcoin 2-Phase Immersion Cooling and the Implications for High Performance Computing

I think you can trace much of this technology back to the 1980's when Cray Research used "Flourinert" from 3M in it's Cray-2 supercomputer. The stuff was pretty pricey at the time, but it did cool the system very effectively. It was by no means simple to regulate and make work.

One small item that I recall. There was a very real hazard that involved an electrical burn in contact with the Flourinert. The byproduct of such an event was Phosgene gas as I recall. I don't know if Novec has similar properties or not. I do know that an electrical burn inside the Cray-2 tank was BIG deal to the people in the machine room with the system.

Just Google "Cray-2" for some very pretty pictures of an immersed computer.

Nice idea for a prototype.  What is your plan for the full scale version?  I mean what size, shape, ASIC configuration are you targeting.  Using custom 3D printing is a good idea to reduce excess fluid usage.   What materials are you considering.  I was thinking of that but my concern would be an interaction between the 3D printing composite and the working fluid.  Too bad there are no 3D printers which can "print" stainless steel.   As an alternative I was thinking about maybe taking the 3D design and slicing it into layers and then cutting each layer with a CNC.  The layers could then be stacked inside the chassis.

On edit: I see you are looking to cool a MB and GPU combo not ASICs.  On the pressure issue. You may want to weigh your prototype with a high accuracy scale over time.  Under pressure Novec escapes most seals pretty well so you likely are encountering fluid loss.  Weighing the entire system over time is a good way to measure that rate of loss.  Improving the relative performance of the condensor and keeping the pressure differential as close to 0 as possible will reduce fluid loss to a minimal amount.  It may not be as much of an issue with your small prototype and low energy load but with a larger heat load and larger seal surface area unless you design it to be hermetic the fluid loss rate might be unacceptable when under pressure.

Agreed.  An example would be to cool the imaging system on a drone.  Every pound counts so the cost, complexity, and maintenance of a spray cooling system is worth it to cut a couple pounds on the final weight.

For hobbyist coolers thinking beyond bath cooling is getting the cart before the horse.  Bath cooling has enough challenges as it is.  Yes alternatives can support higher energy densities but you lose the "simplicity" of the passive nature of bath cooling. Throwing in flow cooling (pumping fluid past the component) or spray cooling (directing jet of fluid at the component surface) is just layering complexity on top of complexity.

Just popping in to say that you typically use nozzle sprays where space/weight is an issue. You can significantly reduce the working fluid required.

Hi all,

Just made a little setup for testing with 3M Novec 7100.
Please watch:
www.youtube.com/watch?v=Ut9zuh7Di4g

The fluid cools two stacked USB AsicMiners in a small olive jar.

I skipped adding thermocouple temperature measureing... it doesn't say much about the temp inside the asic components anyway. And I'm 'practical' And I do believe 3M that gas occurs at 61°C. Or at least... I did. However... now I doubt. Because there is a bit of pressure built up inside the jar. This causes the boiling point to increase. I guesstimate it's around 63-68°C now. But now I wish I added a thermocouple to know for sure.

However.. it's running for 2 days now. No problems so far.

I also have Novec 7000. That is next.

My final goal is to immerse an whole motherboard with 2 or 3 gpu boards. Also SSD drive and PSU will be cooled this way. The case is going to be mostly 3D printed, to minimize liquid usage. And total setup weight. It will include epoxy-poured windows. I already used special epoxy to create stunning optical lenses. Anyway... idea is also to add an underpressure mechanism. This will allow me to change the boiling point of the liquid at free will. And Im also thinking to 3D print custom designed cooling ribs for the condensor (shapeways allow 3D printing of sterling silver!!). But that needs some experimentation and calculations first. Sounds all cool, no? Any interest in this community to support me financially? I know kickstarter... but who needs that if you already have a bitcoin donation address?   I estimate I need 4k euro to realize all this.

http://farm3.staticflickr.com/2867/12572669655_6be0a9f82b_c.jpg
http://farm3.staticflickr.com/2848/12573161034_452e3d420a_c.jpg
http://farm4.staticflickr.com/3754/12572904303_df81d2186a_c.jpg
http://farm6.staticflickr.com/5522/12572928193_8b88b19e26_c.jpg
http://farm4.staticflickr.com/3688/12572951313_8bde0eb888_c.jpg

Not the greenest approach but it would decrease costs a lot.

It will be interesting to see how it works out.

All of my comments about cost would be out the window if you could achieve significant (20%+) overclocking I guess.

I hear your point on the evaporation, it will be interesting to see how that plays out. The health effects of the stuff are not known at this time either.

For the cooler, I'm pretty sure no chiller is necessary, since it only needs to be cooler than the boiling point of the fluid itself, for example if the fluorenert boils at 56 degrees then any loop with a radiator and pump (assuming you don't live in a very hot dessert) would be able to cool it below these levels so it would condense back.

My plan was to use cold tap water flowing through a copper coil, which in this part of the country is always cold throughout the year. No added infrastructure cost, no radiator would even be necessary.

Every set up like this I have seen has problems with the flourinert creeping.

Just because the guy selling to you says losses are zero doesn't make them zero.  With an unpressurized vessel there will always be some vapor phase material exiting the chamber.  I wouldn't care to be breathing that stuff for any period of time.

Cost is the bottom line.  This is an amusing engineering exercise, but you still have to provide chilled water / glycol to the system.  And chillers at less than 100 kW scales are as expensive as AC systems of the same capacity.  So the only benefit is higher density for higher cost.  Unless you happen to live in downtown Tokyo it isn't going to be a good trade.

If you have a look at the 3M site they have a lot of interesting pics including the answer to your question of water solubility. Basically its like the reverse of water / oil, in this case the water floats on top of the flourinert, they don't mix at all. So the way to combat water would be to have desiccant granules in a packet hanging in the case should take care of any water problems.

According to 3M losses to evaporation should be next to nil, but I suppose it means you need to accurately size the condenser radiator for the heat source its cooling, otherwise yes, likely to have some losses.

http://www.mgchemicals.com/downloads/3m/3m-FAQ.pdf

Little bit confused about talk of oil. Are you referring to mineral oil cooling now or compressor oil in a cooling loop? Flourinert product is not an oil, its not hydrocarbon based. The advantage of the flourinert over mineral oil is it can cool more much hardware per sq in than any other known method at the moment, plus its completely inert. It is desirable to have it change phase because it cools more efficiently and there is no need for a pump, at least not inside where the hardware is immersed. The downside of course is its cost.

What is the solubility of water in these fluids? 

The cooling fins are going to condense a non trivial amount of water along with Flourinert so you will either need to separate the water out of the tank, or actively extract it from solution.

Also with a non-pressurised 2 phase system, you will lose an appreciable amount of fluid each day.  That will dramatically increase your operating costs and worse, make every nearby surface an oily mess.

You would be better off immersing the cooling radiator, using a circulation pump and sticking with a single phase temperature zone.  Even with those conditions you will have trouble with losses of oil and mess due to surface tension driven creep of the oil on cables and pipes entering the tank.

Great thread, keep it going! I plan to play with this myself, definitely looks to be the way forward to cool these high power asics. But it will be nice to see how DeathandTaxes makes out first before blowing all my coins on Novec.

Here's a nice video thought I'd share from 3M. http://www.youtube.com/watch?v=ivVoANqFBuY
Anyone else building a test system yet?
The overclockers have been playing with Novec too! http://www.overclock.net/t/1209583/3m-novec-7000-group

AM partnered up with Allied Control in this matter. (electronics-cooling.com + report here). Thanks for the sources.

I used to ask 3M for a 'rendering' tank. The representation tells me about his clients briefly, including bitcoin (The ASICMiner may have implemented a tank like this).

The question is, Container needed can be in the lowest cost, not on Novec solution so only few people ask for it. For serious consideration. I will consider the Novec 7100 for the case. I will need to study about PCI-E connection for further run as x1 will do the most jobs.

Nice write up OP! This is definitely the future of high-density computing, traditional cooling methods like air and watercooling can't even compare to the efficiency of these systems. Novec is a pretty amazing substance.

I saw this today from the company that did the install of ASICMiner's immersion mine-

http://www.allied-control.com/immersion-2

Cool videos of immersion cooling rigs (cannot beat how awesome it looks visually)-

GPU Rig

http://www.youtube.com/watch?v=EN3m1bJvS4Q

Server Clusters

http://www.youtube.com/watch?v=NymeDU96pac

http://www.youtube.com/user/petuma1

Desktop PCs

http://www.youtube.com/watch?v=Z_X_hgtlJpA

http://www.youtube.com/watch?v=CIbnl3Pj15w

(apple g5) http://www.youtube.com/watch?v=HKcWUaC3s1c

OBI Tank construction

http://www.youtube.com/watch?v=ZrPq4ltZa0g

FPGAs

http://www.youtube.com/watch?v=QsRQrNSNISg

I guess my point is that with a heatsink and spraying, maybe you don't need the FC-72 because the liquid only needs to touch the heatsink and you don't have to worry about shorting things out. But yeah I can see the requirement on precision and complexity. It may not be worth it for a small scale shop.

Well with spray cooling you don't even need a heatsink.   The advantage is that it can handle extremely high heat loads 90W/cm2 is possible and with subcooling that can reach 300W/cm2 or more.   So pretty much insane power densities that would cause the component to melt with air cooling.

The disadvantage is that spray cooling requires some pretty incredible precision and if anything fails the chip will die in a matter of seconds (if that).  IMHO immersion cooling is already pretty complex as DIY project anything beyond that is starting to get into a serious engineering challenge.

I've actually been thinking about this. It may be possible to design a closed heat sink on top of the IC chips that encloses the metal fins, spray nozzle, and pipes to the pumps/condensers. That way you may be able to choose a fluid that causes less corrosion.

Flow cooling (what you described) is also an option and it can handle higher thermal loads.  Another option is subcooling where the evaporated fluid is cooled below the boiling point (not just to the boiling point).   Some high end defense components use a third concept called spray cooling where a nozzle sprays fluid directly onto the chip.  

The advantage of two phase (semi) open bath cooling is the simplicity of the system.  You have a tank, the heat source (chips) boil some of the fluid, it rises as a gas, hits a colder condenser, changes back into a liquid and "rains" back into the tank.  You still need a method of keeping the condenser "cold" and that usually involves circulating water or glycol but the actual immersion tank itself is passive with no moving parts.  No fans, or pumps, pressure regulators, or spray nozzles.  The fluid level should be roughly constant so using a float switch one could just cut power to everything in the tank if the fluid drops below a threshold.

So for a DIY the relative simplicity of two phase makes it more attractive than other solutions which are capable of higher thermal loads.

I would strongly recommend not buying Fluorinert on eBay.  The chance of getting diluted, contaminated, or used fluid is simply too high.  There is no cheap way to test the purity of Fluorinert so most likely they way you find out is "poof" you instantly destroy thousands or tens of thousands of dollars worth of gear.

Very Very cool, I was just thinking of this actually.

I'll grab some, find a container and toss a usb miner in. see how the temp sits.

For the high density chips, I was thinking of using a pump to return cooled liquir threw multiple tiny tube's. one pointed at each chip. Or just massive flow, unsure which would be more effective.

I purchased the Flourinert from here:
http://www.parallax-tech.com/fluorine.htm


They don't have the versions suitable for immersion cooling in stock but they can special order.   3M keeps pricing pretty tight so don't expect much price different between vendors.  Fluorinert runs about $80 per Liter depending on the amount purchased and the type of Fluorinert.  Novec is roughly the same but a little cheaper. 

I need to get final power consumption values for HF or CT but Fluorinert has a critical heat flux of ~20W/cm2.  Originally it looked like HF was going to use an integrated heat spreader like KNC does but they have gone w/ "naked" dies so some sort of heat sink will be needed.   It remains to be seen if CT will use an integrated heat spreader and if the heat flux will be below the critical heat flux of fluorinert.

KNC package size and integrated heat spreader may be large enough to avoid needing a heatsink.  Still given the cost of the immersion fluid I believe HF and CT are the best choices at this point.  Still the heatsinks don't need to be particularly large.  Lets assume HF uses 250W per chip (4 dies).  Even a flat plate heat spreader would only need a surface area of 250/20 = 12.5 cm2 to keep the heat flux below the CHF.  Obviously we would want a margin of safety, so lets say 25 cm2.  That is ~5cm x 5cm flat surface and would be less for a "3D" heatsink as it increases the surface area beyond the linear dimensions.  The nice thing about HF design (the company itself is pretty scammy but they have some nicely designed hardware) is there is a large open space around the die and it uses standard Intel CPU cooler mounting holes so finding a low profile heat sink which lowers the heat flux shouldn't be difficult.  I have looked at a couple low profile server heatsinks which should do the trick but am hoping to find a cheaper solution.

Well the low GH value of the chip contributes to the low GH density.  Note this isn't to say it wouldn't work.  It certainly would it would just mean using more immersion fluid per GH/s.  With the high cost of Fluorinert that means a higher cost per Gigahash cooled.  Still to be clear it will work fine it just will be less economical.

actually, i'd be worried that the high GH chips from hashfast and cointerra (300-400 watts per chip) would be too hot in terms of spot heat, and you might need some heat spreader thing to increase the surface area.

the low GH chips from bitfury et al will be no problem for immersion cooling.

even the knc chips, medium heat (150 watts per chip).. will need some kindof heat spreader

Also, when choosing the heat spreader, you will want to find one that doesnt stick out too much so that it doesnt affect your board density in the bath.  ie: long thin heat spreaders may be more useful than the high tower heatsinks that knc currently uses on their board with an air fan

So it's really not the chip but rather the PCB design that is "low GH density". If we could redesign the PCB, it may achieve a much higher density and much smaller volume?

I found it available from this retailer, although no prices listed:

http://www.fasttechnologies.com/filtration/3m-fluorinert-%E2%84%A2

and also on fleabay

http://www.ebay.com/itm/3M-FC-40-FLUORINERT-32oz-New-and-Unused-Ready-to-Ship-/110716728982

Can you recommend any source for Flourinert? I'm really looking forward to read more, great thread.

I am busy and going to be even more busy in the coming weeks so while I have the Flourinert I need to put this on hold until after the new year.   I have the tank, the rigs (well KNC and Bitfury, HF hopefully eventually), and the immersion fluid I just don't have the time to make the conversion right now.

Very interested.

Any news if you will continue this experiment?
I'm very interested in this.

Possibly BA or CT gear? Or the KnC... except for the lack of chips for a DiY design

It is very likely you won't need heatsinks using bitfury chips.  Which is a good thing given the number of chips needed for 1 TH.  FC-72 has a critical heat flux under normal conditions (no flow, no subcooling, flat surface, normal pressure) of ~ 15 W/cm2.  Beyond 15 W/cm2 you get boiling failure and the wall superheat (difference between fluid temp and device temp skyrockets).  Under the critical heat flux the wall super heat is <10 C.

Bitfury (overclocked) ~1 W/GH (chip only).  
Package dimensions 0.7cm x 0.7 cm
1/(0.7*0.7) = 2.04 W/cm2.
Bitfury Heat Flux: ~ 2 W/cm2  < FC-72 Critical Heat Flux: ~ 15W/cm2
No issues having proper boiling without heatsink.

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

Oh, we're definitely interested all right.

Our test subject will probably be an old Gen1 Avalon. We're going to shoot for 1GH/s for fun if we test this out. 

Im interested. I will read more when I have a few minutes, but I love the idea.

It looks like your need for bare boards, anything bitfurry based would be your best bet, with small sinks on each chip to aid in surface area.

Reserved - Prototype details

Reserved - Advantages & Challenges

Major Systems
1) Overview (coming soon)
2) Cooling Tank
3) Connectivity (coming soon)
4) Condenser (coming soon)
5) Chilled Water Supply (coming soon)
6) Safety Systems (coming soon)
7) Power Supply (coming soon)

2) Cooling tank

Material considerations.  Fluorinert is a highly inert fluid, it is also a poor solvent with most materials, and it is hydrophobic.  These characteristics make it an ideal working fluid.  However Fluorinert is an effective solvent for fluorinated compounds (materials containing fluorine),  plasticizers, and some additives to soft "plastics".  

This presents two challenges (and will be covered more in challenges section).  The first it that Fluorinert can replace by volume a portion of the compatible material in gaskets, seals, and orings.   For example when silicon adhesive is immersed in FC-72 up to 7% of the volume is replaced with Fluorinert.  This can lead to failure of seals in sealants, gaskets and o-rings.  Even when they use the same underlying compound "hard" materials are less prone to replacement than "soft" materials (i.e. rigid PVC pipe vs PVC wire insulation).

Material compatibility analysis by CERN
http://detector-cooling.web.cern.ch/detector-cooling/data/3M_FAQ_Fluorinert.pdf
http://detector-cooling.web.cern.ch/detector-cooling/data/Fluoro_Compatibility.htm

It is not possible in one post to do an exhaustive analysis of potential material risks so if you are interesting in working with immersion cooling I strongly recommend reviewing both of the links above and conducting your own research.  DISCLAIMER:  All of this is experimental.  Damage to equipment can occur. If you are unwilling to do self research and testing this type of project may not be realistic.   The challenges of material compatibility and handling dissolved contaminants will be covered in the challenges section.


Material choices for tank
Glass aquarium can be used for a prototype but due to the use of silicon adhesive below the fluid line I would caution against its use long term.  Stainless steel tank is a good option for the ability to design and construct a custom sized tank.  To reduce heat conduction a double tank design could be constructed with an inner stainless steel tank surrounded by insulation, surrounded by an outer tank.  To reduce costs the outer tank could be constructed from a cheaper material than stainless steel.  

For my initial prototype I will be used a polycarbonate NEMA 4 enclosure with the opening facing upward.  It would be best in any inlet be through the sidewall of the tank not the lid however in my prototype I will be cutting through the lid as any mistake can be fixed by replacing the lid instead of the more expensive body.  Some NEMA 4 enclosures come pre tapped with inlets that are gasketed.  That should be avoided for the reasons above instead look for a solid body design.




Design of the cooling tank.  
The tank should be designed to have no penetrations below the fluid line.  The fluid depth should be sufficient to cover all components plus a safety margin of 30mm.  In normal operation the fluid in the tank should be relatively constant (<5mm change in depth) however a failure of one or more cooling components will result in the condenser temperature rising and the condenser will be unable to condense (cool) the working fluid as fast as it is being boiled off and the fluid will fall.  The safety margin ensures that there is a delay between any failure and the components boiling off their fluid and likely being damaged.  To protect the safety margin a depth switch connected to a relay can be used to cut off power to the tank if the fluid level drops.  When planning the tank there should be sufficient space in the tank to contain the boiled Fluorinert gas.  As a starting point you should plan for no less volume than the volume of the fluid.  In addition you should consider the height of the heat exchanger.  


The components to be cooled are submerged in the working fluid, if possible the tank size should be optimized to minimize the amount of working fluid necessary to submerge the components.   Due to the high cost of Fluorinert (~$80 per Liter) a design which achieves the minimum amount of fluid to transfer the heat will be more economical.  Remember Fluorinert can handle a significant heat load so you can use it sparingly.  3M has shown effective heat transfer with as little as 1L for 4KW of heat load while it is unlikely that you will acheive that level of energy density it shows that excess fluid isn't necessary.

One design consideration is what components will be included in the tank.  An SHA-2 hashing system consists of three major components.  One or more processors boards, a host/controller, and one or more ATX power supplies.  It is possible to use immersion cooling to cool just processor boards or the entire system.  There are advantages and disadvantages to both.

Lets consider an example 4TH/s system using HashFast ASIC boards
Processing Boards - 2780W (87%) [1]
Host - 100W (3%) [2]
Power Supply (10%) [3]
Total: 3200W

[1] 10 boards @ 400 GH/s nominal, 278W per board
[2] Low power PC, using a embedded computer like Beagle Board would reduce wattage further
[3] The wattage is the "lost" power converted to heat.   Multiple 90% efficient ATX style 80-Gold or 80-Platinum PSU.  Power supply could possibly be in n+1 fault tolerant configuration.   In: 3200W AC Out: 2880W DC + 320W heat.

Immersion cooling only processing boards
---------------------------------------
Pros:  
 Highest energy density.  
 Reduced tank size.
 Most efficient use of Fluorinert (highest W/L)

Cons:
 Complicates power delivery.  Either custom high current cables are needed or large number of cables need to pass through the tank bulkhead.  
 Still need to air cool other components.  
 Potentially prevents deployment to areas where temp outside the tank is ill suited for air cooling (i.e. non-air conditioned warehouse).
 Not silent due to PSU fan noise.
 If cooling water line bursts the components outside the tank (as well as operators) are vulnerable to being damaged by short circuit conditions.

Immersion cooling all components
---------------------------------------
Pros:
  Simplified tank connection (can be reduced to: data cable, power cable, water in, and water out lines).
  Power supplies may need to be modified.  Fans should be removed and power supply fan monitoring bypassed (if present).
  Near silent operation (if cooling water heat transfer is in another location).

Cons:
  Larger tank (more significant than may initially appear.  remove heatsinks and fans from a hashing board and the power supply has almost the same volume as the boards it will power).
  Less efficient use of Fluorinert (lower W/L).
  Internal wire management can be more difficult if tank is cramped.
  Replacement of failed host or power supply is more complicated.
 
Depending on the boards used a hybrid between the two options would be to use immersion cooling for the processors and power supplies and place the host/controller outside the tank.  For larger operations this may be desirable as a "hot swap" replacement of defective host could be performed easily.  This may not be possible with all SHA-2 processors but in at least one case the HashFast boards are connected by USB to any host capable of running cgminer and linux.  So data connection to tank could be a single usb port and the host host system be as far from the tank as allowed by USB spec or even one host controlling multiple tanks.  The USB cable length can be extended using cheap Cat5/6 cable with an active converter.

I figured I would start this thread as a place to collaborate on immersion cooling.  The type of immersion cooling I am interested in is two phase passive semi-open bath.  Wow that is a mouthful.  So lets start with the basics.

What is immersion cooling?
Immersion cooling is a method of heat transfer where the device to be cooled (namely SHA-256 processor) is immersed in a heat transfer fluid (also called working fluid).  It is helpful to remember that "fluid" doesn't necessarily mean liquid, "Freon" is a form of working fluid although most people think of it as a gas.  Technically even air is a working fluid.  Heat sinks transfer heat to passing air and fans are pumps designed to move low density gas instead of higher density liquids.  Most people don't refer to air cooling as immersion cooling but it may help to understand some of the concepts if you think of Freon, water, and air as working fluids in heat transfer.

Is immersion cooling the same thing as "water cooling"?  
With immersion cooling the heat is transferred directly from the heat source to the working fluid.  In "watercooling" the working fluid is potentially harmful to electronics and thus flows through a sealed loop isolated from the heat source.   A watertight waterblock is used to indirectly transfer the heat from the heat source to the working fluid.  With immersion cooling the working fluid must be non-conductive and that generally limits us to three families of fluids: deionized water, mineral oil, and fluorocarbon-based fluids (namely Fluorinert made by 3M).  Immersion cooling systems have higher fluid cost than watercooling but this is partially offset by the elimination of individual waterblocks.   

What is Two Phase Cooling?
Well lets start with what is one phase cooling.  In a mineral oil or deionized water system the working fluid never boil or freezes and always remains in a single liquid phase.  Cool fluid is pumped past the heat source, where thermal energy is transfered to the fluid by conduction which raises the temperature of the fluid.  The heated fluid is then pumped to a heat exchanger where it is cooled and pumped back to the heat source.  The heat transfer is known as "sensible heat", and the more heat (thermal energy) transferred into the working fluid the more its temperature rises.  The rise in temperature can be controlled by the fluid flow rate.  The faster the fluid flow rate the less energy will be transferred into each unit of fluid and the lower the temperature rise will be.

In two phase cooling the working fluid boils and thus exists in both a liquid and gas phase.  The system takes advantage of a concept known as "latent heat" which is the heat (thermal energy) required to change the phase of a fluid.  The working fluid is only cooled by boiling and thus remains at the boiling point ("saturation temperature").  Energy is transferred from the heat source into the working fluid will cause a portion of it to boil off into a gas.   The gas rises above the fluid pool where it contacts a condenser which is cooler than the saturation temperature.  This causes the fluid to condense back into a liquid and fall (rain) back into the pool. 

What is "open bath" or "semi open bath" means?
Open Bath (sometimes more correctly called semi open bath) means the tank containing the heat source, working fluid and condenser is not a pressure vessel.  The pressure inside the tank/container will be roughly the same pressure as the outside air (<1 PSI difference).  This makes the tank easier and cheaper to construct and the system is never pressurized at dangerous pressures.  In a open bath system equilibrium is achieved by having a condenser capable of sufficient heat transfer to condense the vapor at the same rate it is being produced by boiling.   The volume of vapor remains relatively constant and prevents a rise in pressure.  If the fluid boils at a faster rate than the condenser can condense it back into a liquid then the pressure in the tank will rise and vapor will be lost.  This can be prevented by a safety pressure switch.

Why not used deionized water (DI)?
DI is electrically non conductive but it isn't inert.  It will rapidly pull ions from the surrounding material until it reaches a point where it becomes conductive and damages the cooled components.  Proper immersion cooling with DI requires expensive ion exchange systems and replacement ion exchange resin to continually remove build up of ions in the water.  This process needs to operate continually which increases the cost and the system is never stable.   Without maintenance and continual inspection the system can allow a build up of ions that eventually destroys the equipment being cooled.  While immersion cooling with DI is a viable method it is ill-suited for unmonitored environments.

Why not use mineral oil?
Mineral oil is a single phase cooling system and for any significant heat load will require actively circulating the oil across the processors.   Mineral oil also has a high viscosity and will need powerful pumps capable of handling the higher pressure.   The system needs to be carefully designed to ensure each processor receives sufficient flow.  Mineral oil has a relatively low heat capacity and thermal conductivity which increases the required flowrates and necessitates the use of larger heat exchangers. 

What types of power densities are possible?
3M has conducted experiments cooling 4KW heat loads using 1L of working fluid so in theory heat densities approaching 4,000 W/L are possible.  The constraint on commercialization is that existing servers have relatively low energy densities (well low relative to the limits of immersion cooling).  Even a high end 3U server (4 CPU, multiple GPUs, 4+ 1200W PSU) may only have an energy density of 100W per Liter.    However SHA-256 ASICs have very high energy densities although current systems have server like energy densities due to the limits of air or water cooling.  Take a look inside the case of any 2nd gen ASIC design what takes up the most space?  Air.  The actual ASIC boards are very energy dense however there are surrounded by a significant amount of empty space.   Remember these are using boards designed for air/water cooling.  It may be possible to improve energy density by making custom compact boards.

As an example of what is possible KNCs ASIC boards are 225 cm2 of surface area and use ~120W.  If boards were stacked 1cm apart that would be an energy density of >500 W/L.  Hashfast pcb design (subject to change) is even more energy dense, 280W of power in 240 cm2.  In spaced in a cooling pool 1cm apart that would be an energy density of ~1,200 W/L*.   Another way to look at it is hashing density.   At 1cm spacing KNC would be a hashing density of >500 GH/L and Hashfast would be > 1500 GH/L.    

* There is limited information available on height clearance of HF boards.  If the large FET heatsinks are not removable it may limit the spacing of boards in a cooling pool.

How is the condenser cooled?
The condenser could be cooled using refrigerant directly but a water loop (glycol/water mix) may be a more flexible and economical solution.  The water loop can either be cooled using a commercial water chiller or using a dry tower in ambient air (heat exchanger and fans outside).  The cooling loop will need to be sized large enough to transfer the full heat load out of the water loop.  The advantage of FC-72 over other two-phase working fluids is it has a 56C boiling point which allows reasonable efficiency when cooling using outside air even during summer temps for most parts of the country. The larger difference between the input air (ambient outside air temp) and the input water temp (~50C to 56C) means more efficient heat transfer is (smaller fans, smaller heat exchanger).

What working fluids are available?
3M makes a large number of synthetic working fluids however for two phase cooling we are interested in fluids which boil below the temperature limits of ASICs.   Remember in two phase cooling the fluid will remain at saturation temp (boiling point) excess thermal energy is removed from boiling however the temperature of the fluid will remain in equilibrium at saturation temp.  It is not possible with two phase cooling to have temperatures less than the boiling point of the working fluid.

3M datasheet on all heat transfer fluids: http://multimedia.3m.com/mws/mediawebserver?mwsId=tttttviZIdW5_y7VPZA_qZ0t2XV62EW9iXut2Xut2tttttt--&fn=bro_heattrans.pdf

For this reason the following fluids are most applicable:
Novec 7000 - Boiling Point 34C
FC-3284 - Boiling Point 50C
FC-72  - Boiling Point 56C
Novec 7100 - Boiling Point 61C
FC-770  - Boiling Point 95C (likely too high outside of custom design)

The cost depends on supplier and volume being purchased but generally runs $80 to $100 per Liter.  Fluorinert is often priced by the kilogram and has a high density (1.6 kg per Liter) so take that into consideration if you think you found a "deal".