Topic: VirtualCL / 2x Quad HD 7990 (Read 8285 times)

I'm not sure you realize that ASIC stands for Application Specific Integrated Circuit - it's a cover term for any sort of circuitry (silicon based or otherwise) that performs a specific task or set of tasks.  GPUs and CPUs are both arguably ASICs - just performing a very generalized set of tasks.

There's more to ASICs than just bitcoin.  Bitcoin is a niche within a niche, in terms of global semiconductor manufacturing.


FPGAs are certainly reprogrammable for different tasks.  The main problem is that writing the hardware description (in Verilog/VHDL) is much harder than the parallel coding for GPUs, and depending on the task, the insane memory bandwidth of GPUs may be of significant use.  For large data processing, the onboard memory and bandwidth of GPUs is incredibly useful, and generating FPGA boards with the same capacity would be significantly more expensive.

Not really, it doesn't have to be SHA-256 - there are tons of different types of asics, or if he is willing to develop his own asic.  Most likely he isn't though.

This is actually where fpgas can come in handy, if he's doing the same type of processing over and over, otherwise gpus would still be preferred.

OP could only use ASIC if his work included only doing SHA-256 hashing over and over.

Lets face it, he's not. He needs GPU's.

Unfortunately most bitcoin miners are trying to minimise their capital by making open air designs that arguably are not suitable for OP. That way they reach ROI much faster. Wheras in OP's case it seems that they have more or less limitless money to throw at the problem. Good luck with that.

I assumed it was someone who had just heard that "ASIC is better than FPGA is better than GPU" and had no idea of the actual time and money costs of doing ASIC development.

Plus, for many problem sets, by the time you hang a few gigabytes of high bandwidth memory off your FPGA or ASIC, you're halfway to a GPU anyway, at only 20-30x the cost of a modern GPU.

Not everything is bitcoin.

Are you trolling or what?

Here's an early iteration:



This system went through several iterations, eventually ending up without fans on the GPUs because the case fans provided enough airflow to keep them cool, and the removal of the fan shrouds reduced flow resistance enough that they didn't need their fans.  It went down to fewer cards as well, because we needed space for the Infiniband controller (for our purposes, we found that while the bandwidth use of VCL isn't that much higher than gigabit can handle, the latency reduction is well worth the cost of the IB cards/switches).

We've since moved beyond that design to a quad 7990 design with better cooling (we don't have any full photos of it, so I'm linking a build photo).  This lets us fit the full 8 GPUs supported by the driver in, while still keeping space for the IB controller.  Also, cooling is better with the gaps between the cards, and we don't have to physically modify the cards.



Now, if your desired build cost is "hang cards off some scrap aluminum in a milk crate with a used mainboard you got off Craigslist," these aren't the systems for you.  They're not aimed at bitcoin mining, but I get the impression the OP isn't doing bitcoin mining.  For a professionally done VCL cluster, that works reliably and can be run in a data center, we've got a number of options.  We also have some systems with fewer GPUs that are a bit less expensive if you want to start smaller, and we also typically will quote a dedicated cluster controller for the systems - again, we have a lot of experience with VCL, and this is what we've found works best.

Can u show some pictures of that machines?

I beg to differ.  We have a number of 8 GPU systems deployed right now, on air cooling, that work just fine.  They are designed for rack deployment, as they need rather more power than a 120v 15A circuit can provide, so we run them on 240v most of the time.

We've got a box with 6 7970s running fine (it had 8 previously, but we needed a few for some other testing and a slot for the infiniband controller), and we have a number of boxes with 4x 7990s running quite comfortably on air.

They're a bit higher price than "milk crate computers," but they're also supported with a warranty, you can put them in a data center (try racking up milk crate systems), and they're designed for 24/7/365 operation.

The threadstarters needs are not realistic with aircooling.

Closed case + 6/8 GPUs in 1 machine can't work. U need to get that massive amount of heat off that case. No
way without watercooling.

For air cooling: Open milk crates !

I sent you a PM with some info on solutions I have available with a friend.


I entirely disagree - a homebuilt water cooling system for production work is asking for trouble.  I have know many people who have tried water cooling for GPUs before in the password cracking realm, and all of them have gone back to air cooling due to reliability issues.  The 7970s are great cards with excellent coolers on them, provided they're in a chassis that can handle their airflow and power requirements.  It's hard to do this with standard builds, but we have a few options that fit this need perfectly (24/7/365 sustained high load).

May I ask what kind of openCL computations you are doing ? I think it can be better with 3x3 GPUs if it doesnt hurt virtualCL, The worker machines can be automatic configured, maybe you already doing that.

Just a heads up for a second PSU, be a little careful if you run nr 2 out of the loop (by that I mean the PSUs doenst start stop together). Example if PSU nr 1 stops system goes down, everything ok.
If PSU nr 2 stops and its only providing power to the cards, then they will try to get as much as they can from PSU nr 1 and from the MB. In my test case it the cards fans start spinning weird and overall doesnt fell good. Hasnt checked how the mining changed. Felt a need to close the test rather quick. This was with a couple of 6950, with even more powerful cards there can be sideeffects. I would say that 3-4 doubleGPUs is a more bumpy ride.

Nickle/copper is fine. Nickle just doesn't tarnish like copper does so your blocks won't turn brown. I have been running a mixed nickle/copper loop with a silver kill coil since ~2011 with no problems. I'm just running distilled water and shitty home depot 1/2" tubing. Get quality barbs and utilize zipties or hose clamps and you won't ever have problems with hoses stretching and leaking. I have been using Bitspower's 1/2" barbs with great success.

try this guy on youtube here's really good at cooling*

http://www.youtube.com/watch?v=xnQASvYeRtE&feature=c4-overview-vl&list=PLWYPRHLWSBL4B04Ab1v3TNsj1akqyW_-E

SingularityComputers

Distilled water all the way, some people add small amounts of 'wetter' which reduces the surface tension of the water. Not sure as to the benefits of nickel plating but mixing metals isn't the best idea when it's alu/copper cause you essentially turn the loop into a battery! Not sure what the consensus is on mixing nickel/copper (pretty much all radiators are copper). Secondary PSU is a very good idea too!

Looks like water cooling is the way to go. Expecting the cards and chassis to come in somewhere next week. Got a very detailed e-mail from EK with their products which also mention the full cover 7990's so will look into that and puzzle it together so the cards, chassis and hopefully the water cooling products arrive at the same time.

Also did put in a secondary PSU to make sure I do not run into power issues when the four 7990's are running at max speed.

Regarding the liquids to use, i see they have copper and nickel plates, any advice there what to use in combination with what liquid?

I was posting from memory but I think I actually meant non conductive additive (as in it doesn't suddenly turn the water conductive) - I once filled up a loop and the pushfit fitting were dodgy - ended up soaking a 6800 Ultra + motherboard but they both survived! Can't remember exactly what it was, was blue if that helps

Btw a chiller to handle that sort of loading would be ridiculously expensive, the good thing about a water loop is that if the temps are too high at first try you can just keep adding radiators or more powerful fans till the cooling is sufficient. You can also use it for underfloor air heating with rigs like that

Get a silver kill coil and distilled water to handle any biocide requirements. What is an anti-conductive additive? Short of using something like fluorinert I don't think there is a way to prevent the water from ionizing given the constant contact with copper.

You wouldn't need a water chiller, much simpler design for 4x7990 would be a triple liang dcc and dual/triple 480mm radiator setup. Then you don't have to worry about chiller control etc.

1 x Thread Laing DDC Triple Acrylic Pump Top : Plexi
3 x Laing DDC-1T Pro Pump 440 L/hr 10W
2-4 x Swiftech MCR420XP Extreme Performance 480mm Quad Radiator
8-16 x Zalman ZM-F3 120 x 120 x 25mm 1800RPM Case Fan
4 x EK FC 7990 SE Full Cover Waterblock for AMD Malta Series Reference Design HD 7990 Graphics Card : Copper Acetal

Plus some tubing and some high quality hose clamps. T line is much easier than using a reservoir and means there is less to spill if something goes wrong, I'd still recommend distilled water and some kind of anti-conductive additive though.

Although expensive it should keep the cards at a delta between 10-20 degrees max compared to ambient air at full load depending on how many radiators you decide to use.

Look into some of the open-air frames that have been built. I know gigavps sells his frame in the marketplace here. By giving the cards a few inches air-gap you can run 100% load on the stock cooler. Waterblocks are fantastic for compute dense configurations but there are all sorts of maintenance issues around those. For example I just discovered that a large quantity of my HD 5970 waterblocks have warped acrylic covers. Pain in the ass that could have resulted in a dead motherboard and possibly dead video cards.

For simplicity and if you have a location where noise isn't a concern, go with open-air rigs. If it must be cased then look into waterblocks.