Pages:
Author

Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 12. (Read 169531 times)

hero member
Activity: 546
Merit: 500
If there is a way to use virtualization but still access the GPU hardware, that could allow for some really interesting ways of doing things.
There is, and a lot of commercial virtualization software offers it. It's usually called "GPU pass-through", or something along those lines.

Nice!  I see that Citrix' XenServer has it.  I wonder if that means that Xen (open source version) also has it.

Going to have to do some testing tonight.
donator
Activity: 308
Merit: 250
If there is a way to use virtualization but still access the GPU hardware, that could allow for some really interesting ways of doing things.
There is, and a lot of commercial virtualization software offers it. It's usually called "GPU pass-through", or something along those lines.
hero member
Activity: 546
Merit: 500
I'm guessing there still isn't a solution to the lame 8 GPU limit with AMD drivers? I've read through a few of the pages and it sounds like short of AMD adjusting the limits in the drivers you'll be stuck at 8 cores.
Yeah, but I'm going to attempt to experiment with virtualization and things to see what I can do.

^^ This is perhaps the most interesting part of the whole project from my perspective.  If there is a way to use virtualization but still access the GPU hardware, that could allow for some really interesting ways of doing things.
rjk
sr. member
Activity: 448
Merit: 250
1ngldh
I'm guessing there still isn't a solution to the lame 8 GPU limit with AMD drivers? I've read through a few of the pages and it sounds like short of AMD adjusting the limits in the drivers you'll be stuck at 8 cores.
Yeah, but I'm going to attempt to experiment with virtualization and things to see what I can do.
sr. member
Activity: 512
Merit: 250
...

Second, what app are you using to model the rig frame?  I need to do something similar but don't know what would be good software to use for mocking up ideas.

Its the the free Google Sketchup.  There's a library of computer component models at http://scc.jezmckean.com/home, which you can import into your own model. 
hero member
Activity: 697
Merit: 500
I'm guessing there still isn't a solution to the lame 8 GPU limit with AMD drivers? I've read through a few of the pages and it sounds like short of AMD adjusting the limits in the drivers you'll be stuck at 8 cores.
hero member
Activity: 546
Merit: 500
First of all - this has been really interested to watch it develop.  The more rigs I've accumulated the more management of rigs becomes an issue and higher density setups look more appetizing.  Can't wait to see how it turns out.  Specifically how well it works and what the final bill is.

Second, what app are you using to model the rig frame?  I need to do something similar but don't know what would be good software to use for mocking up ideas.
rjk
sr. member
Activity: 448
Merit: 250
1ngldh
DeathAndTaxes, DiabloD3, thank you for your suggestion of ArctiClean for cleaning off the thermal paste from my GPUs. This stuff is awesome.

ArctiClean is citrus based, and this makes it really pleasant to work with as opposed to harsh chemicals and/or alcohol. It comes in a bottle with a metered drip tip that lets out a drop at a time, and the $7 I spent on it is going to last forever. Cheap as hell for what you get. You drip on a few drops of the first cleaner part, and the existing thermal paste dissolves instantly, just waiting for you to wipe it up with a soft lint-free cloth. On some of my GPUs, it took 2 applications to completely clean it. Then, drop on a drop of the second part to finish the job, and admire the mirror finish on the GPU die!

The thermal paste that I chose (Tuniq TX-4) comes with a plastic scraper to spread the stuff around real thin, but it ends up being somewhat uneven anyway. I decided that having the paste so thin that I could read the lettering on the die through it was probably not enough, and ended up using an even layer about as thick as some heavy paper. I haven't tried it yet, but it looks far better than Arctic Silver that I used on some other GPUs.

Finally, the project is sort of on hold at the moment until Spotswood finishes my custom frame - it should be here in a few weeks though. If anyone needs something custom, talk to him - he can make whatever you want. The best part about it is that he whips up the design in CAD, and will discuss exactly what is needed as part of the design phase first, while sending you renders of what it will look like as the design goes on. The final frame should look something like this:



That includes a tray to fit the custom hole pattern on this backplane. The PSUs will go on top, and the side sections are removable. It is really wide for testing with a large number of GPUs, but I eventually hope to compress this project into a 7-8U 19" rack formfactor. The fans are Delta PFC1212DE-PWM 120mm models with 252 CFM airflow.

As usual, all the project costs are in the second post, and I will keep that updated. I also keep adding to the imgur album, available here: http://imgur.com/a/0oist

rjk
sr. member
Activity: 448
Merit: 250
1ngldh
I know this has probably been covered in the 14 pages of this thread, but:

Why won't risers work to use the inaccessible slots?
They will. On the previous page, I discuss how I can remove the plastic shroud and blower fan on the video cards, allowing me to use my own fan setup and allowing more space between them. The additional space between them will allow me to use risers to have a 2 level design.
hero member
Activity: 628
Merit: 500
So I found a good price on some fans, and have ordered them. They are far more loud and powerful then your average Delta screamers. Link: http://www.ebay.com/itm/220832473067 (Got 8 )

To go with these, I need someone to build me a fan controller. See this thread: https://bitcointalksearch.org/topic/wtb-make-me-a-custom-fan-controller-pwm-control-6-24-channels-50wchannel-74402

Just the fact, ma'am: Delta PFC1212DE-PWM [PDF link], 252 CFM, 48 watts at 12 volts DC, 5500 RPM, 66dBA (!), and 35 mmH2O static pressure.
EDIT: fixed link, sorry.

I guess this is going underground to dampen that noise. Smiley
legendary
Activity: 938
Merit: 1000
What's a GPU?
I know this has probably been covered in the 14 pages of this thread, but:

Why won't risers work to use the inaccessible slots?
rjk
sr. member
Activity: 448
Merit: 250
1ngldh
So I found a good price on some fans, and have ordered them. They are far more loud and powerful then your average Delta screamers. Link: http://www.ebay.com/itm/220832473067 (Got 8 )

To go with these, I need someone to build me a fan controller. See this thread: https://bitcointalksearch.org/topic/wtb-make-me-a-custom-fan-controller-pwm-control-6-24-channels-50wchannel-74402

Just the fact, ma'am: Delta PFC1212DE-PWM [PDF link], 252 CFM, 48 watts at 12 volts DC, 5500 RPM, 66dBA (!), and 35 mmH2O static pressure.
EDIT: fixed link, sorry.
hero member
Activity: 497
Merit: 500
Usually Copper coated with zinc.
rjk
sr. member
Activity: 448
Merit: 250
1ngldh
That makes me worry of scratching the bare die. I'd consider maybe a nylon spudger though .

silicon is 7 in the mohs hardness scale

steel varies according to its quality from 4 to 6.5

you'll have a hard time scratching it unless you actually hammer the die with the tool also there is going to be a paste between the scrapper and the die, although you may scratch it if you don't clean it first and there are sand particles present but that would happen with a nylon spudger too


Yeah but the actual packaging isn't silicon is it? The logic gates inside consist of silicon, but the die packaging/heat spreader is some kind of shiny metal.
sr. member
Activity: 303
Merit: 250
That makes me worry of scratching the bare die. I'd consider maybe a nylon spudger though .

silicon is 7 in the mohs hardness scale

steel varies according to its quality from 4 to 6.5

you'll have a hard time scratching it unless you actually hammer the die with the tool also there is going to be a paste between the scrapper and the die, although you may scratch it if you don't clean it first and there are sand particles present but that would happen with a nylon spudger too

legendary
Activity: 1274
Merit: 1000
i've found it more efficient to even the TIM with a small scrapper like this one i use to evenly apply flux in reballing of BGA packages, they have a nice curvy shape that really helps you do the job, just apply a thin layer.



That makes me worry of scratching the bare die. I'd consider maybe a nylon spudger though .
sr. member
Activity: 303
Merit: 250
i've found it more efficient to even the TIM with a small scrapper like this one i use to evenly apply flux in reballing of BGA packages, they have a nice curvy shape that really helps you do the job, just apply a thin layer.

sr. member
Activity: 349
Merit: 250
How many GPU's are you hoping to use in this box?
As many as I can cram in... I count 18 slots, which gives me 36 GPUs IF I am able to use all the slots, and IF I can get software to support them. At least one, and maybe even 2 of the slots appear to be "reserved" or may require an expansion board in order to be useable. The other 16 are fair game though. Dual GPU cards would then mean 32 GPUs total.

I have been experimenting with removing the plastic shroud on my 5870s - as you may know, 5870s (and most other graphics cards) fit very close together if you use adjacent slots. There is usually no gap to speak of, which makes airflow impossible unless you use shims to wedge the cards apart. When the plastic is removed, there is tons of space between the cards. See pic:

I have discovered that doing this leaves enough of a gap to fit in an x16 extender, which means that I could mount 8 cards directly to the board, and an additional 8 to 10 cards one level up via x16 extenders, double stacker style. Combined with some special cooling apparatus, this should allow me to air cool all the cards, while still fitting within a 7 or 8 U chassis.

Watching this thread with interest Smiley I'm currently running 4x 6990 under Win7, but I might be going to 8x 7990 once they come out (if that's even possible...)
legendary
Activity: 1400
Merit: 1000
I owe my soul to the Bitcoin code...
For GPUs I generally use (2 rice grain) size and press as hard and even as I can while screwing cooler back on using opposite corners.  Has worked well so far, no bubbles and good coverage.
hero member
Activity: 518
Merit: 500
I have found the grain of rice method doesn't work well on GPUs.

With a CPU you have a single heatsink which mounts to the flat heat spreader and applies a lot of pressure.

With a GPU the heatsink covers multiple components and is tightened down with multiple screws.  Very hard to get the same amount of direct pressure on the GPU die.

The other advantage of CPU is the heat spreader is larger than the chip surface so if thermal compound doesn't get all the way to the corners it doesn't really matter because there is no heat source there.  With GPU what you see is the actual chip package and even the corners are putting out significant amount of heat.

For CPU the single grain of rice method works fine.  For GPU I use the thin line and spread with a credit card.  It doesn't need to be perfect the heatsink will apply some pressure and spread it out. 

Still regardless of the method you want as little as possible.  I try to get it thin enough that it looks almost translucent.

Nope... Grain of rice method does not work for any bare dies. It leaves the edges bare like you said.

I use an X method which works fantastic. Two thin lines in an X shape from each corner of the die. Spread method works good too. Really, any method will work as long as the die is covered.

I used big grain of rice and my temperatures were fantastic ...

As said before YMMV depending on how much you apply and how much you press etc.
Pages:
Jump to: