Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 24. (Read 169362 times)

That's pretty cheap for a 2KW power supply . any disadvantages in using this rather than a standard ATX power supply?

Found some info on the 8 GPU limit and part of the issue appears to be BIOS limitations.

http://fastra2.ua.ac.be/?page_id=214

Would Infiniband allow access to more than 8 GPU's?

Initially, when we first connected 13 GPUs, the system refused to boot. After discussing this with ASUS, we concluded that the problem was that the GPUs required a larger block of physical address space than the BIOS was able to provide. The 32 bit BIOS can only map the PCI devices (including PCI-E) below the 4GB boundary, so this meant there was at most roughly 3GB of address space available for the devices. Because each GPU requires a block of 16MB, a block of 32MB and a block of 256MB, only 8 or 9 GPUs worked, depending on how many on-board devices we disabled in the BIOS setup. Adding more cards than that caused a boot failure.

ASUS was extremely helpful with solving this, and they provided a custom BIOS for our motherboard that skipped the address space allocation of the GTX295 cards entirely.  This is also the reason we have a single GTX275 card in the FASTRA II: it is the one card that is fully initialized by the BIOS and can provide graphics output to the monitor.

With this custom BIOS, the system booted successfully, but without working GTX295 cards since those were not initialized yet. To enable these cards, we modified a Linux 2.6.29.1 kernel (the latest at the time) to allocate physical address space to the GPUs manually. Since the kernel is 64-bit, we could map the large 256MB resource blocks above the 4GB boundary, thereby ensuring there was plenty of room for them. The smaller 16MB and 32MB blocks easily fit below 4GB, where the GPU required them.

The remaining problem was unexpected: each GPU requires a block of 4KB of I/O port space, for which only 64KB is reserved in total. Together with low-level system devices and devices like network and USB controllers also taking up I/O space this was a very tight fit. We needed to re-map inefficiently allocated system devices and disable as many devices as possible entirely, such as the RAID controller and the second network controller. From later experiments we suspect it might actually only be necessary to allocate this 4KB block of I/O ports for the primary VGA controller, but we haven’t verified that.

For power, I got myself a Dell 2360 watt power supply from their current-generation blade servers - it cost me $95

From around 2.5 ghash to almost 3. woot.

These are 5870s? Shouldn't you be getting ~415-425MHash/s with these clocks?

EDIT: now I see, it is because of 160MHz VRAM clocks. Raise them to 260MHz to get ~70MHash/s more.

These are 5870s? Shouldn't you be getting ~415-425MHash/s with these clocks?

EDIT: now I see, it is because of 160MHz VRAM clocks. Raise them to 260MHz to get ~70MHash/s more.

That's cool.

Are you going to add 8? 9? 10? I thought I read the issue with 8+ is the linux ati drivers? Anyway this is freaking badass, please let us know how it goes adding more.

UPDATE:

I was having software issues with one of my existing 6-card rigs, and I had a spare card around, so I whipped this together. The frame and mobo next to the backplane aren't doing anything; I just didn't feel like taking the PSUs out of the frame to hook this up. I ran out of molex connectors, so I had to use a third PSU. The main PSU is a 1300 watt Rosewill Lightning, and the other 2 are Cooler Master GX750s.

Very pretty but will 5870s get enough air like that?

EDIT: Spoke on IRC, and it seems that VT-d was the idea, but unfortunately my current hardware doesn't support it

Seems like VT would work though; how much are VT-d capable host boards?

Do tell? If it is anything with running multiple X sessions, I am enough of a linux noob to not know how this works or where to begin. If you mean by using VT-d and some virtual machines to overcome the driver limits, sure sounds good but my hardware is not capable (no VT-x or VT-d unless I get a more expensive host board). Or, if you mean modifying the drivers in any way, or using specific versions without a limit, perhaps you will enlighten me in that regard?

EDIT: Spoke on IRC, and it seems that VT-d was the idea, but unfortunately my current hardware doesn't support it

Really? Also please don't say virtualization these are non VT-d capable CPUs.