Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 2. (Read 169362 times)

Dunno if this project is dead or what... but if you continue, for the software side, you might want to consider a Xen based platform.  Xen is without doubt an ugly stepchild in the virtualization world, but it's been used extensively for PCI passthrough -- indeed, if I'm not mistaken, it even supported come crude virtual PCI passthrough capability before hardware IOMMUs came to market.

However I will warn you that Xen adds some complexity and if you're already struggling you might not like it.  Another approach would be to get rid of Proxmox and use regular linux.  If I were doing this myself, I would probably build it as a Gentoo system with a hand-configured kernel -- but again that's kinda complex.

I've tried proxmox and found it to be pretty buggy and fragile.  Great concept, but they just need to do more work on their implementation.  Everything seemed to be half-way implemented, so as soon as you deviated from the simple use-cases they designed it for, everything fell to pieces.  Actually everything fell to pieces even when I tried to follow the simple guides in their wiki.

Proxmox is basically debian with an openvz kernel and a bunch of Red-Hat cluster software.  You don't need the extra complexity that openvz or Red Hat cluster suite bring into the system -- you might have better luck just running ubuntu and kvm.  It's not that hard -- just follow the guides in the wiki and you'll be 90% of the way there.

I understand the allure of a GUI like Proxmox, but often the "simplicity" offered by such tools is illusory and ends up acting more like a "complexity loan" -- installation will be easy, sure, but then try to actually /do/ anything and you will spend more time than you saved during installation working around all the bugs and limitations of whatever mysterious pile of software is sitting underneath that pretty GUI.

this looked awesome

why you no use http://bitmit.net/en/?ref=1644 ?

I really wish I could

I wasn't thinking of selling it right away unless you were going to make a really great offer.

So so sad....would have been epic.

So, you are selling the video cards ?  Sad, I was really hoping to see this completed one day. Backplane up for grabs ??

I don't think so I use cgminer with "-I d -g 1 --gpu-dyninterval 250", on my least powerful cards, intensity is then set to 10.

I'm planning to test this on another system with several reference 5870s next time I'll have to stop it for maintenance, I'll report then.

I've compared outputs of lspci -vvv for the two Sapphire 5850 (only one of which was on a double extender), there were some rare differences but not meaningful to me. If you want me to collect some data, I'd be happy to do so on my next maintenance work.

6% is very high. It sounds like your -f/-I is too high/low.

I don't recall any issues like that, but that was a while ago. Have you tried other cards and other boards? Wonder if it is related to a PCIe timing issue.

@rjk : I tried chaining pci-extenders and surprisingly it worked, but with a reduced hashrate
I lost ~6% on the card I connected in a 1x -> 1x -> 16x configuration with cablesaurus' extenders.

I checked several times that I didn't make a mistake (only inserting or removing the 1x -> 1x cable). Did you see anything like it (it was with a Saphire 5850 extreme) ?

I've got 8GB and the board doesn't support any more.

Try adding more RAM.  The 12th might not be detected because of being out of memory.

I don't think it's a driver limit because I don't have any drivers loaded. I am just listing the PCI devices, which as far as I know does not require a driver. That's why I'm pretty sure it's a BIOS limit. There will of course be a driver limit, likely still of 8 devices, which is why I was planning on using VT-d and PCIe passthough with some virtual machines, but I didn't get very far on that.

Hello rjk,

Your rig is looking really nice!

Quick question on your setup. Which driver(s) are you using? Is the driver causing the 11 GPU limitation? I believe I read somewhere that the AMD drivers could only handle 8 GPUs but your rig has proven otherwise, kudos!

Been shelved for a bit. Current hurdles are:

Set up a bus - currently have 2x 2360 watt PSUs waiting to juice everything up, but no way to connect them properly.
Figure out how to load a lot of cards - currently with 12 5870s plugged in, only 11 are detected.
Figure out how to properly implement virtualization. As a Linux noob, it's really difficult and time consuming.

I am in the process of consolidating lots of the info from this thread into one big new mega post, but that is also time consuming and difficult.

This die for some reason ?

BFL ASIC convinced you to sell off

Sweet... Thanks for the information, one problem less to think about for my growing rig.

Yep. For mining with its minimal bandwidth, it should be fine. Has worked in my other rigs as well, you can chain 1x as well as 16x or combine them however you like.

Am I mistaken or are you chaining PCI-e extenders ? I was wondering if I could do that...