Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] (Read 169362 times)

Hey if you wanna have a go, I've got all the parts except the video cards... Just name your price.

Hello 420,

Sorry for my late reply,

The system is called Fastra II: http://fastra2.ua.ac.be/

Hi, rjk. It's a great pity that this interesting project was finally failed.
You should have made more progress if you had asked help from AMD directly.
I heard there was an informal "Multi-GPU" project proposed by AMD last year and finally
they got 16 GPUs (8x FirePro S10000s) to work in a single rig successfully.
So I think AMD might have already developed a proprietary driver to overcome the 8-gpu limit.
See:
http://devgurus.amd.com/thread/158863
http://fireuser.com/blog/8_amd_firepro_s10000s_16_gpus_achieve_8_tflops_real_world_double_precision_/

rjk, do you still have the entire pwm fan controller setup that I built for this project years ago? I remember taking a lot of pride in that, being my first bitcoin-related project

Good luck selling, I hope someone will be eager to take on the challenge for sCrypt mining!

kickass is there a video of this?

A research group from the University of Antwerp managed to get 13 GPUs on one ASUS motherboard. They confirm the limitation is BIOS related as you mentioned. The group ended up customizing the BIOS and hacking the Linux kernel.

I don't think it's a driver limit because I don't have any drivers loaded. I am just listing the PCI devices, which as far as I know does not require a driver. That's why I'm pretty sure it's a BIOS limit. There will of course be a driver limit, likely still of 8 devices, which is why I was planning on using VT-d and PCIe passthough with some virtual machines, but I didn't get very far on that.

It's still available. I unfortunately don't know how well 3D would do on a link smaller than x16. I think there have been studies on it to show that the gaming performance hit would be nil, but rendering and other such workloads would take a major hit.

This is the most powerful SHB on the market that I know of: http://www.trentonsystems.com/products/single-board-computers/picmg-13/bxt7059
It supports up to two 8-core Intel Xeon E5-2448L processors (16 physical and 32 virtual cores with HT), and up to 96GB of RAM if you use 16GB modules in all 6 slots.

If you will be processing video, you will want this backplane instead: http://www.trentonsystems.com/products/backplanes/picmg-13-backplanes/video-processing-gpu-backplane-bpg8032
It has x16 links on all slots.

As for how many cards you can actually install and use... you would be on your own there. It requires virtualization and/or driver tricks that I have no idea how to do (that's mainly why this project flopped), and possibly BIOS mods depending on how you are trying to make it work.

If you want it, let me know.

Probably a bunch of people, myself included, but I guess I would need some more info.

How's the 3D performance of the PCIe cards through the x4(?) electrical connection?

To your knowledge, is there an SHB available (probably AMD-based) that will allow me to pack at least 36 cores into the box?  And at least 64 GB of RAM?

I've built two multi-headed gaming boxes now out of my old mining hardware.  Fun stuff.  If there's a realistic way to stuff 18 heads into a single case (Oh hrm, I'd need more GPUs ), I'd be open to looking into it.

At the beginning of the thread is a list of parts and the total that I have spent on all of them. Wonder if someone is interested in buying it...

Yep. For mining with its minimal bandwidth, it should be fine. Has worked in my other rigs as well, you can chain 1x as well as 16x or combine them however you like.

For the record, XenServer, isn't a product I'd recommend for most purposes due to my not being too fond of the company that makes it.

Xen, by contrast, is open source -- it doesn't cost anything; that includes the hardware virtualization support.  In other words, it just so happens that XenServer uses Xen as part (most) of its "commercial product," but there's no reason regular people who don't hate money can't use it also, for free

It's possible that XenServer might work, but as far as I know the hardware virtualization requires an expensive license to use, and using open source Xen would be Greek all over again. 

Dunno if this project is dead or what... but if you continue, for the software side, you might want to consider a Xen based platform.  Xen is without doubt an ugly stepchild in the virtualization world, but it's been used extensively for PCI passthrough -- indeed, if I'm not mistaken, it even supported come crude virtual PCI passthrough capability before hardware IOMMUs came to market.

However I will warn you that Xen adds some complexity and if you're already struggling you might not like it.  Another approach would be to get rid of Proxmox and use regular linux.  If I were doing this myself, I would probably build it as a Gentoo system with a hand-configured kernel -- but again that's kinda complex.

I've tried proxmox and found it to be pretty buggy and fragile.  Great concept, but they just need to do more work on their implementation.  Everything seemed to be half-way implemented, so as soon as you deviated from the simple use-cases they designed it for, everything fell to pieces.  Actually everything fell to pieces even when I tried to follow the simple guides in their wiki.

Proxmox is basically debian with an openvz kernel and a bunch of Red-Hat cluster software.  You don't need the extra complexity that openvz or Red Hat cluster suite bring into the system -- you might have better luck just running ubuntu and kvm.  It's not that hard -- just follow the guides in the wiki and you'll be 90% of the way there.

I understand the allure of a GUI like Proxmox, but often the "simplicity" offered by such tools is illusory and ends up acting more like a "complexity loan" -- installation will be easy, sure, but then try to actually /do/ anything and you will spend more time than you saved during installation working around all the bugs and limitations of whatever mysterious pile of software is sitting underneath that pretty GUI.