The PCI mounting bracket is removed. The (new) overhanging piece of aluminum bar presses down onto an edge of the GPU card.
Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 8. (Read 169362 times)
April 19, 2012, 09:43:45 PM
Here's a pic of what I was thinking...
The PCI mounting bracket is removed. The (new) overhanging piece of aluminum bar presses down onto an edge of the GPU card.
The PCI mounting bracket is removed. The (new) overhanging piece of aluminum bar presses down onto an edge of the GPU card.
April 19, 2012, 08:36:20 PM
Sorry to here about that.
What if the top "clamp" part of the PCI support bracket was a lot wider? Is there enough of an edge/surface on a GPU card that a wider clamp's foam could press onto?
Not sure if I totally understand, but the support beam clamp arrangement is plenty wide. However, the note about the screws interfering is just because the clamp sections are thick, and the GPU support bracket is designed to be sitting on thin steel, so the screws are right up near the top of the bend in the bracket. With the stock bracket, the screws aren't totally necessary because the DVI connectors have screws that hold it on.What if the top "clamp" part of the PCI support bracket was a lot wider? Is there enough of an edge/surface on a GPU card that a wider clamp's foam could press onto?
Also, the screws that hold the clamp together are dead center in the width of it, but ideally should be in the empty space where the screws on the card bracket usually go. Sorry the picture is blurry, but you can see the repeating pattern in the foam where the brackets are, and if you look to the right side where there is a dot that is sticking through, that's where a screw hole should be since that is a void in the card's bracket.
April 19, 2012, 06:52:23 PM
Either that, or cut the backplanes and only leave the very top screw mount and the overhang (so it balances on the beam).
This is the part that I want to get fabricated, so I can keep the original brackets for resale. Unless someone wants to sell me a bunch of brackets? They could actually be from any reference card model, as long as it has the 2 screws at the top, since the rest would be getting chopped off.Would this single wide bracket work? EDIT: More info here.
Available here for ~$2 each.
Sorry to here about that.
What if the top "clamp" part of the PCI support bracket was a lot wider? Is there enough of an edge/surface on a GPU card that a wider clamp's foam could press onto?
April 19, 2012, 04:19:29 PM
Either that, or cut the backplanes and only leave the very top screw mount and the overhang (so it balances on the beam).
This is the part that I want to get fabricated, so I can keep the original brackets for resale. Unless someone wants to sell me a bunch of brackets? They could actually be from any reference card model, as long as it has the 2 screws at the top, since the rest would be getting chopped off.Would this single wide bracket work? EDIT: More info here.
Available here for ~$2 each.
April 19, 2012, 12:37:36 AM
If you need an example of a working .conf I can provide it.
However, I may require you to come over to relay chats just to be mean since you quit forever.
aha, still haven't forgiven old cocktopus for disappearing? However, I may require you to come over to relay chats just to be mean since you quit forever.
If I can't get something going, I'll be sure to show up and ask you for tech support. 
April 19, 2012, 12:31:49 AM
To the best of my knowledge sir passthrough works out of the box in PVE. I've admin'd production deployments of it now to virtualize KVM machines since .9
I know it isn't available in the web gui like I said, but you can edit your .conf files and provide device ID's (lspci, lsusb) and pass-through devices from your host. I've done it with a USB dongle/key, a PCI modem, and an external USB hard drive, possibly serial - all from debian host to windows guest, all successful.
Basically PVE installs openvz and KVM and an apache web interface on debian. If you get it installed and get logged into the web interface and can't create a KVM machine only openvz, enable virtualization in your BIOS. They have fairly good support forums.
Sweet, good info. I know it isn't available in the web gui like I said, but you can edit your .conf files and provide device ID's (lspci, lsusb) and pass-through devices from your host. I've done it with a USB dongle/key, a PCI modem, and an external USB hard drive, possibly serial - all from debian host to windows guest, all successful.
Basically PVE installs openvz and KVM and an apache web interface on debian. If you get it installed and get logged into the web interface and can't create a KVM machine only openvz, enable virtualization in your BIOS. They have fairly good support forums.
April 19, 2012, 12:06:13 AM
Depends on the CPU arch's address space (hint: 64 bit processors, internally, do not have 64 bits of address space, its usually something like 40 or 48 bit).
So there rest is reserved for something else?
No, it just doesn't waste the space internally.
April 18, 2012, 11:57:46 PM
Virtualbox from Oracle would also be worth checking out.
Virtualbox is a type 2 hypervisor, I.E. it won't work.I don't know if it would help you or not RJK, but since unlike it's predecessors the PCI-e standard is plug and play(unless it's different on your motherboard) isn't it just a matter of plugging the cards to the MB after it boots to avoid the boot issue and configure the VM that way? If it works it would still suck to have to reboot the system(unplugging all the but 4 cards) but if it works... why reboot the rig ever again right :p hehe
I've tried this on other platforms with mixed results. I'm not going to risk this expensive board though. Since KVM has been brought up, Proxmox VE would be my obvious choice because it's a bare-metal debian distro with KVM already rolled in. You would have to manually edit your vm.conf files with the device ID's for the VGA passthrough, but this should be easy enough for a linux admin.
So it supports hardware passthrough out of the box? I thought even vanilla KVM needed some special configs and modules to be loaded. Also, alas, I am not a linux admin. Just a linux tinkerer.In other news, I ordered the board today to the tune of a little over a grand. It comes with a 30 day evaluation to see if it will even be compatible with my backplane, since the backplane is made by Trenton and the board is from Advantech. I guess we will see whether it works or not.
April 18, 2012, 10:51:17 PM
Depends on the CPU arch's address space (hint: 64 bit processors, internally, do not have 64 bits of address space, its usually something like 40 or 48 bit).
So there rest is reserved for something else?
April 18, 2012, 09:32:30 PM
I don't know if it would help you or not RJK, but since unlike it's predecessors the PCI-e standard is plug and play(unless it's different on your motherboard) isn't it just a matter of plugging the cards to the MB after it boots to avoid the boot issue and configure the VM that way? If it works it would still suck to have to reboot the system(unplugging all the but 4 cards) but if it works... why reboot the rig ever again right :p hehe
April 18, 2012, 09:25:13 PM
KVM and Xen in linux both support PCI (and pcie) passthrough to guests:
http://wiki.xensource.com/xenwiki/XenVGAPassthrough
Virtualbox from Oracle would also be worth checking out.
http://wiki.xensource.com/xenwiki/XenVGAPassthrough
Virtualbox from Oracle would also be worth checking out.
April 18, 2012, 12:02:29 PM
Have you found any good HOWTOs on setting up the virtualization?
...
Nobody's made anything quite like this, to my knowledge. There is a KVM tut on the previous page that I will be looking at....
This seems to be for esx and maybe esxi. The thing is that there seems to be a limit on the number of devices that can be passed through (4).
http://blogs.vmware.com/performance/2011/10/gpgpu-computing-in-a-vm.html
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789
April 18, 2012, 08:36:56 AM
Have you found any good HOWTOs on setting up the virtualization?
I guess the options are:
VMWARE esxi: the easiest to setup, just install and select devices for passthrough and install guest OS. It might even boot existing BAMT from usb stick(pass usb controller). Free to test! any dual GPU is no go though
Citrix XenServer: Has GPU passthrough support on a super expensive licenced copy... so no way to tell if it is any easier to implement than the opensource project. I think that some documentation would be available but one might still need to be a Linux guru.
Opensource Xen: According to Xen wiki AMD passthrough works OOB. However some GPU bios extracting might be involved and all kind of hacking to get actually in the point of attempting a first boot. On the upside some reports of having 5970 working on this solution are floating in the net.
My GoogleFu is apparently weak as I have not found a simple and thorough enough guide on how to actually tame this Xen-beast...
Nobody's made anything quite like this, to my knowledge. There is a KVM tut on the previous page that I will be looking at. I guess the options are:
VMWARE esxi: the easiest to setup, just install and select devices for passthrough and install guest OS. It might even boot existing BAMT from usb stick(pass usb controller). Free to test! any dual GPU is no go though
Citrix XenServer: Has GPU passthrough support on a super expensive licenced copy... so no way to tell if it is any easier to implement than the opensource project. I think that some documentation would be available but one might still need to be a Linux guru.
Opensource Xen: According to Xen wiki AMD passthrough works OOB. However some GPU bios extracting might be involved and all kind of hacking to get actually in the point of attempting a first boot. On the upside some reports of having 5970 working on this solution are floating in the net.
My GoogleFu is apparently weak as I have not found a simple and thorough enough guide on how to actually tame this Xen-beast...
April 18, 2012, 06:49:37 AM
Have you found any good HOWTOs on setting up the virtualization?
I guess the options are:
VMWARE esxi: the easiest to setup, just install and select devices for passthrough and install guest OS. It might even boot existing BAMT from usb stick(pass usb controller). Free to test! any dual GPU is no go though
Citrix XenServer: Has GPU passthrough support on a super expensive licenced copy... so no way to tell if it is any easier to implement than the opensource project. I think that some documentation would be available but one might still need to be a Linux guru.
Opensource Xen: According to Xen wiki AMD passthrough works OOB. However some GPU bios extracting might be involved and all kind of hacking to get actually in the point of attempting a first boot. On the upside some reports of having 5970 working on this solution are floating in the net.
My GoogleFu is apparently weak as I have not found a simple and thorough enough guide on how to actually tame this Xen-beast...
I guess the options are:
VMWARE esxi: the easiest to setup, just install and select devices for passthrough and install guest OS. It might even boot existing BAMT from usb stick(pass usb controller). Free to test! any dual GPU is no go though
Citrix XenServer: Has GPU passthrough support on a super expensive licenced copy... so no way to tell if it is any easier to implement than the opensource project. I think that some documentation would be available but one might still need to be a Linux guru.
Opensource Xen: According to Xen wiki AMD passthrough works OOB. However some GPU bios extracting might be involved and all kind of hacking to get actually in the point of attempting a first boot. On the upside some reports of having 5970 working on this solution are floating in the net.
My GoogleFu is apparently weak as I have not found a simple and thorough enough guide on how to actually tame this Xen-beast...
April 17, 2012, 10:31:14 AM
Apparently ASUS Crosshair V (990FX) comes with AMD's version of VT-D hardware visualization. They call it IOMMU. Would be interesting to see someone try it.
All FX series chips have it, none of the rest do. That includes 790FX, 890FX, and 990FX. Its really a downscaled version of their server chipset, not a member of their desktop series. IOMMU is the literal description of what that is, both Intel and AMD (plus Sun's and IBM's) are called that. All it does is allow two way remapping of hardware interrupts and device-side device memory map remapping (such as unified remaps of device->system ram and cpu->device ram).
If you're going to do any virtualization of suitably complex devices, you need IOMMU. This includes, but isn't limited to, GPUs, higher end NICs (10gbit and up typically), high end RAID controllers, etc.
April 17, 2012, 09:51:22 AM
Apparently ASUS Crosshair V (990FX) comes with AMD's version of VT-D hardware visualization. They call it IOMMU. Would be interesting to see someone try it.
April 17, 2012, 09:28:06 AM
This depends on the motherboard... For instance, my GA-990FXA-UD7 wouldn't even boot with 8 GPUs until I started disabling some devices in the BIOS (USB3, eSata, etc...). After a couple of tries, everything works and this box has been running stable for several months now.
That is a good point. The BIOS maps all onboard functions to the virtual memory space. Most are not anticipating 8 GPUs. Good idea to turn off everything you don't need (it also saves some watts). From the NVidia cluster blog it loooks like BIOS designers are horribly inefficient with memory space allocating giant chunks of memory to each device.
Still move >8 GPUs requires more "exotic" solutions. NVidia makes it easier since GPU are decoupled from xorg and have no driver level limit. AMD driver level limit and tight coupling make virtualization a requirement.
GPU limitations:
BIOS limitation - (NVidia cluster w/ 13 GPU required a custom BIOS built by manufacturer)
Driver limit - AMD drivers have a hardcoded limit of 8 (NVidia has no driver level limit but doesn't support >8 due to other limits)
Virtual space limit - >8 GPU likely mandates x64 OS
Kernel limit - Large number of addressable devices may require changes to kernel
xorg limit - xorg can't handle > 8 GPU (NVidia bypasses this by making drivers independent of xorg)
So it is "hard" w/ Nvidia but downright impossible w/ AMD. Virtualization should work though ... maybe.
April 17, 2012, 04:56:14 AM
Do you guys know what is the limit of GPUs that a typical mobo(bios) can initialize? The fastra2 had 13 GPUs and they needed a custom bios to boot.
I have tried to look into this virtualization business and damn there's a little of good howtos available. The esxi could have been an easy solution but 5970s won't play nice with that.
I have tried to look into this virtualization business and damn there's a little of good howtos available. The esxi could have been an easy solution but 5970s won't play nice with that.
This depends on the motherboard... For instance, my GA-990FXA-UD7 wouldn't even boot with 8 GPUs until I started disabling some devices in the BIOS (USB3, eSata, etc...). After a couple of tries, everything works and this box has been running stable for several months now.
Yeah, you need to disable as many devices filling up address space blocks as possible. It might be able to get more than 4 on 32 bit Linux just by doing that, and also by putting less memory in it (but its hard to get less than 1gb DDR3 sticks that aren't crap or volted wrong as the DDR3 manufacturing process was targeting chip sizes for 1gb minimum sticks).
April 17, 2012, 01:55:48 AM
BAMT is worthless shit if it isn't a 64 bit Debian build already. You can't even get more than 4 cards working in 32 bit Linux, sometimes only 3.
Nope, BAMT is still 32-bit.
April 17, 2012, 01:55:21 AM
Do you guys know what is the limit of GPUs that a typical mobo(bios) can initialize? The fastra2 had 13 GPUs and they needed a custom bios to boot.
I have tried to look into this virtualization business and damn there's a little of good howtos available. The esxi could have been an easy solution but 5970s won't play nice with that.
I have tried to look into this virtualization business and damn there's a little of good howtos available. The esxi could have been an easy solution but 5970s won't play nice with that.
This depends on the motherboard... For instance, my GA-990FXA-UD7 wouldn't even boot with 8 GPUs until I started disabling some devices in the BIOS (USB3, eSata, etc...). After a couple of tries, everything works and this box has been running stable for several months now.
Jump to: