Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 21.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: BCMan on February 27, 2012, 12:49:29 AM

Retarded design. Who will install so much cards and without any space between them.

...because I won't? The pic with 7x 5870s so close together was just a 24 hour trial run. Here is the frame I hope to obtain for this rig:

Anyways, for ultimate density I'd want to watercool 18x 5970s. Damn that would be epic. With proper waterblocks, they could all fit in a 4 U rack case, except for the hoses and shit.

BCMan

hero member

Activity: 535

Merit: 500

Quote from: DiabloD3 on February 27, 2012, 12:50:50 AM

Quote from: BCMan on February 27, 2012, 12:49:29 AM

Retarded design. Who will install so much cards and without any space between them.

Because thats meant for 10/40/100gbit cards, which are one slot wide. I'm laughing because you actually thought it was meant for video cards.

Well, yes. lol.

DiabloD3

legendary

Activity: 1162

Merit: 1000

DiabloMiner author

Quote from: BCMan on February 27, 2012, 12:49:29 AM

Retarded design. Who will install so much cards and without any space between them.

Because thats meant for 10/40/100gbit cards, which are one slot wide. I'm laughing because you actually thought it was meant for video cards.

BCMan

hero member

Activity: 535

Merit: 500

Retarded design. Who will install so much cards and without any space between them.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: DiabloD3 on February 27, 2012, 12:39:52 AM

Yeah, the board is basically perfect for mass mining... also, it doesn't really matter if two or three of the slots are blocked, you can only use 16 slots (ie, 8 cards) without fucking with virtualization, the last two are wasted anyways.

Yeah, except for the price - oh the price. If I can't get more dense than 16 GPUs or greater, I am going to be pissed. That's why I have been looking into the virtualization. I have yet to hear from anyone who has attempted virtualization with more than 1 card (or even more than 8).

DiabloD3

legendary

Activity: 1162

Merit: 1000

DiabloMiner author

Quote from: rjk on February 27, 2012, 12:32:45 AM

Quote from: DiabloD3 on February 27, 2012, 12:24:45 AM

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

Those are 2x EPS12s. However, they get in the way of 2 or 3 slots when in use, which is why I am using the terminal blocks. The SHB I currently have uses 2x low voltage xeons, so not much power at all. However, additional power is needed by other parts on the board such as the PCIe fanout switches, and other SHBs can use hugely powerful CPUs such as dual hex core Xeons. So the board is rather overdesigned for power. (200 amps max at 12 VDC alone, just for the board).

Yeah, the board is basically perfect for mass mining... also, it doesn't really matter if two or three of the slots are blocked, you can only use 16 slots (ie, 8 cards) without fucking with virtualization, the last two are wasted anyways.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: DiabloD3 on February 27, 2012, 12:24:45 AM

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

Those are 2x EPS12s. However, they get in the way of 2 or 3 slots when in use, which is why I am using the terminal blocks. The SHB I currently have uses 2x low voltage xeons, so not much power at all. However, additional power is needed by other parts on the board such as the PCIe fanout switches, and other SHBs can use hugely powerful CPUs such as dual hex core Xeons. So the board is rather overdesigned for power. (200 amps max at 12 VDC alone, just for the board).

DiabloD3

legendary

Activity: 1162

Merit: 1000

DiabloMiner author

I'm not going to scroll through this thread to ask this question

Those two 8 pin plugs on the board, are those EPS12s or PCI-E 8s? Because if they're EPS12s (384w * 2) and they're wired to the PCI-E 12v rails, you have enough power for 5 5970s/6990s (ten slots worth, so you're still short four cards), or 18 cards of any kind (typically 25-35 watts off the slot, you have a budget of 42 watts).

If those are PCI-E 8s, then thats 150w each so.... 4 5970/6990s, or 9 or 10 normal cards.

I'm also assuming the computer is a low wattage model and only needs the 12v off the ATX24.

check_status

full member

Activity: 196

Merit: 100

Web Dev, Db Admin, Computer Technician

I think I might have an interesting lead.
PICMG 1.3 board with a chipset that supports IOMMU and also the chipset can be built using Coreboot linux.
The board:
http://www.ryaim.com/products/singlebd/new_SBC/SPCIE-3600AM2.htm

This board uses the Nvidia MPC55 chipset, which Coreboot has been built in to, but for a Tyan S2912.
http://www.coreboot.org/Supported_Motherboards#Servers

If Coreboot is better to work with like for allocating resources for pci devices above 4G to expand the # of GPU's that are recognized similar to the fastra II project, that would be sweet.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: check_status on February 25, 2012, 02:19:34 AM

I wonder how the mapping would go with 16x 5970's, 4 per OS, conflicts?

Good question, I have no idea. From what I read so far, it may not even be able to boot unless I get a custom BIOS the doesn't initialize the cards on boot.

Quote from: xurious on February 25, 2012, 09:05:21 PM

Splurge the 20 bucks at get the 1230 which supports hyperthreading. You're spending bucketloads on this already.

Totally, if it's only 20 bucks more. Cheap already at $220

xurious

sr. member

Activity: 413

Merit: 250

Quote from: rjk on February 25, 2012, 12:22:30 AM

http://www.newegg.com/Product/Product.aspx?Item=N82E16819115088 <---- Xeon E3-1225 $219.99

Splurge the 20 bucks at get the 1230 which supports hyperthreading. You're spending bucketloads on this already.

check_status

full member

Activity: 196

Merit: 100

Web Dev, Db Admin, Computer Technician

Looks like the chipset options on intel for VT-d are Q35, Q45, Q67, and C20*. So far your board choice looks pretty good.
http://forums.bit-tech.net/showthread.php?s=66d3b34a50d14b446bdb2987a3f069a1&p=2965518#post2965518

But you keyboard commandos are a little quicker than I am. Cheesy

I wonder how the mapping would go with 16x 5970's, 4 per OS, conflicts?

Your board:
Q67 for QG2 version; B65 for QVG version; C206 for WG2 version

Some additional chipsets that have VT-d:
VT-d is enabled on the following chipsets:

Intel Q35 GMCH with ICH9 DO (Bearlake chipset)

The following chipsets have VT-d capability, but OEMs may not have enabled in systems based on these:
Intel X38
Intel X48
VT-d will be enabled on these future products:
Intel Q45 (Eaglelake)
For Intel Desktop Boards, these have VT-d support enabled:
Intel DQ35JO
Intel DQ35MP
Intel DX38BT
Intel DX48BT2
These future Intel Desktop Boards will have VT-d support:
Intel DQ45CB
Intel DQ45EK

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

So I've been looking around, but it appears that none of the available options come with processors. Instead of resurrecting the Core 2 series from the dead, it would probably be best to spend a hundred bucks more and upgrade to socket 1155. Bearing that in mind, how does this look for a parts list?

http://www.l-trondirect.com/pce-5126wg2-00a1e.html <---- SHB with C206 chipset $570.38
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115088 <---- Xeon E3-1225 $219.99

If I wanted to stick to socket 775 and the Core 2 series, this seems to be a decent Q35 based board: http://www.ebay.com/itm/230658116351 $399
Then I would need to find a proc to fit that.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: DeathAndTaxes on February 24, 2012, 10:01:25 PM

Quote from: rjk on February 24, 2012, 09:42:28 PM

Quote from: check_status on February 24, 2012, 08:43:52 PM

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

What about something like this:
http://www.l-trondirect.com/pce-5124vg-00a1e.html?gclid=COWr5_GQuK4CFQ40hwodDC_-JA

$491. Has Q35 chipset which supports VT-d.

Hmmmmmmmm. I never considered that there could be other companies making the same things that would be compatible. I guess it is an open specification after all. (PICMG 1.3). I'll look into that and any similar alternatives, and see what I can budget for. Even though Q35 is a little old, it is way newer than what I have now.

EDIT: I just realized that I have been searching for PCIMG the whole time, not PICMG. Undecided

DeathAndTaxes

donator

Activity: 1218

Merit: 1079

Gerald Davis

Quote from: rjk on February 24, 2012, 09:42:28 PM

Quote from: check_status on February 24, 2012, 08:43:52 PM

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

What about something like this:
http://www.l-trondirect.com/pce-5124vg-00a1e.html?gclid=COWr5_GQuK4CFQ40hwodDC_-JA

$491. Has Q35 chipset which supports VT-d.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: check_status on February 24, 2012, 08:43:52 PM

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

Cool, let me know if you find something for less than what I posted earlier. (An E3-1225 SHB for about $1600)

check_status

full member

Activity: 196

Merit: 100

Web Dev, Db Admin, Computer Technician

P67 and Z68 are the southbridge chipsets. My southbridge is SB850, which can run IOMMU but only with a BIOS from Gigabyte tech support.

I've learned a lot reading up on VT-d VT-x issue. Since the SBC uses a board manufactured pre-2005, the chipset on it doesn't support VT-d. According to Intel it requires the chipset to support IOMMU and the CPU to support VT functions.

I'll search around and see if I can find some PICMG with IOMMU.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: check_status on February 24, 2012, 05:51:00 PM

Quote

VT-d is not a feature of the processor itself but rather a feature of the Chipset. As long as the chipset/BIOS combination support VT-d and the processor support VT, there will be VT-d support.

http://www.intel.com/support/processors/xeon5k/sb/CS-031637.htm

I don't know how that could be correct. For instance, I know for a fact that the system I am typing on supports VT (i.e., VT-x), but not VT-d. It is a Z68 system. However, there is a chipset that does support VT-d, and it is the Q67 chipset. You can compare them here: http://ark.intel.com/compare/52812,52816

Compare these 2 processors, as well: http://ark.intel.com/compare/52213,52214
The 2600K certainly supports virtualization, but not VT-d. Here is a table to demonstrate this:

	P67	Z68
2600	VT-d	No
2600K	No	No

Perhaps that bit of info is specific to Xeon 5000 series or something? I don't know.

check_status

full member

Activity: 196

Merit: 100

Web Dev, Db Admin, Computer Technician

Quote

There can be some confusion regarding VT-d support on specific processors. VT-d is not a feature of the processor itself but rather a feature of the Chipset. As long as the chipset/BIOS combination support VT-d and the processor support VT, there will be VT-d support.

http://www.intel.com/support/processors/xeon5k/sb/CS-031637.htm

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: check_status on February 24, 2012, 04:04:22 PM

There are no single core socket 604 cpu's that support VT-x. Cry

VT-x isn't available until "Dempsey" Xeon 5020, 5030 for socket 771.

Edit: There are some dual core 604 sockets, Xeon 7020 that support VT-x. 165w
http://www.compuvest.com/Desc.jsp?iid=1124397 $30

VT-x is hardware virtualization. VT-d is virtualization with directed I/O. Directed I/O means that a virtual machine can get full and unrestricted access to certain bits of hardware, such as video cards. That is what is needed to overcome the GPU limit in the drivers.

Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] - page 21. (Read 169531 times)