PCI-E 1-to-7 extension card, anyone tried? - page 2.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: ice_chill on April 07, 2012, 04:16:44 PM

For VT-d all you need is a motherboard based on Intel Q series chipset, Q35/Q45 for 775, Q57 for 1156 and Q67 for 1155. It doesn't have to be an expensive server. Ofcourse CPU must support standard VT.

Take a look at my sig link. Wink

Can't get much other than server parts for it. If you know of an inexpensive, VT-d capable SHB to fit on it, by all means please let me know. I've been looking at a C206 based board, because I want to ensure compatibility for the future, as well as giving me sufficient RAM headroom for all the VMs I need to run.

bulanula

hero member

Activity: 518

Merit: 500

Quote from: Bigpiggy01 on April 07, 2012, 10:11:42 AM

I don't have a clue about intel as I'm AMD all the way.

But this is running atm with 11 GPUS using qemu-kvm and it's going really nicely

The 8th GPU issue is fixed by not having it on the mobo during driver install so that's a fairly simple fix. Atm I'm working on a proper rig case for 2x extender and MSI GD 70 mobo including PSU mounts.

I fully expect that 2-3 of these extenders attached to a single rig is fully doable by instead of assigning individual GPUs to each VM assigning each controller to each VM in order to simplify the process. If anyone has any kind of experience with doing something like that please share

I am sure you are bound to hit some limit sooner or later.

32 GPUs I think is the max even for Nvidia per one system board.

ice_chill

sr. member

Activity: 336

Merit: 250

Few questions:

Will the board take triple slot cards in all slots ( such as ASUS HD7970 DirectCU II )
When can you ship to UK and how much ?

Thanks.

ShadowOfHarbringer

legendary

Activity: 1470

Merit: 1006

Bringing Legendary Har® to you since 1952

Quote from: macbook-air on February 16, 2012, 03:30:44 AM

Price: ~US$230

Pictures:
• http://vga.zol.com.cn/269/2699569.html
• http://item.taobao.com/item.htm?id=14062839037

ice_chill

sr. member

Activity: 336

Merit: 250

For VT-d all you need is a motherboard based on Intel Q series chipset, Q35/Q45 for 775, Q57 for 1156 and Q67 for 1155. It doesn't have to be an expensive server. Ofcourse CPU must support standard VT.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: Bigpiggy01 on April 07, 2012, 10:11:42 AM

I don't have a clue about intel as I'm AMD all the way.

But this is running atm with 11 GPUS using qemu-kvm and it's going really nicely

The 8th GPU issue is fixed by not having it on the mobo during driver install so that's a fairly simple fix. Atm I'm working on a proper rig case for 2x extender and MSI GD 70 mobo including PSU mounts.

I fully expect that 2-3 of these extenders attached to a single rig is fully doable by instead of assigning individual GPUs to each VM assigning each controller to each VM in order to simplify the process. If anyone has any kind of experience with doing something like that please share

This is good news. I have been in discussions with Advantech about lending me a VT-d capable host board for 30 days, I guess we will see how that pans out. They haven't tested it with my board, but their engineers say there isn't any reason that it shouldn't work.

Bigpiggy01

hero member

Activity: 1138

Merit: 523

I don't have a clue about intel as I'm AMD all the way.

But this is running atm with 11 GPUS using qemu-kvm and it's going really nicely

The 8th GPU issue is fixed by not having it on the mobo during driver install so that's a fairly simple fix. Atm I'm working on a proper rig case for 2x extender and MSI GD 70 mobo including PSU mounts.

I fully expect that 2-3 of these extenders attached to a single rig is fully doable by instead of assigning individual GPUs to each VM assigning each controller to each VM in order to simplify the process. If anyone has any kind of experience with doing something like that please share

ice_chill

sr. member

Activity: 336

Merit: 250

Very interested in these, power should not be a problem. I wonder how these would run if you connect 2 to 1 PC for 14 GPUs with VT-d.

Also has the problem been solved, where running an 8th GPU on the motherboard would cause issues ?

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Just a quick word of warning on the new model you need a MONSTER PSU to power up the board properly the two 6 pin power in lines each need around 350W for the board to work properly.

If you happen to have a large single rail PSU no problem but getting that much juice from single lines on a multi rail psu is a bitch. Should you happen to have a nice single rail it's easy as pie. Atm I'm working on a rig case to hold two of these babies along with a mobo. Please note this power is not used by the board it only actually uses about 6-10W (which adds up with what the supplier states on Taobao) it just needs it to meet the pci-e standards.

Some of you guys have been asking for prices on these boards in BTC all I can say atm is bear with me as I'm still working on freight pricing, this is a much more complicated procedure from here than it initially seems as A I don't feel comfortable shipping these boards as regular courier freight (I had two damaged ones arrive initially) but rather as fragile freight furthermore B they have a pretty awful volume to weight ration making air freight overly pricy Sad

.

PatrickHarnett

hero member

Activity: 518

Merit: 500

Could be fun, and they are not too expensive.

Beaflag VonRathburg

sr. member

Activity: 472

Merit: 250

I'm interested in one of these boards as I have a need for higher density to free up some space. The issues though have me slightly deterred. Macbook-air's setup seems to work with 3 gpus. Bigpiggy01, have you attempted pulling a couple cards out of the setup to see if it is the number of cards causing the issue or is it just the 6950s? I was having major boot issues with my 6870s in 11.04 as well. I don't think the 6 series cards and linux like each other very much. My plan was to use a 5870 in the zero video out slot and the 6870s in the others, but I just stuck with windows and it has been running fine for a month.

At minimum could we get a list of people interested in ordering some of these boards once they're confirmed working so we can get a bulk price break. I'll start as number one:

1. Beaflag VonRathburg - 1 board
2.
3.
.
.
.
etc...

Copy, add your name, and paste.

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Quote

I'm guessing you are using KVM and IOMMU?

qemu-kvm and IOMMU Grin

One of the things that has been giving me trouble has been linux and the linux amd drivers not liking my saphire 1gb 6950s (the ones that don't unlock but the memory down clocks nicely even under windows). dISh tweaked that during the initial install that worked out ok but even repeating what he did initially it hasn't worked for me, as I am like I said a linux idiot. (and before someone brainy tells me to boot with nomodeselected in grub, that isn't enough it's a driver bug apparently specific to the card I was dumb enough to decide on for this build)

The driver issue seems to be specific to this card only as I tried with a pair of 6990s and some 5870s and had no issues what so ever.

Under WIN7 it boots up and detects just fine provided that you remove the GPU from the mobo during the initial driver install.

rjk

sr. member

Activity: 448

Merit: 250

1ngldh

Quote from: Bigpiggy01 on April 04, 2012, 06:31:23 AM

Nope with virtualization you can get beyond that by assigning gpus to individual VMs.

Sunbreak had a working setup at one point so does I believe Luke-jr.

Embarrassed

I had it running with 11 GPUS as well at but then decided to do a :tidier set up and ended up breaking it. So getting that working again will have to wait until dISh comes over to help me out as I'm kind of a linux idiot.

It's a shame you are testing on such an unstable platform, but if you are able to test on a bigger board without the special expansion card, I would be interested in the results. So far, the expansion card seems to be causing most of the instability, am I right?

I'm guessing you are using KVM and IOMMU?

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Nope with virtualization you can get beyond that by assigning gpus to individual VMs.

Sunbreak had a working setup at one point so does I believe Luke-jr.

Embarrassed

I had it running with 11 GPUS as well at but then decided to do a :tidier set up and ended up breaking it. So getting that working again will have to wait until dISh comes over to help me out as I'm kind of a linux idiot.

bulanula

hero member

Activity: 518

Merit: 500

Why need to virtualize ?

8 is the GPU limit ...

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Long promised pics:

Problems so far,

The connector only being 20cm effectively means that you need to mount the mobo on a tray especially if you want to run 69XX or the slightly longer reference 58xx. I have tried replacing the two sata cables that the board ships with with 2x sata III in 35cm I get a board working light as per the instructions on
http://item.taobao.com/item.htm?id=14062839037 but I get weird detection errors.

Detection of GPUs is flaky at best under both ubuntu 12.04 11.10 and Win7. Even if you remove GPUS from the main mobo.

Atm this still need quite a bit of improvement those being A) better preferably longer connectors if you want to use the sata setup. B) An atx power connector from the board so you could use it to power a mobo as well. C) bringing back the pci-e 16-16 connector.

EDIT: the setup in the picture is 8x Saphire 1gb 6950 with 7 on the extender and 1 on the MSI GD-70 when running they do 2900 m hash in total under Win7 I will post hash results once I get proper virtualization up and running.

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Pulling my hair out atm with getting this to work with a VM with GPUs assigned to it.

Pics in the morning.

juve4v

hero member

Activity: 505

Merit: 500

News ?

Bigpiggy01

hero member

Activity: 1138

Merit: 523

Sorry guys,

I'm having trouble with my PSU for this build so pics will have to wait another 24 or so hours.

What I will say atm is don't get overly excited. This supplier has ISSUES to work on like improper placing of holes in the board for support legs or fitting into rack mounting and they can't even pack domestically shipped product so it arrives undamaged Sad

(only snapped off pci-e locks but still damage). The sata connectors are a nice touch but the ones that ship with the board are well on the short side for anything but a motherboard next to the splitter/bridge.

I'll do proper pics etc once I have proper lighting in the morning.

fabrizziop

hero member

Activity: 506

Merit: 500

I'd be interested in buying this product, message me when someone starts selling these things in the US.

Topic: PCI-E 1-to-7 extension card, anyone tried? - page 2. (Read 11311 times)