Topic: How to expand the number of PCIe devices on a Main Board? (Read 507 times)

Sorry Proper(S)!  How cool is it to refer a poster to their own thread?  One point I will make is in a very few weeks Threadripper II will be released and all the MB vendors will release new UEFIs.  Some have already.  GIGABYTE has released UEFI F10 with AGESA 1.1.0.0.  I think ASRock did also but then removed it.  That start/stop stuff may be memory training.  If you will explain the "weird" boot stuff detail, I will think on it.  I would prefer you use AMD forum, but this is OK if you prefer.  Enjoy, John.

You are referring to my original post on AMD forums, =)

I recently did some tests and got 13 GPUs to post and work, the system performs in a stable manner but it does weird things during boot. It would start then shut down and start again but once it gets past that it boots up fine

 ASUS did not respond with anything useful and abandoned my supports requests without closure.

ProperS,  thanks for the reply.  Here is the link to a thread with a similar experience as your's: https://community.amd.com/message/2864370#comment-2864370.  The user believes and I agree that the Above 4 GB option would allow use of all GPUs that are boot-able.  I assume your MB does not support this option.  What board are you using?  Hopefully a UEFI update will correct this soon.  Threadripper II is rumored to be released next month (August 2018).  Perhaps that will spark some new UEFI updates.  Thanks and enjoy, John.

Hello, I am the original poster that explored this issue.

I want to share my findings as it may be useful to others.

Big thanks to MisterJ for starting a thread here to help!


My initial issue with X399 has been that motherboard would take up to 7 GPUs and then stop posting, all hardware was tested and worked outside of the build.
Although it posted with 7 and worked system tossed errors in the os when one of the devices was used, it will work but just had errors that according to OS logs were resolved(OS used is UBUNTU) and errors posted in the CLI
When i say 7 GPUs i mean number fo GPU devices detected, some video cards have 2 processing units and those are detected as two separate units, this means you are getting 2 GPUs as you connect each card.
The conclusion is that threadripper with default config will work in a stable manner with 6 pcie devices in the default configuration.

I wanted to connect allot more GPUs and for my application, I need allot of ram and a lot of CPU power, threadripper platform was a perfect fit on paper.
But 6 GPU limit kill all of it, I did continue to dig through and got much further

I got information that address allocation may be an issue and tried disabling sound controller, and to my surprise, it booted fine with 8 gpus, then I disabled network controller and it booted with 9. I was on the right track
It became clear from my research and testing that biggest consumer of address space is usb controller,  in fact on asus X399 22 addresses are allocated to USB controller and there is another usb 3.1 controller that also eats space
When usb controller was disabled, system boots even with 20 gpus but OS does not see more than 12.  All 12 GPUs work as expected, i have enough capacity on this CPU to go to 20 but for now, i am exploring what that limit could be.

Maybe it is driver or chipset related if you know of anything let me know

Thanks, remauto1187ma.  Good advice.  I will keep it in mind.  Enjoy, John.

Go into device manager and Disable them there or even uninstall the Drivers for them then Disable.  Dont forget to show hidden device under Device Manager then View Tab then Show Hidden Devices.  Remove all failed previous attempts of an install for GPU including the Audio Drivers that failed.   *HINT* Look for failed codes or exclamation points in device manager.

Thanks, remauto1187ma.  I will be more careful.  Please tell us how to increase the number of PCIe devices allowed by the UEFI.  To date, the only method known is to Disable all the devices not needed.  One real problem is that some MBs do not allow most devices to be disabled.  Perhaps you can tell us how to do this or recommend what to ask the MB Vendor to implement.  Thanks and enjoy, John.

Of course its an Inetl MB...Its an ASUS B250 Mining Expert and thats what that conversation was about. Read ALL of the context.

Thanks much for all responses!  Here is a screenshot from a different user using an ASRock AMD MB.
https://s33.postimg.cc/bq0bc11f3/Above_4_GB.jpg
This user reported that the setting did not work.  ASUS does not seem interested.  Thanks and enjoy, John.

EDIT: The video above is from an Intel MB, not AMD. 

Yes it does indeed. See post#8 for a link to youtube video showing the bios. (not my video)

OP was talking about a X399 Threadripper board. But yeah - ASUS B250 boards, including most Intel ones, have Above 4G.

yup. I have 11x RX580 + 8x P104x100s on that mobo. working just fine

The ASUS B250 mining expert motherboard can be used in conjuction with PCIE slot expanders, you just have to disable unused components. Some guy on youtube (if i recall correctly, Bits Be Trypin is the name of the channel) was able to mine with 21-GPUs on this motherboard.

Note that you need to use mining edition GPUs in addition to your normal GPUs to effectively get beyond the 13-GPU limit.

umm, what?
ASUS B250 Mining Expert is an Intel board, I have one and I'm pretty sure that it has the "above 4g decoding" option in UEFI.
12 GPUs are recognized because Adrenaline Driver for Windows can run up to 12 AMD GPUs max.

Unlikely, because he's using an AMD board and Above 4G is an Intel feature - I'll need to dig into my AMD rig's bios to see what the option was. I'm not sure what are AMD's limits because it could be utilizing the PCI-E lanes differently.
OP, please try running Linux.

Not released yet.

Oh yeah. i need it. Where do to buy it?

Absolutely, I havent heard anything concerning price.  Not even speculation as of yet.  I suspect this year at current coin prices there wont be an 1/8 of the demand that existed for the mining expert 8 months ago for the mining master motherboard.  That could work in our favor and maybe get them for pretty close to the cost of a mining expert.

It's happened on one or two occasions to me, but I keep my rigs securely placed and I personally think the potential savings are worth it on a used B250 Mining Expert- I guess we'll have to see the pricing on the H370 and some reviews of it in actual miners' hands to see if the board is as good as the articles I've seen about it claim. Hopefully a true 20-gaming-card board comes out in the future, though- those mining cards are still hard to find and expensive.

Youve never had that damn PCI slot to USB adapter shut down your whole rig because it got jostled or the wind blew the wrong way?  Id love for my Mining Expert board to have all USB connections instead of PCIe.