Topic: How to expand the number of PCIe devices on a Main Board? (Read 529 times)

Sorry Proper(S)!  How cool is it to refer a poster to their own thread?  One point I will make is in a very few weeks Threadripper II will be released and all the MB vendors will release new UEFIs.  Some have already.  GIGABYTE has released UEFI F10 with AGESA 1.1.0.0.  I think ASRock did also but then removed it.  That start/stop stuff may be memory training.  If you will explain the "weird" boot stuff detail, I will think on it.  I would prefer you use AMD forum, but this is OK if you prefer.  Enjoy, John.

You are referring to my original post on AMD forums, =)

I recently did some tests and got 13 GPUs to post and work, the system performs in a stable manner but it does weird things during boot. It would start then shut down and start again but once it gets past that it boots up fine

 ASUS did not respond with anything useful and abandoned my supports requests without closure.

ProperS,  thanks for the reply.  Here is the link to a thread with a similar experience as your's: https://community.amd.com/message/2864370#comment-2864370.  The user believes and I agree that the Above 4 GB option would allow use of all GPUs that are boot-able.  I assume your MB does not support this option.  What board are you using?  Hopefully a UEFI update will correct this soon.  Threadripper II is rumored to be released next month (August 2018).  Perhaps that will spark some new UEFI updates.  Thanks and enjoy, John.

Hello, I am the original poster that explored this issue.

I want to share my findings as it may be useful to others.

Big thanks to MisterJ for starting a thread here to help!


My initial issue with X399 has been that motherboard would take up to 7 GPUs and then stop posting, all hardware was tested and worked outside of the build.
Although it posted with 7 and worked system tossed errors in the os when one of the devices was used, it will work but just had errors that according to OS logs were resolved(OS used is UBUNTU) and errors posted in the CLI
When i say 7 GPUs i mean number fo GPU devices detected, some video cards have 2 processing units and those are detected as two separate units, this means you are getting 2 GPUs as you connect each card.
The conclusion is that threadripper with default config will work in a stable manner with 6 pcie devices in the default configuration.

I wanted to connect allot more GPUs and for my application, I need allot of ram and a lot of CPU power, threadripper platform was a perfect fit on paper.
But 6 GPU limit kill all of it, I did continue to dig through and got much further

I got information that address allocation may be an issue and tried disabling sound controller, and to my surprise, it booted fine with 8 gpus, then I disabled network controller and it booted with 9. I was on the right track
It became clear from my research and testing that biggest consumer of address space is usb controller,  in fact on asus X399 22 addresses are allocated to USB controller and there is another usb 3.1 controller that also eats space
When usb controller was disabled, system boots even with 20 gpus but OS does not see more than 12.  All 12 GPUs work as expected, i have enough capacity on this CPU to go to 20 but for now, i am exploring what that limit could be.

Maybe it is driver or chipset related if you know of anything let me know

Thanks, remauto1187ma.  Good advice.  I will keep it in mind.  Enjoy, John.

Go into device manager and Disable them there or even uninstall the Drivers for them then Disable.  Dont forget to show hidden device under Device Manager then View Tab then Show Hidden Devices.  Remove all failed previous attempts of an install for GPU including the Audio Drivers that failed.   *HINT* Look for failed codes or exclamation points in device manager.

Thanks, remauto1187ma.  I will be more careful.  Please tell us how to increase the number of PCIe devices allowed by the UEFI.  To date, the only method known is to Disable all the devices not needed.  One real problem is that some MBs do not allow most devices to be disabled.  Perhaps you can tell us how to do this or recommend what to ask the MB Vendor to implement.  Thanks and enjoy, John.

Of course its an Inetl MB...Its an ASUS B250 Mining Expert and thats what that conversation was about. Read ALL of the context.

Thanks much for all responses!  Here is a screenshot from a different user using an ASRock AMD MB.
https://s33.postimg.cc/bq0bc11f3/Above_4_GB.jpg
This user reported that the setting did not work.  ASUS does not seem interested.  Thanks and enjoy, John.

EDIT: The video above is from an Intel MB, not AMD. 

Yes it does indeed. See post#8 for a link to youtube video showing the bios. (not my video)

OP was talking about a X399 Threadripper board. But yeah - ASUS B250 boards, including most Intel ones, have Above 4G.

yup. I have 11x RX580 + 8x P104x100s on that mobo. working just fine

The ASUS B250 mining expert motherboard can be used in conjuction with PCIE slot expanders, you just have to disable unused components. Some guy on youtube (if i recall correctly, Bits Be Trypin is the name of the channel) was able to mine with 21-GPUs on this motherboard.

Note that you need to use mining edition GPUs in addition to your normal GPUs to effectively get beyond the 13-GPU limit.

umm, what?
ASUS B250 Mining Expert is an Intel board, I have one and I'm pretty sure that it has the "above 4g decoding" option in UEFI.
12 GPUs are recognized because Adrenaline Driver for Windows can run up to 12 AMD GPUs max.

Unlikely, because he's using an AMD board and Above 4G is an Intel feature - I'll need to dig into my AMD rig's bios to see what the option was. I'm not sure what are AMD's limits because it could be utilizing the PCI-E lanes differently.
OP, please try running Linux.

Not released yet.

Oh yeah. i need it. Where do to buy it?

Absolutely, I havent heard anything concerning price.  Not even speculation as of yet.  I suspect this year at current coin prices there wont be an 1/8 of the demand that existed for the mining expert 8 months ago for the mining master motherboard.  That could work in our favor and maybe get them for pretty close to the cost of a mining expert.

It's happened on one or two occasions to me, but I keep my rigs securely placed and I personally think the potential savings are worth it on a used B250 Mining Expert- I guess we'll have to see the pricing on the H370 and some reviews of it in actual miners' hands to see if the board is as good as the articles I've seen about it claim. Hopefully a true 20-gaming-card board comes out in the future, though- those mining cards are still hard to find and expensive.

Youve never had that damn PCI slot to USB adapter shut down your whole rig because it got jostled or the wind blew the wrong way?  Id love for my Mining Expert board to have all USB connections instead of PCIe.

The thing is, the H370 still requires you to have special 'Mining edition' cards to reach full capacity if you want to use more than 12-13 cards, and there's really not a lot of differences between it and the B250 board besides a gimmicky USB connection instead of PCI. Your current B250 board is just fine and the H370 will bring little to no improvements if you choose to buy it, and the B250s are likely going to be better in the future as well considering the likely higher pricing of the newer board.

actually it has "above 4G decoding" setting.

Check out Asus! They are coming out with the H370 Mining Master. 20 "slots", usb (No pci) connection.  https://www.asus.com/us/Motherboards/H370-MINING-MASTER/

Thanks, nitrobg.  By disabling as much PCIe stuff (USB Root controllers, NICs, etc) the ASUS board will post with 16 GPUs but only 12 are recognized by W10.  The MB does not have "Above 4GB" setting in UEFI.  Perhaps all 16 could be used if available.  Thanks all for the help.  I learned a lot.  Enjoy, John.

Nearly all 4+ slot motherboards have Above 4G decoding available in the bios. It is a necessary perquisite, but it does not guarantee the proper function of the GPUs. It seems like there is a hard GPU limit of 13 cards, which can only be lifted by using mining specific cards.

Check my findings here:
https://bitcointalksearch.org/topic/guide-how-to-run-up-to-13-gpus-on-most-motherboards-2814284

https://www.youtube.com/watch?v=_fNpfOL7ZZA

Thanks, remauto1187ma.  Does the UEFI/BIOS give the user access to the "Above 4GB MMIO"?  Thanks and enjoy, John.

ASUS B250 Mining Expert m/b will do 13 GPU's and another 6 P series cards for a total of 19.  They can be had for around $100 new and $50-$70 used on Ebay.

Thanks, Raggie.  I could not find the sub-forum you recommended in the Move Thread pull down list, so I placed it here.  I do hope someone will help here.  Thanks and enjoy, John.

Hey friend, i see your question is more specific for mining. You can go here : https://bitcointalk.org/index.php?board=160.0
There are many peoples with knowledge of mining there because it is a mining forum section. i believe many will give you solutions. good luck

Thanks, adaseb, but I am looking at a mining type system with 10 to 15 GPUs.  This system is not being used for mining but other intensive computational work.  The UEFI will not POST if some limit of devices is exceeded.  I am looking for what that limit is and how to expand it.  I have tried to help others that were building mining rigs and ran into the same problem, apparently.  I hope some of you will be able to tell me how to circumvent this limit.  Thanks and enjoy, John.

Check your motherboards manual... on some motherboards the PCIe lanes can be shared with different slots.

So if you use one slot, another slot might become unavailable.

And also in your motherboard manual is a diagram which shows where each of the lanes are going to. Sometimes certain onboard devices can use some PCIe lanes, like SATA, sound, network card, etc.

It appears that for newer MBs there is an MMIO memory limit that is usually 4 GB which can limit the number of PCIe cards (eg, GPUs).  There also appears that even if this limit is increased the system may not POST.  By disabling some unused PCIe devices the system POSTs.  This seems to me to be a UEFI limit on how many peripherals can be handled regardless of the MMIO space available.  Right now the system I am talking about is the AMD X399 Threadripper and several MB vendors (ASRock, ASUS, etc).  As you can tell I am not very knowledgeable here and any advice and information would help very much.  Thanks much and enjoy, John.