Here's the situation:
6 1070 GPUS plugged into an ASUS Z270A motherboard (has 7 PCIe slots, the GPUs are using X1 to X16 Risers).
The OS Can see the GPUS, as you can see below. But some are "unclaimed" and it appears the NVIDIA driver does not "claim" them.
This is a stock Ubuntu 16.04 SERVER install, onto which the CUDA 8.0 driver has been installed.
(Driver is installed by a script that shells into the box to do it, here's the relevant part of the script:)
ssh -n $1 'yes | sudo dpkg -i ~/drivers/cuda-repo-ubuntu1604-8-0-local-ga2_8.0.61-1_amd64.deb'
ssh -n $1 'yes | sudo apt-get update'
ssh -n $1 'yes | sudo apt-get install cuda'
My CUDA app can use the four GPUS it can see just fine. How do I enable the last two?
$ sudo lshw -C video
*-display
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:02:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:130 memory:de000000-deffffff memory:a0000000-afffffff memory:b0000000-b1ffffff ioport:e000(size=128) memory:df000000-df07ffff
*-display
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:04:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:131 memory:dc000000-dcffffff memory:80000000-8fffffff memory:90000000-91ffffff ioport:d000(size=128) memory:dd000000-dd07ffff
*-display
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:05:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:132 memory:da000000-daffffff memory:60000000-6fffffff memory:70000000-71ffffff ioport:c000(size=128) memory:db000000-db07ffff
*-display
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:06:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=nvidia latency=0
resources: irq:133 memory:d8000000-d8ffffff memory:c0000000-cfffffff memory:b8000000-b9ffffff ioport:b000(size=128) memory:d9000000-d907ffff
*-display UNCLAIMED
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:08:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller cap_list
configuration: latency=0
resources: memory:d6000000-d6ffffff ioport:a000(size=128) memory:d7000000-d707ffff
*-display UNCLAIMED
description: VGA compatible controller
product: NVIDIA Corporation
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:09:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller cap_list
configuration: latency=0
resources: memory:d4000000-d4ffffff ioport:9000(size=128) memory:d5000000-d507ffff
lspci shows them all, though:
$ lspci | grep VGA
02:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
05:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
08:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation Device 1b81 (rev a1)