I don't usually mine monero, but my gut says you should be able to get a bit more from your XFX RX 580, and quite a bit more from your VEGA 56. Have you considered using XMR-Stak-AMD for mining? Have you adjusted the clock settings at all?
The Nvidia 1060 may be better suited to mining on equihash or lyra2rev, but if consolidating your proceeds on one coin is your goal, then switching the 1060 to mine a different coin by itself might not be a convenient option, since NiceHash isn't doing too great right now.
As far as your OS is concerned, Ubuntu might not be a great choice if you plan on having both AMD and Nvidia GPUs in the same system. Getting the drivers to play nicely with each other can be quite a pain on most Linux distributions.
I have one ribbon riser for the possibility of running the 1060. If you look at the setup of the board the 1st PCI3 is located at the bottom of my case...perfect for a single NVS card but not so for a double GFX card.
So what I did was remove the bottom of 1 of the servers and built out an extension. Laid the server on its side and built a Wood template before I rivet and apply metal.
Actually I'm into veneering as a hobby as well and have thought about building out a nice walnut wall piece.. an Attractive miner.
I put the rx in the 1st slot PCI-e x16 which is controlled by cpu1, middle empty for airflow, vega in slot 3 PCI-e x16 controlled by the second CPU.
Currently getting 62mH/s Eth/2000mH decred.. letting this run for a day before I turn on a cpu miner.
These are tanks with 900w gold PSU thick PCB and very durable. I'm at 600w from the wall....I know I'm running it hot for testing.
FYI got them for 200 each so way less than buying a lesser cpu rig and board.
any other suggestions would be great as I'm open to ideas. Power is not a thing since I have a good day job, I just want to mine and yeas could have put that $ into a coin.. but then where is the fun
I love Ubuntu but your right a pain.. I spent more time sorting things out at first than mining only to get windows keys and Hence the transition to nicehash.. I was done fight Linux and AMD