OK, I'm not using X6500 boards, and my array is half the size of yours (25x Ztex 1.15x boards, plus my initial 1.15d dev board - forgetting the 9 Ghash of GPUs, costing me a fortune and to be switched off soon...) - but I'm having aggro with my systems recognising all the USB devices on the bus.
Theoretically, I should be able to connect 127 of the Ztex single boards to one computer, on one USB bus.
However, this just isn't happening... and that is fucked up, fucked up..
(sorry, listening to Thom Yorke)
There's definitely nothing wrong with the Ztex boards - they're performing brilliantly in idiotically insane environments - ambient air 38˚C, no typo, cheap £3 4-port daisy-chained Amazon USB2 hubs, and MUCH WORSE than I imagined... 5-to-1 2.1/5.5mm barrel plug daisy-chain cable rated at 60W but gets very hot to touch under load, and on top of this - unsupported miner platform... I run Mac OS X and recompiled Stefan's entire SDK and bitminer-src for Mac OS X Snow Leopard on both 32-bit (hackintosh - Dell Mini v10 - double unsupported), and 64-bit (proper full-blown Mac Pro workstation loaded to the gills)...
It appears that you have a similarly sized USB hub / chain problem that I do. I need to connect 25 USB cables to one netbook running Mac OS X.
Stefan explained why my cables were hot - and I now have a solution for that problem.
But the issue remains - are these cheapo Amazon 4-port USB2 hubs simply a disaster waiting to happen? What have you used for all the USB connections you need? Or have you used multiple logic boards? One option for me would be to simply plug the USB cables into my modular GPU miners, which have one logic board (with 6 USB sockets, normally) and 4 GPUs. This should work, but I don't want to be running 5 high-power ATX PSUs when I'm intent on selling off the 6950 / 5850 / 5830 / 5770 GPUs (no use to me as a Mac user, other than the 5770s, which I'll keep to make a 4-GPU Hackintosh tower with 16 GB ram and the i5-2500 CPU clocked to the moon, running Snow Leopard of course).
It's overkill to use separate logic boards and CPUs (wasted power). But high quality 10x USB2 hubs cost a lot of £££ and, if they require their own power bricks, then it defeats the object of consolidating points of failure. A single cheap USB2 hub could easily short out (I've taken photos of the internals of these things, and it's embarrasing - I am awful at soldering, but I could do a better job than these chaps).
Also, part of the project goals include the ability to make the entire structure portable, with a single power input and a single Ethernet output. It also has to look good in Jonathan Ive fashion (i.e. the downside of being an Apple hacker and concomitant need for good industrial design...)
Stefan (Ztex) has given me a good wiring diagram to power the FPGAs. But I'm still left with all 25 of the boards needing USB connections. Since USB2 spec allows for 4-port and 7-port hubs (IIRC), any other number (such as the 10-port hubs) is actually a 4-port and 7-port chained together internally.
This is where quality comes in. I've got cheap hubs... but are the crappiest of hubs 'adequate and functional' if the power supply to the FPGAs is clean, low resistance and properly soldered (i.e. no hot spots)?
After all, I'm looking at 12VDC at 5A - and 5 amps is quite a current for a cheap USB2 hub with ethernet-thin internal cabling...
Any top tips? If simply sorting the PSU to present a low resistance to the FPGAs will allow the cheapest of crap USB hubs, then I'll stick with what I've got... but if quality USB hubs are required... I'd appreciate any recommendations for the USB daisy-chain stage. After all, if I ever get to 127 FPGAs on one bus, the USB hubs need to work. I've had problems with even only connecting *5* to a single port...
I'm a little confused by what's going on here. Maybe you can help me understand before I get too far debugging this.
What it sounds like is that the power supply ground is not good enough, so some significant current is returning over the USB cables, making them hot. Is this your understanding?
If so, I don't think the USB hubs can really be to blame (although powering them could potentially help, since it might help to keep
their grounds from floating relative to the host). The best thing to do is to beef up the grounding to the power supply. This could be done by adding another wire in parallel, increasing the wire guage, shortening the wires, or making sure they are "star connected," meaning they all have their own personal path back to a common ground point, instead of the grounds all running in series.
Let me know if I'm on the right track.