[...]
Each board will need its dedicated I2C bus anyway, so why not have a dedicated JTAG bus as well?
We only need one I
2C bus, it just needs to be fragmented into different partitions by a switch. I mentioned one example for such a switch before, the NXP PCA9547PW. The reason why I
2C needs to be partitioned is the limited availability of addresses on the bus. That is not a problem for the JTAG bus, though. Logically, you can make it as long as you like. Electrically, you need drivers in the TCK and TMS lines for a design with many chips.
Given that there wasn't more than one I
2C bus planned and that no more than one JTAG chain are needed, can you clarify why you think more JTAG chains are needed?
Very long JTAG scan chains won't perform well. Boot up time will be huge, and if you use JTAG for data communication, the overhead will greatly increase on a long scan chain. This probably isn't an issue with like 4 FPGAs in the system, but it is one with 100 FPGAs. While we probably won't build such boards right now, we might want to do this at some point, so I'm just proposing to design the interface in away that allows for separate I2C buses and JTAG chains on each card, as the additional cost will not be an issue.
For the cheap boards, you could just connect to the USB pins via the DIMM connector, and basically just have a hub and power supply on the backplane. The more expensive boards might have an ARM and ethernet.
So not use the JTAG or I
2C signals on the bus connector at all, just the USB D+ and D- lines? That is a very interesting idea: it simplifies the design a lot if it works:
none of the non-supply signals I mentioned in my last post are needed in that case, as the backplane can detect the presence of a DIMM the "USB" way. So a simple backplane contains wires and a couple of mini-USB connectors? Or it contains a home-grown USB-hub? (I am limiting myself to a cheap backplane in this discussion because the intelligent one with a CPU can be build on top of the cheap design in a second step).
This is basically shifting the interface chip completely on the DIMM, removing (by design, not material cost) the overhead of supporting hybrid DIMMs. Of the different options, it is not the cheapest, but certainly elegant:
- slave-only DIMMs, USB-chip only on backplane: cheapest, JTAG and I2C on bus
- hybrid DIMMs, USB-chip only on DIMMs: mid price, simple bus with only USB, but needs hub somewhere
- hybrid DIMMs, USB-chip both on DIMMs and backplane: most expensive, JTAG and I2C on bus
I would expose the I2C, JTAG and USB interfaces on the DIMM, so that the backplane can decide which one it wants to use. Right now we'll probably go for USB-only, which greatly simplifies backplane complexity and cost (the backplane is basically just a USB hub with DIMM sockets and power distribution), but more intelligent backplanes (which might be designed later) might have a processor that talks I2C and JTAG natively, so there's no reason to add the USB overhead there.
[...]
Oh, and don't forget to add a means for boards to interrupt the backplane, e.g. when a share was found or keyspace was exhausted.
Is that actually needed? I agree that a later backplane that contains a CPU may make good use of the interrupt, but for the USB based devices it is only a question of how much data to transmit: you still need to use polling because USB does not have a direct IRQ. I admit that reading the GPIO value of an FT2232 connected to the IRQ signal is quicker than reading the JTAG chain. But how bad is that for even 10 boards each with 16 FPGAs?
I'm talking about a dedicated IRQ line on each board, that can be triggered by all of the FPGAs. Depending on the number of available GPIOs, we might want to have one IRQ line per FPGA that connects to the FTDI, and or them together to some pin on the DIMM, which the backplane can then use to determine whether the card needs to be polled. While this exposed IRQ pin is probably worthless for a USB-based backplane, it can possibly increase efficiency for more intelligent backplanes, and it adds virtually no cost. We should really design the interface with flexibility and future expansion in mind.
In case you meant adding the USB signals of the hybrid DIMM to the JTAG and I2C bus as a third connection protocol: isn't this getting more complicated than it needs to be? You truly need only the JTAG connection or the USB connection. Adding I2C to JTAG makes sense because it allows you to read the description and serial number data of the DIMM. Adding I2C to USB makes no sense: the USB chip on the DIMM contains the I2C master for the local I2C bus that does not go beyond a single DIMM. The same for JTAG: if there is USB on the backplane connector, what is the point of adding JTAG to it? You would have to pay attention to who drives the individual signals, preventing burning out the interface chips if both try to send data at the same time.
I2C is a multi-master bus, so we can easily expose this one to the DIMM. Level shifters might get a bit complicated though because it has a bidirectional data line. We can possibly get around this by just using 2.5V or 3.3V for the FPGA's I/O bank that the bus is connected to.
For JTAG, I'd hope that the FTDI or whichever chip we'll use has a way to signal it that it should tristate its outputs.
This is a point that will need further investigation.
The bandwith requirement for the hashing is not the point, you need ca. 800 bit! for the data, and receive 32bits , both with severe protocol overhead, this is not the limiting factor i think. but its quite complicated to implement. i am workling on this since i have a card with 2 largish stratix and no documentation, so jtag is probably the easiest way to get the card running. But you have to trace which workload is running on which fpga and to associate the returning nonce. Unfortunately my jtagd oder quartus_stp tends to produce errors which would render the whole systems workload useless (except you have a persistent database of course).
So i would prefer a serial connection per FPGA or SPI, with SPI as the more adequate solution. But there is a serial solution that works already.
I2C, well it could be implemented with FPGA too, but i think spi is a better was due to the clock, which omits the need for clock recovery and so on plus the higher speed of spi.
I2C is specified for up to 400 kbit/s, including protocol overhead. I really wouldn't like to go for more than 10% bus utilization on average, to keep latencies low. This means that there should be no more than about 140GH/s per bus, which seems to be a ridiculously high number even for ASICs. So I2C should be suitable for this
Nevertheless I'd let the backplane choose whether to join the I2C buses of the cards with address space extenders, or whether to use dedicated I2C masters per card. This seems to not need any additional effort.
I don't see how SPI is any better than I2C. It will need a dedicated CE signal for each and every chip, limiting the number of chips per card to less than the 127 limit imposed by I2C (which could be worked around with address space extenders), and its clock recovery works basically the same way as I2Cs. It might be designed for higher clock speeds, but does neither support multiple masters, nor is it really a flexible bus system.
I own an LX150 dev kit, and the necessary license to do compiles for it, so as long as the license allows me to distribute bitstreams everything should be fine. At the very least I can verify designs until someone is confident enough to fork out the cash for their own license. 
