Topic: Modular FPGA Miner Hardware Design Development - page 32. (Read 119320 times)

We should try to keep the two solution as similar as possible: if the external power supply and computer connection for the backplane based cards and the standalone cards is identical, this means they can be interchanged in a hardware sense. If the computer-FPGA interface (e.g. FT2232) and the on-board bus to the FPGAs is identical, this means there is no difference from the programmers point of view between the two different approaches. Only the spending plan of the user differs. Basically, I suggest to specify everything so the single board can be thought of as a motherboard without the slots to exchange FPGAs.

lame.duck got me thinking of using JTAG both for booting the FPGA and for later communication. While I originally thought SPI would be the way to go, this is potentially less portable between different FPGA makers (can all of them boot of SPI, can they boot of a mixed chain?). I am now convinced that using a JTAG ring to talk to the FPGAs is all that is needed. Here are the advantages:

• Industry standard: every FPGA can be configured (and communicated to?) via it.
• Only four lines, no special requirements on routing (if you don't go too long)
• The chips can be software IDed: no need for an extra EEPROM that stores the hardware type
• Even a future ASIC has a good chance of being able to speak JTAG

There are two disadvantages:

• For the backplane based cards: you cannot leave slots unpopulated. If a slot is empty there needs to be a jumper or DIP switch that connects TDI to TDO for that slot.
• For mixing different FPGAs the JTAG signals may have different voltages that are not compatible between different chips. So each daughtercard should plan on having levelshifters from a specified motherboard voltage to the local, correct voltage.

Given what I said, there should be minimal specifications that apply to both solutions. Here a first suggestion:

• The (mother) board has one USB-B connector for host communication and one Molex 8981 connector for power supply.
• Only the 12V pins (+12V:1, GND:2+3) on the Molex are used (for simplicity).
• The USB connector attaches to a FT2232 chip that is connected to a JTAG chain via its MPSSE.
• The FT2232 operates bus powered (works without having +12V connected).
• The PWREN# port of the FT2232 controls the on-board power supplies that convert the +12V to lower voltages. They are only on if the USB device is enumerated.
• The presence of +12V (or the lower voltages?) can be read back on one of the GPIOs of the FT2232 (say: GPIOL0, H=voltage present, L=error).
• The FT2232 can control an LED via a GPIO (say: GPIOL3, L=off, H=on).
• Should we specify a vendor and device ID (needs a small EEPROM) or leave it at the default values for FTDI?
• The JTAG signal connects all FPGAs in one ring.
• All FPGAs have the same connected signals:
  • JTAG signals: TCK, TDI, TDO, TMS
  • Clock signal
  • For discussion; open collector IRQ signal (wire-or). If present need to specity its connection to the FT2232.
  • All other IOs are unconnected.
  • There is no reset or configuration, as the JTAG interface handles that.
• For every type of FPGA (for every JTAG IDCODE) the following information is specified and does not change for different implemetations:
  • Voltages and IO standards on the banks
  • Location and frequency of the clock input
  • If we decide to use it: location of the IRQ

This way, the software can talk to every bit of hardware! The initialisation code is identical for all devices. The commands for communication are device specific, but the software can detect what to do via the IDCODE. Firmware for one IDCODE is guaranteed to work independent of backplane or standalone board.

Comments?

For overall system control I'd stay with a standard PC for now. It's cheap and easy to work with. It takes some complexity out of the design, so there's less things that could go wrong.

SO i think we should start to break down this idea into two Basic concepts:

Backplane based solution

- The motherboard is used for housing the daughtercards, routing the power supply and bundeling the Bus system.

- The slots are based on a widely avaidable Pc standert  eg DDR slots ( to be determined) and houses between 6 and 20 such slots ( to be determined)

Power supply

- Standart ATX Pc power supply for global distribution of eg 12 V, 5 V,  3.3 V rails (to be determined) and molex power adapters for more power intensive cards.

- The nessesary current and voltage for each daughterboard  components and FPGA(s) is indiviually produced on the daughtercard out of the rails and molex adapters supplied by the Motherboard.

IO and control

- minimum one EE prom for communication memory

- Bus system using USB,  JTAG, and simmilar ( to be determined)

- ARM CPU for Mainboard control ( to be determined)



Standalone card based solution


Power supply

- similar to the daughtercard solution while just using molex power supply for individual current generation

  [ needs further detailation]

IO and controll

- USB Bus for individual cards (to be determined)

  [needs further detailations]


Global basics

- Both board designs use an FPGA chip wich can process at least one full SHA-256 cycle wich was found to be more effective than a pipelined version.

-> This would result in one of the following : Altera EP4CE75 at minimum  or an EP4SE530 for a upscaled version.
-> For Xilinx  XC6SLX150   or maybe XC6SLX100
                                                                         (to be determined)


According to my knowledge  Xilinx chips have been found to be less performant an harder to programm,
but i think this should be evaluated by those with more experience with the current FPGA series.             (to be disscused)   



So i ask everybody of you who wishes to take part in the actual development to help plot out the individual specification of each of the both solution , so we may decide on one approach as soon as  possible.

Thanks a lot  for your work
 

If you want to go the GPU way, ask a contrac manufacturer or even a card manufacturer for a speciac OEM Version with the graphics port and the memories left unpopulated. I think that will be a much quicker solution if you can afford buying in 1k quantities or so ...

Having your own PCB artwork could save you some $, but with a board that can hold 6 or even 8 GPU chips would be far more expensive as you would also need a proper cooling solution which will have to be  water cooling or even a more advanced scheme.

The point is the low energy consumption. GPUs will not be used in some months because of their high wattage and then we need to be there with our low-energy-boards. Although I'm a bit pessimistic right now because somebody could have non-free very easily pluggable asic system right then and straight away fill a server farm with thousands of these boards.

Would "our" modular system capable of taking asic boards if available? there was a thread right there about someone "founding asics for the community" but i guess this was a troll or got selfish or busy or whatever http://forum.bitcoin.org/index.php?topic=14910.80

The future of bitcoin is asic i believe. asic or death

Good luck reverse engineering AMD's GPU drivers and reimplementing them.

No matter how you try to do it, having a central power supply on the backplane will just not work. An XC6SLX150 will use like 10 amps, and the maximum allowable voltage drop is 50 millivolts, that would mean 5 milliohms of the full path from the voltage regulator to the FPGA. This is hard enough to achieve without a connector in between. If there are multiple FPGAs on a card it gets even worse. There's just no way around point-of-load PSUs here.

So there are basically two options:

Backplane-based:
- I'd favor using a connector that's inexpensive and easy to get, but that doesn't let you plug regular PC stuff into it.
- If we go the backplane route, we should go for a really universal interface, and not something minimal. So I'd say 5V + 12V + I2C + JTAG seems to be the ideal combination. And each card should have a small I2C eeprom (like those SPD things on RAM DIMMs) that tells the backplane what it is and how to talk to it. Oh, and there should be an IRQ lane from the FPGAs (or whatever) to the backplane.
- The backplane should be able to supply at least 500mA per board on the 5V rail and 2-5A per board on the 12V rail. What about using a regular ATX 2.0 power supply, with a 24pin connector on the backplane, and possibly the 8pin CPU connector on the other end, for some more cards? Really power-hungry cards can use the standard PCIe power connectors to allow for more FPGAs.
- Maximum number of chips per card: 127 (JTAG scan chain length might further limit this)
- I'd think 8 to 16 slots on the backplane might be a good idea.
- Does the cost of the backplane CPU really matter? I'd rather go with some semi-decent ARM and some flash and RAM than a crappy PIC, possibly running linux.

Standalone cards:
- USB seems to be the way to go.
- Standard 4pin molex connectors for power supply, and PCIe power connectors for the bigger ones?
- Mechanically stackable boards
- Who wants to build the 127 port USB hub?

Maybe look at this mining problem another way.
The FPGAs don't really get such high hash rates and cost too much.

What about making a main board that can hold 4-8 GPU cards and has a FPGA as a PCI Express master controller. The FPGA would pass data on this bus to the GPU card selected. I don't know enough about the PCI Express protocol to say if this is doable but it seems like this application doesn't require high data rates between CPU and GPU. If that is the case then maybe you can make a USB device, as discussed here, that tells an FPGA to pass data to card slot N on the bus.

I think the current FPGAs have the signal ability to do PCIe control but I'm not sure. There may even be an open core design for PCIe controller.

You could have several of these hanging off a USB hub, all controlled by the python script. This might work and would be a lot less design work (maybe). Or is that just crazy?

One PC with 8 USB hubs each one controlling an 8 board GPU stack. It may even be possible with ribbon cable and connectors tied to the controller board so as not to require a huge PCB.

Hmmm. Gets you thinking.

You can actually reflow at home in a small toaster oven. You wouldn't want to manufacture that way for resale though. I'm sure it's a bit hit and miss and you would need to have the process down before you tried a costly chip. I've also seen the local mobile phone repair guys do BGA chips with their SMD blowers. I live in Bangkok where they do this kind of work in stalls here and there in malls and by the roadside.

I don't think the cost of having a local regulator/filtering on each board is too prohibitive and I would think that having a few supplies lines on the connector would be good. Maybe 1.2 and 2.5 available and a 5V line to allow a board to have it's own if it's needs 3.3 or 5V.

The question of connectors that can handle that is another issue. I think smaller connectors are likely to be limited to < 1A per pin and so you are already talking 4 x 2.5V pins per FPGA and so multi-chip cards get a bit crazy pin wise, and then multi-cards even more so. If you then look at a proper power connector for each card then the data connectors become smaller and simpler, maybe even a daisy chain ribbon cable much like an IDE (but much less pins).

My initial idea was to limit the main board to about 4 slots. Having more than that is challenging power wise.

despite the fact that you have already found an example you could look at opencores. There ist a advanced debug system that is said to work with ALTERA, XILINX and ? with the Vendor supplied  JTAG primitives. Maybe this helps you deisgning the 'Host' software with portability in mind. I did not test this solution since i am trying to burn the bitstream into the config proms of my board an doing the communication via serial lines. My CycloneIII starter board has only  JTAG (and a homebrew unfriendly HSMC connector) and this requires my development PC to constantly polling the jtag connection.

Soldering with a pizza oven was a joke, despite the fact that with a proper setup it should viable way. I have a university lab here and there is at least one guy in town that had an offer to rent his reflow oven for EUR 50 a week.

Next time I promise to search before I post:
http://groups.google.com/group/alt.sources/msg/e04b5117d101a5a7?

Ok, I just found the BSCAN_SPARTAN6 primitive. Still not clear on how to interface it, though. But it answers the question: one can use JTAG in a motherboard and daughterboard design. Just need some DIP-switches to short TDI to TDO for unpopulated slots.

I am more and more thinking about dropping the SPI completely even for the dedicated board in favour of JTAG. But does anyone have an example of how to use the primitive in a design?

Basically, this is a set of shift registers: at the latest when SS# is asserted (going low), the slaves put whatever they want to send into the register. Then the master clocks the register through, sending fresh instructions to the slaves. When SS# is deasserted (going high) then the slaves read that data out of their register. This of corse requires  NOP command if the master wants to address only one slave, the others need to be send this NOP. As for JTAG: it can be used, but is it accessible from the design after configuration is finished? That would be sweet of course! JTAG is slower than SPI at the same clock speed because it has to do more register adressing. But the overhead may be worth it in the motherboard approach to have a universal bus. For the dedicated board with its own interface chips, there is not much difference between JTAG and SPi except for speed and complexity of protocol.


I said 4A, which I got from the National Semiconductor Webbench suggestion. This is not the output of the Power Report of ISE. As for MHash/J: no clue.


I think the savings for the PCB manufacturing are minimal, if they are there at all: after all, you still have to pay for the extra area. It all comes down to the number of board sizes and the number of boards per size.

Pizza oven? This is a 22x22 BGA! Can you really reflow something like this at home? I was expecting to have to have this manufactured.

While i don't think that there would be a shortage of IO-Pins on FPGA i want to point out that the daisy chained configuration does only apply to the very special case of using as an Output Register.  I even know if that ist really SPI or just a bunch of serial shift registers. What if a slve  is supposed to output a byte, that will se next byte after the byte be:  the output byte or the just the byte received? There is, of course a protocol for using a 'unlimited' number of chips with just a constant number of 4 lines. Did you ever hear of those JTAG

So JTAG for uploading the bitstream would be fine, this would work out for ALTERA, XILINX. LATTICE even in mixed configurations by using the SVF Format with a jam player.

One note to the proposed device configuration, the 3A @ 1,2 could be a little short, as this is only 3,6 Watt and the fact that the fpgaminers design just needs 4,4W.  Maybe Xilinx Chips have a better MHash/J ratio but there seems to be a pipelined design in the queue with higher clocks that will need more power.

Multiple FPGA  per board sounds good, but i guess where one just buys one FPGA for testing the next one buys a full tray Of course there is the possibility of partially populated boards but assembly with many different configurations will result in higher setup cost.  I at least can resist the temptation to toast 500 Euro in a pizza oven.

So I am hearing a vote for going back to FT2232D, drop JTAG and using the MPSS to do the SPI communication. Works for me, though I am not sure I will do it in this version of the prototyping stage. Maybe for the next board...


Right, the Xilinx Spartan 6 can work with 1.2V and 2.5V (either a single 2.5V or two, one with additional filters to get rid of spikes). Faster FPGAs may have even lower voltages. As I just wrote in answer to O_Shovah, it depends a lot if one is willing to build motherboards for one specific set of voltages. If yes, then many FPGA boards can share these supplies. If no, then each daughter board needs to contain dedicated power supplies. Then the overhead for adding an FT2232(D/H) to each daughter board and doing away with the motherboard completely seems minimal to me.

So it comes down to these options (from my point of view):

• Build motherboards that are specific to one set of supply voltages and then accept FPGA daughterboards.
• Build boards that contain everything (communication, power, FPGA), e.g. 1xFPGA for entry level and 8xFPGA for later upgrades.

If one goes with the motherboard approach, then the number of FPGA daughterboards that fit is comparable or smaller to the largest number of FPGAs that one could put on one board. It is great for slow upgrading (if boards are available), but it is more expensive and bulkier than having a dedicated board of the same number of FPGAs.

These things are not necessarily mutually exclusive: the schematic for a daughter board and the routed layout around the FPGA can relatively easily be incorporated into a design for an all-inclusive board that contains more FPGAs. So the motherboard for entry level miners and later one PCB that contains everything for people wanting to upgrade in larger steps.

I don't know Altera, but you can combine Xilinx devices of different generations into one configuration chain. If you can combine devices by different manufacturers into one chain is more than I can guess, though. Basically, you should be able to do it with JTAG, but that seems more cumbersome to me than a different interface (personal preference).


I used that chip because I used Xilinx in the past and because the XC6SLX75 is the largest chip the free ISE WebPack lets you work with. It is pin-compatible to larger chips, though. So once someone has a suitable binary, a XC6SLX150 can be used (€123.34 at avnet for the commercial version in the FGG484 package at maximum speed grade). But I agree: there are many different possibilities for which FPGA to choose.


Yes, that is the idea. You will need to invest in USB hubs if you have multiple cards, but that is not that expensive (e.g. all12013: 13 ports for 20EUR).

Framed in different words: I chose a modular approach where the "motherboard" consists of a normal PC or netbook, a large 12V power supply and a USB hub (all off-the-shelf), and the "bus" consists of a single supply voltage line and USB. My "daughter boards" can then be anything (Xilinx, Altera, whatnot; 1 FPGA, 10 FPGA on one board, ...) and it mixes and works (if it works at all, that is ). But I need one power supply solution on each board.

Your approach has merit if all boards use similar supply voltages. Then your daughter boards need no dedicated supply at all, its all on the motherboard. This can work if one makes motherboards for specific FPGAs (e.g. a 1.2V + 1.8V + 2.5V motherboard). This allows the miner to really add one FPGA at a time with minimal extra cost. Just pay attention to the voltage drop across the connector between the boards: 1.2V at up to 4A requires a very low resistance.

The small granularity certainly is important to people starting at mining. But it also assumes that boards are available somewhere at a fixed (low) price in small quantities. Forget about prototyping for now, let's talk about later. Unless a company steps in, we will have to pool together board orders to get reasonable prices. These orders will come in waves, so a budding miner may have to wait some time to get a new daughter board with one more FPGA. That's why I would like to keep the granularity a bit larger (buy less often, but more FPGAs in one go).

As an example: this is for an old design that had nothing to do with computation at all, but the price breaks can be illustrative. Stated is the price per board for different total produced boards in one run.

Total boards	Relative price per board
20	100%
25	91%
40	82%
50	78%

So buying in bulk (finding more people to divide the produced boards between) is probably unavoidable.

If SPI is adequate for both configuration and data traffic then I'm happy with that alone. No need for JTAG except as an option perhaps. The important thing is to come up with a bus standard and protocol (for job data, because the config protocol is fpga defined) and choose a physical connector.

Then people can make either plugin modules, controller boards or both and take whatever approach works for them, knowing that later they will have access to more than just their own work.

As for power I think having a look at what power the current FPGAs need is a start. I thought mostly they didn't need anything over 2.5V but I guess some older ones needed 3.3V. I could see making a main board where some power regulators were optional so that people could choose to add them in the case they need it for their design.

I used SPI because it can also be used to upload the bitstream (JTAG is not actually needed in this case). As for select lines: you can keep using only one even if you have multiple chips: daisy-chain the different devices. From the wikipedia article http://en.wikipedia.org/wiki/Serial_Peripheral_Interface_Bus#Daisy_chain_SPI_configuration:




Good point: the normal FT2232 has a lot more support by existing upload programs. I choose the FT2232H because it has two MPSS units: one for JTAG and one for SPI. The prices for both are nearly identical on Digikey, but you pay extra for the power supply the FT2232H needs. Maybe I will renege this decision and go back to the FT2232D, if I decide no more JTAG support is needed.

There are no bad questions    

So let me explain my point of view:
One central asumption of my idea is that the upload routine of both big companys Xilinx an Altera haven't changed during the last few generations, so i hope them to go on this way.
Further the main board would supply multiple rails of 5 V 3.3 V 1.5 V  and others if needed. During the last generations of FPGA's the voltages supplied were reduced step by step so in future it should just be nessesary to break this further down.

In the end the the Motherboard was just meant to provide the Lanes for basic power supply and hardware interface and storage of the daughtercards.

I have to admit i hadn't checked on all prices of possible IO solutions and Power supplies so you may have a point regarding your parts argumentation.
  

Basically i would have used an Altera chip with minimum 80k cells to unroll one full sha-256 cycle but thats a personal perferation and not fixed
http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=544-2561-ND

I think this Cyclone 3 would be a good compromise between performance an costs, but i'm still waiting for confirmation if this FPGA would be suitible for the current FPGA Miner software

As this chip is around 175 Eur and therefore a lot more expensive than your choice the costs for the power supply would become less relevant.
But i think this chip would be a minimum as pipelined Miner designs for less cells  have proofen to be a lot less effective.

So the approach would be to place just one FPGA per daughtercard together with its power supply and use the motherboard just für holding and basic IO and Power supply.
Given the partcosts you quoted we might end up somewere around 220 Eur for one card wich might seem much but is an improvement in comparison to the current developlers boards.
The costs for the motherboard would be nearly irrelevant in comparison as it would house no essential components.



I like the idea of the USB device. But maybe i've missunderstood, would it be possible to use an chip for each board to provide it USB capabilities in order to comunicate with a standart Pc USB port?
With this way the system should be abeled to use different FPGA's and later even ASIC's.

Thanks a lot for your project files i just had a brief look but will go through them within the next week.

As i see you use EAGLE so maybe i will come back at you with my own idea for layouts.

I would be glad to have you helping to further pursue this project and help me discuss all nessecary components.

regards

Jens

I2C for FPGA <-> host communications. SPI would require far too many chip select lines..

FT2232 would be great, USB + JTAG. It's what I'm looking at if I decide to make a board.