Topic: Modular FPGA Miner Hardware Design Development - page 30. (Read 119320 times)

There are suitable multiplexers. An example of one input to 8 outputs is the NXP PCA9547PW, which supports up to 400kHz. Digikey 568-3381-5-ND for 1.90EUR.

This depends on the number of FPGAs to put on one board. It will not scale linearly, though!


How about this: someone first draws the thing (even with small shapes on a vector-graphics program is fine at this point). Then decide how many FPGAs to put on one board based on size relative to a DIMM slot. Think about FPGA package size: I drew a large package for my prototype because I was not sure about prototype cost for the smaller layout.

All periperals that I can think of need the FPGA supply voltages, not 5V. The oscillator (say, 100MHz) needs to run at V_io. The level shifters need two supply voltages: V_io on the FPGA side and whichever is the signalling standard on the other side. For the mode where the card is slotted into the backplane, the backplane could provide this (say, 3.3V). If the USB chip is used in standalone configuration, this would be USB supplied (either 5V or 3.3V).

I personally want to add that i prefere the first option as far as i understood them both.

Most PSU's may output more than their nominal power on the 12 V rail so leaving the 5 V rail out wouldn't result in a loss of power.

I case the DIMM is sloted in the DIMM socket while operating on the motherboard it processes the voltage for the FPGA core from 12 V to quarantee stability anyway.
the 5 V rail of the DIMM socket is used to supply the FPGA peripherals in this case.

And in case the DIMM is used as a single miner card it has no 5 V supplied via the DIMM rails as it is just supplied via the Molex or the barrel connector so it has to be generated from the 12-16 V anyway. Maybe the USB voltage could be used in this case but im not sure if this is enough.  

I doubt if this work, 2200 mm2 are 2,2x10 cm, this is a quarter of a normal DIMM and even if we could use a bigger PCB the FPGA area and the power supply are schould de clearly separated from each other and we  don't want the in rush current route under the FPGA. Of course we could place the power supply down at the connector side but then the heat sinks will add mechanical instability then ...

How about the Euro format, the format is a little largisch but you could mount heavy heat sinks on them. There are also formats other than 160x100 mm, e.g. 100x100 and 220x100, if there are also fomats defined for 1 and 2 HE, i dont know, this would be a exercise to the reader.

Hi,

as I wrote before, I don't think that supplying a high voltage to the DIMM means a hit in efficiency. So I vote for supplying only 12V to 19V to the DIMM, both when acting stand-alone as well as when being plugged into the backplane.

I dont want to rush things but i'd like to ask you to decide please.

As i see it, there are two options:

Either we use 12- eg 16 V on both the DIMM powersupply rails as also on the molex and barrel connectors and transform them to all voltages needed on the DIMM board.
This should work for the mainboard based as well the single board solution but does not use the 5 V rail provided by an ATX PSU, but might produce a loss in energy efficiency.


Or we use both a 12 V rail and an additional x Volt rail on the DIMM connections. 12 V for the FPGA supply and the x Volt rail for other parts located on the DIMM.
This should increase power efficiency and use the capabilitys of the ATX PSU to its full extend.


Please give me some kind of vote for one of the solutions or corret me if i got one of them wrong.    

Edited

Here is a table that hopefully answers the question if one should use a rail voltage in the power supply or not. I got the table by using National Semiconductor Webbench with the following parameters:

• V_in = 11V to 20V
• T_ambient = 30°C
• 4x Xilinx Spartan-6 XC6SLX150, each using these voltages / currents:
  
  • V_int = 1.2V @ 5A
  • V_aux = 2.5V @ 0.3A
  • V_io2 = 2.5V @ 1.6A
  • Nothing on the other banks

The currents were suggested by Webbench, I don't have the output of ISE power analysis. The 1.6A for V_io2 are probably strongly overestimating our actual usage, I hope V_int is not underestimating it. I didn't use 1.8V for V_io2 in order to have fewer switchers.

Webbench lets you specify if you prefer to use their simple switcher(R) modules or not (though it doesn't exclusively use those). Both settings were tried.

Use modules	Allow rail	Rail [V]	Efficiency [%]	Footprint [mm2]	Price [USD]	Part count	Used parts
N	N	-	80.53	2485	25.60	86	3x LM5117
N	Y	5V	87.52	2162	24.70	76	LM3150, 3x LM21215A-1
Y	N	-	73.86	2294	48.86	32	LMZ12008, 2x LMZ12010
Y	Y	5V	85.52	2203	34.63	68	LMZ12010, 3x LM21215A-1

It seems using a supply with a rail voltage is always cheaper, smaller and more efficient. Don't take the prices to heart, though: these are for large quantities. Assume a factor of 2x in actual prototype price!

Did I or the program make a mistake? If not, I suggest we plan on having a two-stage supply on each DIMM. Then the question is simply: does the backplane supply the rail voltage (and not use the first stage of the DIMMs power supply) or does it supply 12V?

The situation may be different with switchers from other manufacturers. Examples?

So think we may agree on the following points:

- We will develop a hybrid solution, usuable for both the Modular and the single board use

- Each board hast it own power supply via both 12 V and 5 V through a 240 pin DIMM connector. And if not connected to the mainboard or needs further power it is supplied by the 12-16 V  barrel input or the 12 V Molex 8981 wich makes nessesary for a chip capable of transforming 12-16 V to 5 V and lower Voltages needed for all components on the DIMM.

The further i surely agree that we will have to apply heatsinks to the FPGA's. But i also think that these will not need such heavy weighted parts as on graphic cards, so the DIMM slots should be capable of holding that weight.(When i look at the RAM DIMM's in my Pc i also wonder how they may hold such big heatsinks)

Edit:

I would appreciate it if we at first may decide on a more or less final concept for the power supply before disscussing other parts and then move on to the next topic  eg Bus systems ?

  

We are talking about FPGA temperature at the moment. So here a question: do we add one or more temperature sensors to the DIMMs? E.g. the Digikey 296-27725-1-ND, which has an I2C interface and costs 2.15EUR in prototype quantities.

And I just realised: the backplane needs different I2C interfaces for each DIMM, if there is an I2C bus at all! Or is there something like a breakout chip: if it's address is given on the bus master side, it forwards traffic to the second attached I2C bus, acting as a bus master there?

I was only talking about the connector, not the rest. And it seems like the decision to add power supplies to the DIMMs has been made.


I am not an electrical engineer, so I rely on what National Semiconductors Webbench tells me about the capabilities and limits of their products. I looked there because of their handy Webbench program: if someone knows what they are doing power-supply-wise, a different manufacturer may be better. But at least for National, going in two steps and having a rail voltage of 4V or 5V seems to increase efficiency by 10% to 15%. I cannot say why that is. Maybe their product portfolio is not optimised for 12V to 1.2V in one step?


Agreed, having at least some of the cheap power supplies compatible to this design seems like the thing to do.

So if the DIMMs need heatsinks, that means no tight spacing of DIMMs is possible. That means it doesn't hurt to add the power connectors to hybrid boards by default.

3 milliohms for the roundtrip? Including inner resistance of the voltage regulator, PCB traces and FPGA pins, you just can't manage to stay below 5 milliohms this way. Power supply stability is absolutely critical for those FPGA boards. Putting this on the backplane probably isn't going to be cheaper in the end. You seriously want to put a 1.2V 300A power supply on the backplane? Good luck routing the PCB. So you'll have to split the power supply anyway, and whether you do this on the card or on the backplane doesn't matter price-wise. On the power efficiency side you're definitely better off with higher voltages on the DIMM connector.

Also, why on earth would you want a two-step power supply? (12V => 5V => 1.2V)
This greatly increases the load on (and thus cost of) the 5V supply and reduces efficiency.
Either you'll get both 12V and 5V from an ATX PSU (in which case you'll want to use 12V for FPGA supply voltage generation, as it provides the higher wattage on modern PSUs and 5V for all those small FTDIs or whatever), or you'll want to do 12V => 5V and 12V => 1.2V if you want a barrel connector.

Ideally the 12V rail would actually be 10-20V, or at least 12-16V, to allow for some of those laptop power supplies to be used.

Oh, and there is definitely no way to dissipate up to 15 watts from such an XC6SLX150 without a heatsink.

Well, the often shown picture from the Copacobana  shows 6 FPGA in the FT256 BGA, this is 17mmx17mm. The successor with  Virtex4 has bigger chips and as far i remember the photos from a prototype photo showing a more than oversized dimm card.

Having no heatsink on the photos is no argument to me that no one is needed. Maybe they did produce the bitstreams without exceeding the allowed power dissipation by using a reduced clock rate rather than the theoretical limit with cooling.

Ok, matches much better with my desired standalone/hybrid FPGA boards: basically it costs very little effort to transfer a design from a DIMM to a standalone board with an USB chip. One could even build the hybrid version as a standard then (just leave the clunky power connectors unpopulated by default).


What do you need two voltages for? If you decide now on only one, you have the simplest of all designs. The only disadvantage is that you use only half of your ATX power supply... And if you go the Laptop power supply way, then you have only one voltage, anyway.

I would opt for 5V on the bus: yes, the current is higher, but how big a DIMM do you want to build that it requires 5V*45A > 200W? On the other hand, it means one less voltage supply ion the DIMMs: I used the National Semiconductor Webbench to design the power supply for my prototype and when starting from 12V, it (nearly) always goes the two-stage way. It first steps down the 12V to either 4V or 5V and in a second step it steps that further down to 1.2V, and 2.5V. So unless there is FPGAs out there that require 5V stabilised input, then I opt to remove the 12V from the bus.

For a hybrid card, this would mean it needs a 5V input or you need to add the extra step down stage which remains unused if the card is plugged into the bus.

That is important information, thank you. So for 20 pins per voltage and the the same number for GND, we could get to roughly 3 mOhm for one roundtrip and still support 20A (we need 10A for one large FPGA). When calculating 3mOhm I also took the resistance in the GND return path into account. If you have two voltages with 20 pins each, you should have 40 pins on GND.

If we think of number of available pins: 200 pins minus 10 for dedicated DIMMS minus possibly another 10 for hybrids (both rounded up), we have 180 pins. That is enough for roughly 100A of various voltages plus GND, let's put in a fudge factor and say 45A. Now let's assume that the VCCIO and VCCAUX (for Xilinx) don't use much. This leaves enough for 4 XC6SLX150 on one DIMM! With a smaller fudge factor, the 6 FPGAs visible on the copacobana photo seem possible.

So in summary, it seems possible to build DIMMs without any power supply at all. The problem is that there is not much room for putting extra voltages on the bus to accommodate different devices. How to resolve this:

• Put only a relatively high voltage (3.3V or 5V) on the bus and require the DIMMs to have their own local supplies? This is extensible but expensive.
• Design different back planes for different FPGA types, differing only on the number and type of voltages on the bus? This is cheap but locks the user into specific hardware.

If one goes with the cheaper approach (after all: how many different FPGA boards will be available in the near future and how expensive is getting a new backplane?), there might be a way to keep the extra design effort for different boards to a minimum and keep at least a modicum of compatibility:

Partition the supply pins on the bus into different rails, say 10 pin for each rail. This gives a 5A granularity at a fudge factor of 2. That gives 9 rails plus 90 pins GND. The location of these rails are specified and are identical across all FPGA types and backplane types. It is "just" a question of what voltage to put on each rail. The layout of the bus part of the baseplate can stay the same in all cases. It might even be possible to put different smaller FPGA boards (that don't use all rails) together in one backplane because the different voltages are on different rails.

I would strongly recommend that each board generates all needed currents and voltages on the DIMM itself as their quaility is essential to the boards stable operation.
The DIMM should be supplied with 12 V and 5 V rail via the DIMM connectors for lower power needs and Molex or barrel connector for higher power needs.
If we use a standart 240 Pin DIMM connector minus the pins for the Bus and comunication system there should remain enough pins to split up the current to a good level.   

This is my reference for DIMM sockets,

http://www.4uconnector.com/online/Itemadrawing.asp?fldseries=04677&seriesno_a=0702

It indicates 1A max and 30 milliohms contact resistance.

Here's the catalog page.

http://www.4uconnector.com/online/seriesphoto.asp?groupdesp=IC+%2F+RAM+Socket&itemnum=64&SeriesNum=10&GroupNo=07&sample=0

I didn't check price lately but they're quite cheap. $1-$2 if I recall from years ago.

RE: JTAG

I don't know about this "ring" thing. I recall it being just a regular bus. If it needs to pass thru a card and out then you just can't skip a slot. But it's hazy now since it's been years for me.

Edit: I just realised that the word standalone is used in two contexts here: the Ethernet capable backplane that does not need a host computer and the USB-based FPGA board that does not need a backplane. When I write below, I am talking about the FPGA boards.

Thanks for clarifying that. I think a common interface is not impossible, it is actually straight forward! If we decide on using JTAG (at, say 3.3V) as the bus interface for all boards, then it is just a question of how the JTAG signals are generated on the backplane. The first, simpler version could have a dumb USB chip and later people could design a backplane that instead (or additionally!) has some small CPU with Ethernet.

So what signals (apart from power) do we need on the bus? A summary of the current suggestions plus my thoughts about details. This is only suggestions to keep the discussion flowing and provide a central point to criticise to comment:

• JTAG (4-pin should suffice, i.e. no TRST). Connects all FPGA on the DIMM in a ring. Voltage should be something common, e.g. 3.3V
• I2C bus. Connects to all FPGA plus a small EEPROM on the DIMM. Voltage should be something common, e.g. 3.3V, pullups on the backplane (this means a bit of extra work for hybrid DIMMs: they need to enable their pullups in standalone mode).
• A board presence detection (two pins shorted). Can be used to have the backplane automatically connect TDI to TDO on unpopulated slots.

In case one wants to build hybrid DIMMs, i.e. boards that can either work standalone with a USB chip or as a DIMM on the backplane, the following signals may also be interesting. These are just extra, so dedicated DIMMs can still be mixed with hybrid boards:

• The RESET# pin for the USB chip. Pullup on the DIMM, connected to ground on the backplane
• The PWREN# pin for the on-DIMM power supplies. Details depend on the USB chip. Either connected to ground on the backplane or connected to the PWREN# signal of the backplane chip.
• There was a suggestion of adding the USB signal of the DIMM to the bus. Not sure if that is necessary, even if I don't see how it could hurt. If used, the RESET# pin may not be pulled low but instead the JTAG lines must not be used (i.e.: completely different backplane design).

A question remains on local power supply: What to provide through the bus connector and what to generate on the DIMM? I think TheSeven has a point when he is vary of sending small voltages at several amperes through these connectors. Does anyone have a resistance per pin for DIMM connectors? I looked at Digikey A97917-ND and 609-1001-ND, and they don't seem to specify it.

 Seems i've started a quite lively project


So i take it as agreed that we take this both power supply connections.The voltage supplied for the barrel connecter will still be to decide. I will insert a table in my first post with facts i asume to have been decided on.  


Basically i have a rebuild of the hardware arrangment of the copacobana setup in mind http://www.copacobana.org/photos/photo_1.jpg

As you may see they use DIMM like slots to fix the cards (wich i personally prefere) and they use both  Spartan 6 LX150 and Virtex 4-230 without any heatsink at all.
On the other hand they claim the Spartan 6 to reach 200 Mhash/s. Therefore i asume the size of the heatsink we might wish for wouldn't be an obstacle for stability.

A Fan assembley for the cards will be needed anyway. But i don't see the point in a PCI slot. Im think they got even less structural stability as they lack the fixations laps.


My original intention was to create an FPGA slave for a Pc. The backplane just used as a saving, supply an housing unit.Altough i have to admit that a standalone solution is interesting i would postpone such for later developments if it is not possible to create a common interface for both solutions. I would like to ask you to clearify if a common interface  for both solutions is possible or not, then we may decide on that point.  


One other point that has been numerously discussed is the use of an FT2232 device simmilar http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&name=768-1010-2-ND
for Bus system JTAG.
I'd like to know if we can basically decide to use this chip at all or if anybody has any reasonable alternatives ?

You may not even need the multiplexers: if the FT2232 is not needed in when such a hybrid card is plugged into the motherboard, then connect its RESET# pin to the bus: all outputs of the FT2232 are tri-stated when that pin is pulled low and you can connect to the JTAG without the FTDI chip interfering.

I am not set on an FTDI product: anything that works is good! But with a microcontroller, I would ask that one is chosen that can not be bricked by wrong programming: always leave the option of reprogramming through the USB. And what about software support (you can get FT2232D JTAG support in many different programs).

Any feature requests there are for a microcontroller over an FTDI? I heard of these two, now:

• Store additional configuration data (i.e. obviate the external I2C EEPROM for board identification)
• Cheap(er) price

Anything else come to mind?