Topic: BitFury Design, Licensing, Mass production - page 3. (Read 13028 times)

Unloading bitstream is impossible....

About remote activation - it is pretty possible thing. There's actually several possibilities - one possibility that bitstream reads Device DNA code (it's serial number), encrypts and transmit it to server, server sends activation code, bitstream matches it - and if Device is in white list proceeds with execution. But this is possibly moderate protection, as bitstream could be reverse-engineered and wires going to Device DNA could be replaced with connection to logic fabrics.

More complex protection would require 2-stage loading - first-stage bitstream that is auto-generated, and mainly generates "special" kinds of public-private key pairs with embedded proof of work (like calculating it for 4 seconds in true Spartan). It would not work in simulators, and it would be difficult to reverse-engineer as code for calculation is generated. Then comes DPR (dynamic partial reconfiguration) to load first protected bitstream.

Also this bitstream could be "rented" - say working with nTime less than specified.

But pity moment about this I wrote you is that it requires quite high effort to be done, compared to fusing AES key.

It is bad that chip manufacturer implemented AES only, because if they would implement in silicon some public-private key infrastructure with Xilinx certificate - it would be much simpler.

I would just take public key + xilinx signature for public key that you supplied, check against xilinx certificate and done, I can deliver bitstream. This would work like charm, but there's no such feature.

Even with e-fuse it is less protection compared to SRAM + battery for AES key.

Just compare attack cost... You pay $20, and get thing that costs say $50k+ you could pay additionally then $20k just to break it. I understand that you are interested, believe me that when there will be no heat about Spartan - I will disclose bitstream + disclose ways how it was achieved, because it can be valuable for learning.

EDIT: https://bitcointalksearch.org/topic/m.918095 - claims that bitstream format is available under NDA... So - as long as you can reverse-engineer bitstream and patch it - protection won't work. So it would be just lost time. Protection should be assisted with hardware.

Apparently they had a structured ASIC in the works, but no one has seen one yet, and they've been pretty quiet.

No clue, never paid attention to Largecoin. Weren't they the ASIC company that has gone nowhere?

Wasn't this what largecoin was attempting?

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

As explained on their website, they were expecting even higher hash rates. The ISE tool even led them to believe higher hash rates are achievable, but in practice the error rate was too high and they had to downclock to 240 MHz.

Wow, I can't believe I just said "downclock to 240 MHz".

Looks like it is possible to split into smaller rigs.

Explain your marketing:
BFL couldn't sell enough units at $30k so they came up with one at $15k, and you jump in straight with $90k?

I am prototyping my own design as we speak, and thinking of having 100 boards (or 127 boards, because the assembly quote was for 127 - don't ask) assembled in China, at a relatively unknown assembly house, and then shipped to me.

I do not know whether this unknown assembly house would have the technical know-how to burn in an AES key into the FPGAs, in fact right now I have no plans to have a JTAG connector populated. And I do not know whether this unknown-to-you assembly house in the middle of China (literally) would have your "trust".

But, since I'm in NorCal and you seem to be in Los Angeles, maybe I can just drive down to you one weekend, you program in your AES key into all 100 (or 127) boards, and I hand over a check for 2 grand (or 2.5 grand). Or cash, if desired. Maybe we can work something out.

Right now, the current revision of my board has a relatively weak DC-DC converter chip on it (which I actually bought from someone on this forum, but I don't blame him - who could have foreseen bitstreams so fast) and I need to redesign the DC-DC converter to a 12 amp one. While I redesign it, could you be so kind and disclose the PINOUT that you have, i.e. the .ucf file. In other words, I'm redesigning the board ANYWAY, might as well redesign it to conform to your exact pinout...

Well - sending boards back and forth is bad case, and it will not work that way. Better case to program it in place where they are assembled. But that requires high level of trust to one who is doing such programming. Also it is cheaper to assemble in higher volumes. So if there will be 1, 2 or 3 points of PCB assembly with nice flows of chips - it is not problem to get trusted person, who will turn board on, program AES and done. I am ready to solve it, so it would be without any delay. But not for 20 points of assembly (soldering of PCB, not building complete product) with 10 chips per month each.
Anyway - to get good prices you have to make volumes, this is why we started with racks, and not with small units - it was much cheaper that way.

EDITED: Or - how else could it be protected ? Because selling it for $20 is like uploading it to ftp. Say you will not pirate it, but one of 100 users would certainly do, and they'll even think that they did it right way.

EDITED: Or another possibility was offered - that if chips are supplied to different board vendors using same channel, then we could program chips at some point during their initial delivery. Probably this could be best option for smaller purchases, but it depends.

This topic is more intended for developers of FPGA boards. I suppose they would join soon. For better understanding I will explain whole cycle of these devices:

Xilinx (www.xilinx.com) produces programmable logic chips - XC6SLX150 - the key is that they are "programmable" bitstream is configuration that configures wires and logic cells inside of chip, so it is not like usual program of GPU miner etc. There is no such thing like CPU in FPGA by default, although FPGA designer can implement many CPUs. Depending on effort - there could be many different implementations of same algorithm within FPGA. Most difficult part is LAYOUT within chip to get high performances, as unlike programming there you have to deal with wire and logic latency, design synchronization and etc. So - to get nice product you should definitely have best bitstream available for FPGA. - This point is solved - BITSTREAM IS QUITE COMPETETIVE .

Then comes in play PCB (printed circuit board) that provides connectivity and power for FPGA chips, it has many cheaper components. But their sole role to give jobs to FPGA, download results and make FPGA and its user happy. PCB itself is manufactured in multiple specialized factories. Also there's multiple assembly plants around globe who would gladly take PCB without components and solder them. At this point you'll have working PCB.

Then comes assembly of PCBs into devices - basically putting boards into chassis, adding PSU, configuring.

Then it comes to delivery to end-user. With also servicing it within warranty period, if required.

So with the rest of points - making PCB, assembling them - I am looking for partners that would deliver these devices best to their customers.

So if you are in - and would like to buy such devices - you can (a) ask someone nearby you who already have Spartan-6 solution - that you are aware and would like to buy NNN {put your numbers here } devices from them; (b) if there's nobody nearby, you could contact shalab.si for example soon and import modules from Slovenia etc.

Luckily, with help of this topic, I suppose to find those who are interesting in making and selling this stuff. So you could go locally and just order it. To make things more interesting for developers, I would like to ask you tell numbers - how many Mh/s you want. Target price I think is $0.60 for Mh/s without VAT (I suppose that everyone knows well what is Mh/s and possibly only problems with these fancy bitstream Mhz numbers).

Ah, I see.  I know what firmware is, but I've never heard it called a bitstream before.

My 2 cents:
The pricing is pretty much OK, but the AES key programming could turn into a logistical nightmare.
In a few days, EldenTyrell will reveal his offer and if it is competitively priced, but does not involve sending boards back and
forth for AES key implantation, it'll probably be the offer I'll pursue.

Also known as firmware. It is quite literally a stream of compiled bits that the fpga can execute.

I'm interested, but I'm not understanding something.  What is the "bitstream" in this context, and what is it for?

Very interesting. Do you have any plans to license out just the bitstream?

Wow! This is an amazing development.

Dear BitCoiners, BitCoin talk users!

First of all I would like to thank to other developers of Spartan-6 based works, especially those who contacted me soon - shalab.si, fpgamining.com team, Greg and all those who understand how hard that work is. Also I would like to thank all those who supported me during development of these mining racks and invested money, so I haven't went into "preorder"
fall like offering more Mh/s power than hardware actually could deliver.

I am actually in love with bitcoins, because this is exactly the thing I lacked in 2006 year, and tried hard to invent... Unfortunately I were not smart enough to find out proof-of-work based blockchain. And this solution actually useful even in wider range of applications than money transfer themselves, as it can limit or remove human factor in distributed database consistency. It is interesting how this will evolve.

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs, so it is $0,728 per Mh/s. We've spent on boards production about $0,50 per Mh/s and on installation about $0,12 per Mh/s, totalling $0,62 per Mh/s. Count bitstream offer (it will be later in this topic) - it would be $0,69 per Mh/s. SO IT IS $0,03 LESS THAN THAN BFL Mini-Rig and it really works for almost 2 month, so we are sure now that it does not fry chips etc. We have not even managed to fry chip, when fed it with 1.5 V VCCINT core voltage. But, price would be definately higher (near estimated production cost), if counted other issues, like climate control, getting specialized area for them, getting power (as it consumes 20 kW together with chiller). All of these costs were already paid, because it is installed in basement of household with already existing heat pump.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips. Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

motherboard 4-layer $111 ($24,5 PCB and $8,53 soldering), daugthercard 6-layer $18.39 ($2,2 PCB and $5,5 soldering). these are costs with components with VAT and connectors. If connectors will be removed and VAT will be removed, then cost would be like $77,7 for motherboard for 6 chips and daugthercard $12,83 . Totalling $154,68 per board .

So server with 14-15 boards could cost in range $8'445 to $11'470. More likely that actual manufacturing cost plus work will be somewhere about $10'000 to $10'500. Then, say it would have 90 chips. I would point out how important Mh/s  are:

BFL Mining rig quite comparative product promised to be sold at $15'295 for 25,2 Gh/s - that is $0.607 per Mh/s

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

So at 200 Mh/s - there's almost no difference between bulk order and product price. At 250 Mh/s there's $3'143 income.
At 300 Mh/s there's $5'865 income. And at 325 Mh/s there's $7'222 income. Additional 50 Mh/s per chip gives 86%  income increase if prices are set at BFL levels of $0.607 per Mh/s. Additional 75 Mh/s per chips gives +129% income.

Calculating these costs and also costs handling sales, manufacturing etc. lead me to following licensing targets for about 1000 chips per month installed:

$20 - $25 per chip (depending on chip price and costs incurred by AES-key programming) for current bitstream and
$5 - $7.5 for future upgrade (separately), which can be opted when such update actually done.

From our side most important point is bitstream protection. This incurs costs of AES-key programming by moving trusted person to assembly plant from time-to time, powering on boards and fusing chips with AES-key. For simplicity it would be great to have 6 spartans on single board tied on JTAG line via buffers. Then encrypted bitstream could be available without any additional protection to it. Of course if quantities will be small and location distant, it would be difficult to execute programming. We are already planning to program chips with AES key in Hong-Kong, and we have good access to EU, because we are located in Ukraine. Existing boards could be upgraded as well.

Also - why I am insisting that 4U design one of the best sizes:
1) it would consume 1.3 - 1.5 kW per 4U, and that fits into envelope 350W per 1U heat production, which is not difficult
to implement in either datacenter or homebrew setup;
2) Chassis itself could be sold with minimal margin, so people could build mining power within nice chassis step by step;
3) When local price to electricity would be unaffordable, people can send what they built to Iceland for $0.04 per kw*h
special setup, so compatibility with datacenters is nice feature;
4) It gives $1'800-$1'900 "entry-level" price and board-by-board upgrade possibility;
5) "special single board" can be available as well - with USB, and it would be nice option that it can be put into server,
basically I expect to lower costs to have boards with pads for USB-related components, just they will not be soldered for
items that will be put into servers;
6) also it would be nice to have some DRAM on board for other purposes, but again - this would be not soldered at all, but later, if boards will be re-used for other tasks such DRAM capability and at least 1G ethernet external connectivity would make difference for long-term product life. At stage of PCB design it adds only NRE costs of designing PCB itself, and no cost at manufacturing.

Thinking where equipment will be placed - in small quantities - probably in homes, but when it will get larger - I have already found location and discussed this issue with Andreas Fink (Skype: andreasfink ) from datacell.com

[5/22/2012 10:24:36 AM] Andreas Fink: Our datacenter is designed for 20kW cooling per rack
[5/22/2012 10:24:37 AM] bitfury.org: but if air inlet drops 10 degrees Celsius, performance can be increased. and optimal temperature is about 0 degrees Celsius if we use industrial chips. that gives 20% performance increase.
[5/22/2012 10:24:44 AM] Andreas Fink: we use hot aisle concept
[5/22/2012 10:25:17 AM] bitfury.org: buyers of these devices actually concentrate on pure performance and cost of electricity, cooling etc. many of them installing such installations right at home, because it is cheaper.
[5/22/2012 10:25:27 AM] bitfury.org: you can google for "bitcoin mining"
[5/22/2012 10:25:40 AM] Andreas Fink: Well we operate our datacenter in iceland. energy is cheap there and coolign is easy.
[5/22/2012 10:27:05 AM] Andreas Fink: We could build a custom datacenter just for that if there's enough demand.
[5/22/2012 10:28:05 AM] Andreas Fink: what is the potential of your installation if it produces bitcoins?
[5/22/2012 10:29:38 AM] bitfury.org: very difficult to say, because we have just started to announce our solution.
[5/22/2012 10:29:54 AM] Andreas Fink: a rough estimate to get an idea.
[5/22/2012 10:29:54 AM] bitfury.org: current bitcoin network hashpower is 10'000 Gh/s, single bitfury produces 110 Gh/s
[5/22/2012 10:30:03 AM] bitfury.org: so estimation is about 30-40 racks
[5/22/2012 10:30:28 AM] bitfury.org: that's potential for FPGA bitcoin mining.
...
[5/22/2012 11:37:42 AM] Andreas Fink: the electricity company charges like 3-4 eurocents per kWh
...

It depends - if we manage to start with 4U working with them and without using diesel/ups-backed power which is  expensive and climate control equipment - it would be cheapest collocation for mining available in world.

[5/22/2012 11:02:04 AM] Andreas Fink: http://www.sgi.com/products/data_center/ice_cube_air/
[5/22/2012 11:02:17 AM] Andreas Fink: put them in such a shelter, we provide power and internet and put it into the green field

I suppose that shelter should be designed as well as SGI could be expensive, but generally you've got the point. And that is why I am sticking with rackmount design. It will be big pain to move devices that are non-compliant with datacenters, putting Mini-Rig there for example. I suppose this is the moment, when buyers of BFL should start thinking twice, what they would do with their rigs, when income will be about the same as consumed power. For myself that problem is solved - mining equipment is integral part of house heating now. For cold countries this can be as well solution - integrating house / DHW heating with mining. Today you will probably say that it is absolutely crazy, but 5 years in the future things would change, and those who make useful use of excessive heat will save on electricity bills, while those who will pay additionally for air conditioning would loose, as they could not compete.

So finally, I would welcome everyone interested to join efforts. First - design chassis and PCB with more-or-less interexchangable parts. Getting for example with-USB and without-USB version, with-flash and without-flash version, etc. so actual board can be customized by needs of our partner who provide product, but in general they all are near the same, and could be fitted inside 4U chassis. This way we can save on designing this thing, build product in which our customers will be confident, that even if one supplier stops selling boards, he can possibly with higher costs order elsewhere, but still finish his 4U box. More important is different metal pieces like holders of boards, etc. Which would depend on supplier of these chassis solution.

I am going to provide soon draft of 4U box, as I see it in 3D with airflow calculations. For those who asked about current interface, I am attaching right into this message .vhd source code of bitstream side, current .ucf file (but please, it can change - we used 1 mm FGG484C, and it could be possibly better to use CSG 0.8 mm steps), only important point that communication enters BOTTOM of chip. And communication part in dsPIC33F firmware. As you probably would see - same communication can be performed over LPT port without bothering with production of additional conversion boards.

Here are links to download:
http://www.bitfury.org/bfdetails/sha_top.ucf
http://www.bitfury.org/bfdetails/sha_top.vhd
http://www.bitfury.org/bfdetails/shaspi.vhd
http://www.bitfury.org/bfdetails/jobs.c

Meanwhile - we used FGG484 because we though to put more bypass capacitors below chip for overclocking purpose, as with CSG there
will be difficult to place such numbers of capacitors. However we've aimed for 320 Mhz that makes too much errors, with 240 Mhz
it would be fine to place less capacitance, and also get industrial CSG chip, as this would remove any possible overheating problems.
CSG chips have about 1.5 times (!) lower junction-case thermal resistance according to datasheet. However this would be great to be
confirmed by someone who used CSG chips - I saw ZTEX used them.

So - how we shall proceed ? I suppose best way will be to:
1) Write to this topic, if you are in (possibly after getting in touch in skype: bitfury.org );
2) Writing estimated quantities per month, assembly plant location (place where chips can be turned on);
3) Writing your final destination where you are going to ship these boards for further packaging;
4) Writing number of chips you already use - because if there's many already - then possibly re-work on IR station could cost
   significantly less than buying new chips;
5) Writing your requirements to board - specifying way of Spartan programming, currently used interfaces (i.e. USB with USB chip name),
   so we can get common specification for board interfaces and chips that shall be on board. Probably for overclockers it would be nice
   to have ability of voltage setting.

Then next iteration will be to approve specification and doing PCB schematic design, and then doing layout, and
doing similar way with chassis. Hopefully there's people who like working 24 hours + night

Kind regards,
Mr. V // Skype: BitFury.org

PS. Special note for those who know me personally. I dislike indexing of personal information via google, etc, so please, do not write here - "oh I know this guy"... Write it in skype in personal communication (it will be visible at least for
special services and not to general johny hacker who mines information and would like to get some cash), not on public forums  or public web pages. And I discourage everyone from doing so with their information. Imagine that it is possible to enter your name to google not only to your friends and partners, but also to your foes, those who you even do not know...

PPS. I've also uploaded stats file http://www.bitfury.org/bfdetails/bitfury_status.txt with current number of working cores, you may see that some cores are not working, error rates.... There were no special handling with them - we've just put cores "as is" into motherboards, and there are no visible defects. We've not bothered replacing these cores. Just to those who say that it is not real thing

EDITED - also forgot part of conversation with Andreas Fink about Iceland VAT there is 25.5% but for collocated servers it is returned. So it is like having 0% VAT compared to 18-20% VAT in EU countries.