Topic: Request for Discussion: proposal for standard modular rack miner - page 10. (Read 9671 times)

Changing board sizes greatly changes cabling requirements. If you make quarter-sized boards now you have to provide for 32 internal data and power connections. The provision for having several different boards is already allowed in that there are eight separate heatsinks. Buying 16 or 32 small boards instead of 8 big boards is about like trying to make a large-scale miner out of USB sticks. You end up with a lot of needlessly repeated parts. Small boards also limits your topology choices, which limits your efficiency overhead options. I can give you more details about this point with reference to some of our internal board design decisions if you like.

I would prefer to stick to a design based around air cooling. If you can't fit a waterblock in the same space as a two-inch-tall heatsink there's problems with your waterblock. If the rear panel is removable could provide ready access for hoses and such.

Not sure what ATX spacers would be doing anywhere?

Ok chiming in for the first time....

Sidehack, you know i love all the products you do, but in terms of heat dissipation we have our differences, so bear with me

It could be nice if the case could also fit water cooled blocks, as this is a need for most tropical countries, and would make heat management a lot better and let home miners mine without the noise they make (currently writing this with 6 S5 at my back).

Also, how about starting with the de-facto standard for boards that is the S1/S3/S5 hashboard? That would make adoption fairly easy, and the hole design on them allows for several individual boards screwed to a standard set of holes or having several different boards.... lemme explain myself with an image:



The white (and yellow) outline is the S5 board dimension, the red circles are the holes preexistent on the boards, and the green lines are the "logical" separations on the boards.

You could have a case with vertical mounting spacers, which could host 4 Small boards, 2 Mid boards or 1 Large board. The boards themselves could have a nomenclature so you can know how much you can fit of them on the case (S, M, L) and have them mounted with atx spacers.

Also, have some cable managemente in the inside, so you can have them orderly without a mess.

I can make a model if you want (i know you're more of the hands-on type, but having a model doesn't hurts anyone).

So, here's a thought.

Two boards from this machine could fit into an SP20-scale miner with a single 120mm fan and hook up to an external 650-750W PSU. Opinions?

The S4+ runs pretty close to its readline at 35-40C so you'll get better data whacking its frequency down first.

I do have access to an S4+ which could be set on fire and nobody would really complain. Maybe I'll play around with it in a few days when I'm not busy. All the Spondoolies gear is hosted.

Its easiest to show you. If you've got an S4 or S4+ around, remove one of the pair of fans and see, remove the other and see. The one with rear fans will do just about fine, the one with front fans will overheat. If you're really determined, you could try reversing the direction of an SP3X's fans and watch it overheat.

I don't understand why not a single hardware manufacturer standardizes it's PCB's so the heatsinks and housings can be re-used...

I don't plan on doing much with CAD. I prefer to actually build something, so if I test the dimensions it'll be a quick calculation on paper and then seeing how it works when I build a mockup. Hopefully I have time to do some of that in the coming weeks. I agree, it is probably going to be tight and will probably require some adjustments. You're right, maximum outer dimension height of seven inches (and not a millimeter more) must be maintained. Ideally we'd come in 1-2mm under that in case the units are shelved instead of stacked, since there probably won't be horizontal room for rolling rails.


If we want software fan speed control, which I'm not inherently opposed to, we'd need to see about writing a cgminer module that talks to a controller on the USB board. All driver code for the various boards made would have to talk to this module, which would unify all that data and send off to the controller with PWM commands. I'm not sure how hard or easy that'll be to integrate with the existing cgminer framework. Someone with more experience with the guts of it will have to weigh in.

I think the fan dimension point you're arguing with Novak is irrelevant. Removing a fan to fit in the PSUs does not apply, because the PSUs aren't in the same space as the hash boards. The hashboards take up the entire width of the bottom 3U, so you need that entire width with an unrestricted front-to-back airflow path. The PSUs are laid flat in the top 1U, which is almost entirely isolated from the bottom 3U by means of the horizontal separator panel forming a "ceiling" above the hashboards.

Additionally, the S4 reference picture is irrelevant. If we have rear pull fans, there would be no interstitial grilles. The rear would only be grille'd if the only fans were the front fans.

My office is right next to a room with about 30 various Spondoolies rigs in it (at least one of everything they made). I don't need to be told how annoying they can get.
Also, if you know me at all, you know I pretty much despise most of the engineering on SP's rack gear. This includes having a two-foot-long airpath which pretty well bakes the rearmost chips. I'm even a little worried ten inches will cause problems. But that's not really what you were talking about, I know; you mean the suction from rear fans instead of push from front fans.

If someone who knows the mechanics of air cooling can weigh in with a quantitative consideration of front push fans versus rear pull fans versus both would be nice.

Think of the SPXX (all of them ), they used the absolute perfect cooling layouts. You don't need to ram air in because it really doesn't do anything, as evidenced by swapping out SP120s on an S4's front fans not changing the temperatures pretty much at all. You then suck all the air out, in a long, smooth and continuous column. This column can be wedge shaped if your intake is larger than your exhaust and it works fine - air is a wonderful fluid.

Its all about them rear fans, and you certainly don't want grills on your exhaust (see above).


I think you're going to be super close (see above again) and it might fit on CAD but will probably be prohibitively tight to use IRL.


The fan is far, far far louder than the SP35 which can run 40% fans and its the thing you can hear through walls, floors and really gets to you. It was similar on the SP10 although that PSU was overloaded and the main fans were only 50mm. It was such a shame on the SP35 as the entire unit was quieter than an S5 if you underlocked it enough so that the PSU fans wouldn't spool up to max.


a) Put it on a test board in a test case, measure it, see what the rough delta is tracking. Use that in software. It might vary between chip type but you'll still be able to get the delta from your test boards.
b) Ignore ambient and just fan control off chip temp, which is really all you care about. Again as I said above you don't care if its a 60C ambient as long as the chips are happy.


Thermal expansion is a consideration and not an issue. Its countered by using clearance holes to mount the heatsink to the PCBs, so its sliding on the bolthead rather than loading your PCB.


In your current layout I don't think things will fit nicely (discussed under 2)), but I was actually referring to external being too large. U racks are explicit in that 4U = 4U, 1mm over and you're impinging on the next 4U of space and pretty soon screw holes don't line up. Which is why I suggested designing for external of <7" rather than =7".


Yeah, dimensions are in each individual guide. The majority are thinner than 19".

1) is worse than closed, as seen on the S4 testing. The initial cases had an open mesh next to the fans but it does performance because of this... *searches for picture*. You could actually feel this behavior with your hand and reduced the effectiveness of the rear fans significantly. Later models were



2) That's going to be a tight, tight squeeze along that side. Just drop a fan and mount the PSUs peacefully.
170-175mm external
= 166-171mm internal
Fan = 120mm
5-10mm spacing
= 36-41mm for an unmounted 44mm PSU. Putting the PSU above a huge fan like this is also a nono as it suffocates the PSU's fan which generates significantly lower static pressure. So you end up with barely any air movement there or sometimes backwards airflow. Its more of a problem when you use negative pressure flow but its still a design concern with positive pressure. Its the reason you see PSU "compartments" sectioning off the airflow - like in the S4, S4+ and SP3X.


4) What I was really trying to say is that the only temperatures you care about are your chip temps. If its 60C ambient and your chips are still fine there's no reason to stop mining.


1) Its not 15mm between fans, its 15mm total between case fan fan fan PSU PSU case. Yes fans are measured / rated across their linear dimension. 120mm fans are usually + / - 0.3mm, you do need a gap between all those components and all mounting holes will have a tolerance. You're not going to be comfortable with less than 12-15mm across all those things.


It was in reply to 2112 who was talking about thermal stress.

Dogie, regarding your various points:

1) You want negative pressure displacement, likely you'll get better performance dropping the fronts to 3x120's or 2x140s and leaving a grill. Ramming in more air than is being released = high pressure internally = lowers airflow significantly.

With 3x140mm the entire front face is pretty well covered by fans. Looking at bladed miners like the S2, you can tell exactly which boards are between fans by looking at the per-board temps. I'd like to avoid poor-coverage zones like that if possible.
The rear fans were dropped to 120mm because of height requirements for fitting in PSUs. If that ends up actually restricting the airflow, we remove them and put in a grill instead. I'm not a fan expert and these points need to be evaluated by someone who is.



2. 3x120mm = 360mm
Spacing = 15mm
PSUs    = 2x44mm = 88mm "mounted at the top rear on an internal shelf"
Material thickness = 2x2mm = 4mm
= 467mm
Your specified back side is 438mm

The hashboard volume occupies the lower 3U (approximately) of the 4U case. The top 1U is separated from them with an internal panel above the hashboards (which mostly seals them off and keeps main cooling airflow restricted to between the heatsinks and boards) and is where the PSUs, controller and all cabling live.

3) 1U = 40mm fan = annoying as hell. (See SP10 and SP3X where the PSU fan was far louder than the actual miner)

Yes, 1U fans are really annoying. If I'm thinking right, the DPS1200 fan is quieter than the thing on the SP supplies but it's been a while since I fired one up. I prefer server supplies pretty much universally to ATX and would consider installing an ATX supply to be a waste of space, a waste of cost, and really asking for failure.


4) Thermocouples are a shipping hazard as they can come loose and end up in fans and so addd another layer of inspection. They take too long to install and they're also superfluous as you should have already characterised the relationship between ambient intake and chip temps.

Yeah, temp control hadn't been thoroughly discussed hence why I made brief suggestions for both hardware and software control. This point needs to be ironed out.
Regarding characterising ambient intake versus chip temps, how do you suggest we measure ambient intake if not by the same means as measuring exhaust? Additionally, as Novak pointed out, since the board design is deliberately unspecified there is no specific requirement (at present) for this to be true.


5) Remember that the distance you actually care about is between the outer two mounting holes, not the length of the entire heatsink. The material on the ends doesn't contribute to thermal expansion mounting stress.

There are no screw patterns specified yet, but it seems likely that the outer two mounting holes will be very close to the edges of the heatsink.
I'm no expert in thermal management so the heatsink point should be evaluated by an engineer knowledgeable in that field, make sure we can safely dissipate the heat spec in the volume provided with the expected mass flow of air.

6) You've not explicitly specified how many PCBs.

"8 boards will fit widthwise across the inside of the case with a bit of room inbetween"
The maximum number of boards you could fit is 8. The minimum is one. The machine must work properly with any number of boards between 1 and 8 inclusive.


7) 4U should be smaller than 7" if you want stuff to actually fit, even if its shaving off a few mm.

The hashboards are specified as 5" tall, which is about 6mm shorter than 3U. If the upper room (cable tray, whatever you want to call it) is at most 1U high, we should come in under this. If that's not good enough (which can be determined without a lot of trouble), I'd probably convert inch-measure specs to metric with a fixed 1" = 25mm conversion which gives another 2mm height reduction to the boards. If that's still not good enough I reckon we'll have to do some tweaking.



Regarding horizontal width - have you measured many rackable miners for width, or just servers? Most miners I've seen aren't too concerned with rails, which eats about half an inch off each side.
A custom case design is exactly what we'd need anyway, I think, so we should be able to specify our dimensions any way we want that doesn't violate rack requirements of maximum height and width.

Using PCI or PCIe requires using a backplane board with sockets. That can be a significant expense. It can also be significantly more fragile and cause board-mounting reliability concerns. It also, within this design, would require a lot more disassembly to disconnect one board. How many bladed miners have used an internal cable connection to the blade instead of a backplane connector, and how many of those cable connections have been unreliable?
We're not intending to use USB jacks (and especially not garbage like USB micro) which can be pretty flakey. The spec we propose is a pin header, which a decent pin header with a decent cable can sit there for years without losing connection.
Since USB hardware isn't used, the only real USB in the spec is the protocol. I assume not many off-the-shelf SBC (or whatever little guys like the Pi are considered) have PCIe built in, which means even if we want to use PCIe hardware for connection (which I don't, from a mechanical reliability and cost standpoint) we can't use PCIe protocol without moving to either a custom-designed or significantly more expensive controller. If, then, we want to use PCIe hardware we need to use a different protocol. Something everything supports. Something that is easy to adapt into the chip-level protocol. Like USB. I propose that USB protocol is common enough and flexible enough to do the job well, but that using USB hardware connections internally is a bad idea so something more resilient (and also cheaper, and also allowing for more mechanical reliability in the overall design) is suggested.

Dogie over in a Bitmain thread summed up my opinions of a PCI backplane pretty well in response to an S2 question: "It required a motherboard, slots that often broke, large areas of dead space for cooling and is probably the only Bitmain miner that was susceptible to shipping issues."

 You want POSITIVE pressure in the case with filtered fans, anything else = more dust in the case = WORSE heat dissipation over time.
 Also, positive pressure DOES NOT REDUCE AIRFLOW. Dunno where that myth originates from.

 15 mm spacing between the fans is way excessive. 5 would be more than enough to allow for the differences in fan size tolerances.
 The specified "size" of a fan is the outside dimension of the fan housing, NOT the diameter of the blade (except for "no housing" type fans).


 AND relatively poor cooling. Personally I'd prefer provision for up to 4 long-case ATX power supplies, even if the case has to be longer to fit them.


 He was looking at overall length to figure physical mounting dimensions and limits, not for thermal stress calculations.
 Realistically, stress on the board isn't much of a factor with the relatively low heat density on the heatsinks at his specified max wattage per board vs. heat sink size.

1)  Or have it open across the back to let the air out.

2) No, the PSUs go over the top of the fans, not beside them.  Hence the 4U.

4) Not really any reason to use thermocouples over any other board mounted temp sensor with a cable alongside the USB.  I agree that a totally separate thermocouple is probably asking for trouble.  It's not correct to assume that you would know the input and output air temperature as the number of boards and the type of boards is explicitly undefined.

6) No more than 8 PCBs would fit given the heat sinks, but that many would not be required.

--
novak

1) You want negative pressure displacement, likely you'll get better performance dropping the fronts to 3x120's or 2x140s and leaving a grill. Ramming in more air than is being released = high pressure internally = lowers airflow significantly.

2) There is also a problem with your sizing, 3x120mm don't actually fit at the back.
3x120mm = 360mm
Spacing = 15mm
PSUs    = 2x44mm = 88mm
Material thickness = 2x2mm = 4mm
= 467mm
Your specified back side is 438mm


3) 1U = 40mm fan = annoying as hell. (See SP10 and SP3X where the PSU fan was far louder than the actual miner)


4) Thermocouples are a shipping hazard as they can come loose and end up in fans and so addd another layer of inspection. They take too long to install and they're also superfluous as you should have already characterised the relationship between ambient intake and chip temps.


5) Remember that the distance you actually care about is between the outer two mounting holes, not the length of the entire heatsink. The material on the ends doesn't contribute to thermal expansion mounting stress.


6) You've not explicitly specified how many PCBs.

7) 4U should be smaller than 7" if you want stuff to actually fit, even if its shaving off a few mm.

If you use good fans, push-pull configuration is a waste. The only effect it really has on a rack-mount case is to reduce back pressure, VERY little effect on actual flow.

 Based on measuring all of my existing rack mount cases, I'm not sure if you CAN fit 3x140mm - be nice if you can, but 3x120 is good if you can't.
 ALL of the 4u cases I own have appx. 16.5" of space between the handles (I have one 2U case model with about 17" thoiugh). Height isn't an issue.
 Might need a custom case design, or at worse have to use a specific case model that doesn't have handles.


 IMO don't use USB. USB connections to miners have ALWAYS been an unreliable PITA. Tolerable for small stuff on the block erupter scale as those are more "toys" than real miners anyway, NOT tolerable for a full-up rack mount large miner. PCI-E 1x or just PCI would be tons better from a RELIABILITY standpoint.


 Would be nice if an industry standard existed, but I doubt a custom design would be practical from a cost standpoint and I don't see the MANUFACTURERS ever getting together to create or work with a standard. 8-(

I guess I'm overly cautious. I always worked on the devices with very strict quality assurance requirements: medical or industrial. Things like self-adhesive thermal pads were unacceptable, only mica leaves would do.

The lottery-ticket-printing devices that you are designing have much shorter lifespan, so a more relaxed design approach may be used. On the other hand, things like retail GPU cards have very carefully designed heathsinks: the die or at most 2 dies are in the center, everything else has a heat spreader or some other kind of interposer.

It is interesting, but I need to keep my nose very close to my own grindstone. I'm not in the position to really get involved in the new projects. I'm fine with openly sharing knowledge and commenting in public on the forum.

We checked several other miners for heatsink size comparisons. The S1/3/5 heatsink is 9 inches long; the Avalon2 box and S2 have about ten inch heatsinks. I think the Dragon was more like 8 inches. None of them do quite 300W per heatsinks at stock power either though.

The heatsinks should not be adhered to any chips, unless your thermal paste hardens up I guess. I've never really liked not having some kind of isolation between heatsinks and any electricals. A top-cooler like the BM1384 wouldn't need isolation of its own, but if you had the backside (with its multiple ground planes) on the sink you'd definitely want an electrically insulating thermal pad.

The boards wouldn't have to be string. The 1" at the top can be used for logic level shifters in a string design (or if you drop the small heatsink you have the entire backside for a double-sided board like the S5) or it could be used for VRMs. We don't want the board dimensions to limit your topology choices.

Course, there's also no particular reason the heatsink has to be a single slab. It could be two five-inch chunks maybe with a slight gap between to allow for expansion. It's exactly for considerations like that that we wanted to open discussion - if it's going to be any kind of standard, it needs to be as good as it possibly can be.

I'd like to see the big guys adopt a particular standard, or at least utilize it. Being able to build boards for an existing case without having to build the case also opens up potential for independent manufacturers to get in the game. I mean if you want a computer you can call up Dell or HP or whatever and get a complete system, or you can dig out an ATX case and go fetch guts from NewEgg that all fit inside. You got choices. Some little two-man outfit like mine can build boards but doesn't have the tools to manufacture cases? So what? They can still put out a decent product because you can get a case from someone else - or maybe you already have one. Updating cgminer would be the only really tricky part. If something like PlanetCrypto's independent chip dev happens, or the main guys get back to selling chips to third-party manufacturers like some of 'em used to do pretty regularly, we can have mom-and-pops coming up with innovations and servicing the small-time miners again and doing some pretty good stuff.

2112 - have you talked to PlanetCrypto about chip dev at all? I figured something like what he's doing might interest you.

Testing boards doesn't have to be that tricky. Instead of screwing it onto a greased-up S1 chassis, plugging it in and then taking it all apart again, just rig up a clamp system on a free-standing cooler. Drop a board in, maybe have some pegs to keep alignment on screw holes, light pressure clamp presses it to the heatsink and you plug it up and go. Release the clamp, pull the board and put it in a bag.

I've dreamt about this for quite a while, I hope to see it come to fruition.  The real challenge IMO will be mass adoption in order to keep costs low enough to make it a worthwhile venture.  People don't mind paying extra for the "novelty" factor when it comes to USB sticks, but when you're purchasing a full blown dedicated miner, it's stritly about dollars and cents.  I know your stand on out-sourcing manufacturing, are you planning on sourcing all materials and labour in the US?  Also do you think it is a remote possibility that manufacturers would be willing to produce boards designed with this framework?  

From a challenges point of view, I think of Bitmain and their reluctance to even sell and market upgrade kits to their S1 at the time.  From what I gather part of it had to do with the extra time & labour required to install boards to test them, and then dis-assemble them for again for packaging and shipping. I just can't help but wonder if their decision not to provide kits themselves had to do with the actual production and re-using existing framework, or if it had more to do with not having the ability to run "kits" in their farms and instead sitting on the shelves.

I guess after re-reading the original post I missed the fact that the hashing chips may be in the supply-voltage-serial a.k.a. string configuration. Since each chip will have different ground potential then there is some sort of galvanic isolation provided between the chips and the heathsinks.

So the heathsinks may be able to slide over their isolation layer enough to accommodate thermal expansion.

As in how it really only contacts VRMs and GPU die? yes...

The Bitmain S4 uses big ole aluminum heatsinks... they are mounted horizontally and not vertical but that's what comes to mind... this design is from what I read is several Bitmain Blades in a row with different connectors.  Very much reminding me of this...

https://ip.bitcointalk.org/?u=http%3A%2F%2Fe30.us%2F4sale%2F4u8s1%2FIMG_3871.JPG&t=555&c=bxIiCrGuw5dYEQ