Topic: Stay away from ASICSPACE (Read 7625 times)

I observed cold aisle temps around 35°C. My estimate of cold aisle temps around 55°C in the weeks before I visited was based on several factors:

1. There were network problems when I visited, resulting in less heat generated. It looked like about 50% of ASICSPACE's miners were off while I was there due to the network problems. If the 50% estimate is correct, then the delta T versus outside might be sqrt(2) to 2 times higher, depending on how much higher inside temperatures assist the stack effect. As the cold aisle temps were already 20°C hotter than outside, this alone might be enough to produce cold aisle temps of 55°C even on a somewhat cool day. However, my 50% estimate could be off. Also, many of the machines were S4s, which may also produce heat while the network is disconnected. For my estimate, I think I was generous and presumed that the network problems reduced temps by 8°C.

2. The nine flexible ducts supplementing the cold aisle with outside air at ambient pressure looked to be hastily added. I presumed they were added within the last week before my visit, and were not present when the damage occurred. I estimate they added about 20% to the airflow into the cold aisles. Roughly speaking, that should have decreased the cold-aisle/outdoor delta T by 20%, which would be about 4°C on the day I visited, or possibly as much as 8°C on other days. There may have been other similar quick fixes that I did not notice. I used 2°C in my quick estimate.

3. When I visited (6pm to 11pm), the outside temp was 16°C or lower. The high the day before was 26°C. That's 10°C. I think there was a day or two in the week or two prior that was even hotter.

8°C + 2°C + 10°C = 20°C
35°C + 20°C = 55°C

Other effects: as the temperatures rose, many machines may have turned off. On the other hand, reliance on the stack effect may have worsened the effects of a high outside temperature.

My math might not be right, of course. It's just an educated guess.


The damage we observed was not linear. In terms of deformation, pretty much all of the plastic panels were deformed. Most of the panels were only slightly deformed, maybe 1 or 2 mm per panel. Other panels were very heavily deformed, with about 20 mm of deformation at several different points on each panel. If the maximum temperature seen by each miner was normally distributed, and the amount of deformation as a function of temperature is something like D(T) = e^T, that would fit our observed distribution of deformation pretty closely.

We saw something similar with the amount of electrical damage on each hashboard. Most of the hashboards had either zero bad stages or more than two bad stages. This could also be caused by a domino effect, though.

It's also worth mentioning that the damage to the two hashboards in a single miner were correlated. Most of the miners we received either had two bad hashboards or zero bad hashboards. The number of machines with exactly one bad hashboard was less than I had expected.


I disagree. I do not see how air pressure alone could have caused this deformation. I just went over and tested, and using my finger, it takes about 20 pounds of force to permanently deform a S5 side panel at room temperature by about 3 mm. That would mean you'd need close to 1 atmosphere of pressure difference across the panel to deform it from pressure alone. Typical HVAC system pressure differences are on the order of 100 Pa (0.015 PSI). Axial fans typically have static pressure capabilities around 300 Pa. If we are very pessimistic and say that the hot aisle was 600 Pa higher pressure than the cold aisle due to the two S4 fans (i.e. the S4s had zero net airflow, 100% static pressure), then that would mean about 0.06 newtons per square centimeter, or 0.08 PSI. I think temperature must have been a large factor.

Polylactic acid has a glass transition temperature of 60°C to 65°C. What are the panels actually made of? I don't know. Also, even below or above the glass transition temperature, plastics will become softer and more plastic as temperature increases. The Vicat softening temperature for HDPE is about 70°C, for example. The 0.08 PSI might not be enough to deform it at all at 20°C, but it might be enough to deform it by 1 mm at 70°C, and enough to deform by 10 mm at 80°C. I do not feel inclined to test this right now, but if this goes to court, it would be simple enough to verify.

We essentially completed our investigation of the damaged units a week ago, but now we're busy trying to repair them (and the PSUs that ASICSPACE included with them) and to install another 100 S5s, so we haven't had much time to comment here. We kept CryptoCoin2015 updated about the progress of our investigation as we performed it. Here is some information:

Electrical problems

About 40% of the hashboards are damaged and defective, and do not hash at all. My initial count was 89 damaged hashboards out of 200, but I think that was a slight overestimate because I made that count on a weekend and misinterpreted a pile that one of my employees had made. I'll have to make another count soon, but the total is probably closer to 70 damaged hashboards. There may be another half-dozen hashboards that still hash with some dead ASICs. We did not observe any otherwise dead hashboards working if we reduced the clockspeed.

About 30% of the damaged hashboards had visibly burned or popped SMT G337 2V capacitors. This means about 12% of all of the hashboards. Many had several capacitors damaged on a single board. We didn't count all of these; this is just an estimate based on a small random sample.

The resistance across the 12V and Gnd pins for a disconnected hashboard should be around 56 to 58 ohms. This translates to about 4 ohms per stage. (I think it's actually around 4.5 ohms per stage with a parallel resistance of ~330 ohms from 12V straight to Gnd, but that doesn't matter much for this.) All of the working hashboards were in this 56 to 58 ohm range. All of the broken hashboards had lower resistances than this. About 30% had resistances close to 53 ohms, suggesting that they had one bad stage. The rest had resistances below 50 ohms, and about half of them were 30 or below. This suggests several bad stages is typical for the failed hashboards.

We measured the voltage across each stage for a few different hashboards. Here is one such:

15-14:   0.990
14-13:   0.953
13-12:   0.055
12-11:   1.023
11-10:   0.999
10-09:   0.780
09-08:   0.360
08-07:   0.994
07-06:   0.948
06-05:   1.054
05-04:   0.477
04-03:   1.017
03-02:   0.995
02-01:   1.061
01-gn:   0.478

All of the working stages have nearly equal voltages. On this hashboard, those voltages are around 0.980 V +/- 0.070 V. On a good hashboard, they are around 0.800 V, +/- 0.070 V. The voltage abnormalities indicate failed stages. I don't have the resistance measurements recorded, but they were similar, with powered-on voltage per stage corresponding closely with powered-off measured resistance. Several stages were measured at 0.8 ohms, which is the same resistance I measure when I short the leads of the multimeter together. (Multimeters typically are not good at measuring very small resistances, due to the additional resistance of the test probes and leads.)

All of the visibly damaged capacitors were on stages with abnormal stage voltage and stage resistance. Not all stages with abnormal voltage or resistance had visibly damaged capacitors. Removing and replacing damaged capacitors did not fix the abnormal voltage or resistance. (I did not test to see if they hashed after replacing the caps; I just assumed they didn't.) I also tested three severely damaged capacitors after removing them, and despite the crazy physical damage, they still showed the correct electrical characteristics (320 to 370 µF capacitance, DC resistance > 10 Mohm). This suggests that the functionally important damage was to a component other than the capacitors. The most likely candidate is the ASICs themselves.

The G337 2V capacitors are probably aluminum polymer capacitors. Those are typically rated for 105°C operating temperatures. This is lower than the 125°C silicon operating temperature for most mining ASICs and most other discrete power components. I think it's possible that a cooling failure could have caused the capacitors to explode before or at the same time as damage to the ASICs occurred, even though the capacitors were not themselves generating any heat.

We overnighted four bad hashboards to Bitmain Warranty in Denver, CO. I haven't heard back from them about those specifically, but my guess is that they discarded them as unrepairable.

Plastic shield deformations

Most of the S5s we received from ASICSPACE have some deformation of the plastic air guide shield panels. The severity varies widely. There seems to be a correlation between the deformation of the plastic shields and whether the hashboard is broken. There may also be a correlation between deformation and presence of damaged capacitors. I haven't made a close study of that, though. Mostly, all I know is that the shields we took off when investigating broken hashboards tend to be worse than the shields that we left on.

It appears that the worst point on the plastic shields is on the "tail" end (exhaust side) of the miners. The deformation is typically greatest (i.e. shortest radius of curvature) in between the screws. The top edge (where the PCIE connectors and control board are) is also heavily deformed on many machines. In all cases, the deformation causes the shields to bend away from the case of the Ant. In a few cases, the plastic has a wavy appearance in between the screw holes up to approximately 1 cm in from the edge, suggesting that the plastic there had been stretched or elongated and had caused the surface to take a non-linear path between the two fixed screw points. None of the S5s that have been in the Toom.im facility since January show any hints of deformation like this. The only machines like this that we've seen were in ASICSPACE during early Aprli.

Most of the shields have no scuff marks or indications of physical trauma. The deformations are smooth, with no creases. I do not think these deformations were caused by contact with a solid object. Deformation due to air pressure would explain these deformations in every case I've looked at, as long as the plastic were soft enough. I'll describe how I think that happened after I mention some of what I've seen about the environment in which they were operating.

First-person observations

When I visited to pick up the S5s, ASICSPACE's cold aisle was negatively pressurized relative to the outside air. The hot aisle was positively pressurized relative both to outside and to the cold aisle. As a result, the cold aisle was also by no means "cold". Near the cold air supply ducts, the air felt like it was about 15°C. In most of the rest of the cold aisle, it felt like it was 35°C. When I first arrived, I walked past a gap in the cold aisle containment where an Antminer S4 had previously been. The velocity and volume of airflow through that hole was comparable to the airflow coming out of their cold air supply duct outlet, except a lot hotter. I think it felt like about 500 to 1000 CFM through that hole. As we removed S5s, and more holes appeared in their cold air containment, the velocity through each hole decreased substantially. ASICSPACE had noticed this pressure difference, and as a way of mitigating it had set up 9 air ducts (approx. 0.5 m in diameter each) going from their garage door to their cold aisles to serve as supplementary air intake. Note that in proper cold aisle containment design, the cold aisle should be positively pressurized, so ducts like this would normally let air out (and thus normally would not be used). I also spoke with Robert about this, and he was (fortunately) aware of this problem. The ducts looked like they had been hastily added, likely within the previous week. While I was there, their network was experiencing severe problems, causing a large proportion of their machines to stop hashing (and thus not produce heat). It was also not a very hot day, and it was evening when I arrived, making it about 11°C cooler than the daytime high the day before. I can only imagine what their facility was like during the weeks prior.

I did not see any significant exhaust fans installed at ASICSPACE to remove hot air. There were some small fans mounted above the hot aisle, but they were unducted, small, and not very numerous, so I guess they were not significant. There may or may not have been some exhaust fans on the roof of ASICSPACE. From what I've heard through other channels, they chose not to install exhaust fans, and were instead relying on the stack effect to move heat up through their tall (30m?) building and exhaust through natural convection at the top. Unfortunately, the stack effect relies on the interior of the building being hotter than the outside temperature, so when the outside temperatures increase to 30°C, their interior temps would have to rise too. Since they're pre-cooling with evaporative coolers, this means that the air inside their building would have to recirculate several times until it was enough hotter than the outside air for the stack effect to blow it out. However, if they have 300,000 to 800,000 cfm of exhaust fans that I didn't see, this paragraph is irrelevant. (Edit 5/6/2015: We visited ASICSPACE again today, and Robert showed me their exhaust system. They do not have exhaust fans, and are relying on the stack effect plus (in principle) positive pressure from their intake fans.)

The containment sealing system (which looked pretty tight, to be honest --  kudos to ASICSPACE for that) sealed off the exhaust end of each miner, with a gap between the case and the edge of the sealing panel on the order of 1 or 2 cm.

The S5s were installed next to pairs of S4s along both sides of the cold aisle on each of the shelves. This indicates that the S4s and S5s were competing with each other for airflow. The S4 is a sealed tunnel with 4 fans arranged in a push-pull configuration -- i.e., 2 in parallel by 2 in series. Placing fans in series multiplies the amount of static pressure they can produce, and also allows them to maintain their airflow quite well when working against a significant positive pressure gradient, but does very little for the airflow when working in a neutral-pressure regime. This means that the S4 fans were able to to pressurize the hot aisle quite effectively. The S5, on the other hand, is an open semi-tunnel configuration with a single fan in push configuration. The semi-tunnel has large gaps near the exhaust end through which air can escape out the top, as well as small gaps on the bottom.

The power supplies which ASICSPACE had obtained for these S5s were the DPS-800GB using the Gigampz breakout boards. These boards are miswired to connect pin 30 (voltage adjust pin) to the 12V rail, which increases their output voltage to around 12.80 V (no load). Power consumption for the S5s' BM1384 (as with most ASICs) is proportional to frequency times voltage squared, so the 6.6% higher voltage should result in 13.8% more power consumed and heat generated.

What I think happened

As the single S5 fans were unable to compete with the doubled S4 fans in terms of pressure output, the airflow from the S5 fans would instead curve out through the gaps in the S5 pseudo-tunnel. Meanwhile, the positive pressure from the hot aisle may have been strong enough to cause airflow to go in reverse from the hot aisle into the "exhaust" port of the S5, and then out the top of the miner. This retrograde flow was likely strongest in the side areas, in between the hashboards and the plastic shields. These two effects caused the interior and side spaces of the S5s to get very hot and positively pressurized. The heat caused the plastic sides to become soft and "plastic" (in the non-elastic sense of the word). The positive pressure then stretched and deformed the heat shields away from the mounting screws. The deformation caused a small gap to appear at the tail end of the shields in between the two screws. This gap allowed for a very short path -- approximately 3 cm -- for the air to go from the hot aisle to the cold aisle, by entering the tail of the miner through the side slots, making a roughly 150° turn to curve around the steel frame (and partially bouncing off the plastic shield), and then exiting backwards/sideways through the tail/side gaps in between the screws. This air was the closest to the hot aisle, and thus the hottest and highest pressure, and consequently caused the greatest deformation, enlarging these gaps considerably over time. Additional airflow that came in through the side slots would have passed out the top of the miner. Though this airflow was greater, the size of the gaps on the top were much larger, causing lower pressure differences, which made the radius of curvature of the deformation in that area smaller.

The intersection of the anterograde and retrograde airflows near the center of the miner caused airflow there to be relatively stagnant. This, combined with the higher temperature of the retrograde flow, would have caused very high temperatures about 3 to 10 cm away from the miner's exhaust port.

The Antminer S5 has a bug (linked elsewhere on this page) in which the miner's fans will stop immediately when cgminer dies (or when the network is disconnected), but the hashboards will continue to generate heat for several minutes. Given that ASICSPACE was having frequent and persistent networking problems during this time, I expect that to be a contributing factor in this case. However, I think that is at most a contributing factor, since many people have reported that bug in the absence of heat damage, and I haven't seen any other reports attributing actual heat damage to that bug.

Based on what I saw when I visited and what I know had been changed recently at ASICSPACE, I estimate cold aisle temps were likely 20°C hotter during early/mid April at ASICSPACE, and likely reached about 55°C. I have also heard people allege to have seen 57°C intake temps on their miners at ASICSPACE at one point (I think that was March), although I can't mention the source for that right now so you should treat it as a rumor. I have also seen screenshot evidence from a KNCMiner device indicating temperatures in the same ballpark. With mostly S4s and a few SP35s, plus with the fans working against positive pressure and consequently having reduced forward airflow, their delta-T might have been around 10°C, or possibly a little higher. I thus estimate their hot aisle temps at around 65°C. The air passing from the hot aisle through the S5 heatsinks would have gotten even hotter before hitting the side panels; perhaps 15°C hotter. Many plastics have glass transition temperatures around 80 to 120°C. ABS, for example, is 105°C. I thus think it is plausible that the plastic side panels were heated close enough to their glass transition temperature that air pressure differences caused them to permanently and plastically deform. Having a 55°C cold aisle with strong positive pressure in the hot aisle would also explain why the ASICs and capacitors in the S5s would have burned out in such large numbers, especially if the 80°C protection was bypassed either due to a Bitmain bug or due to an attempt to get the S5s to hash despite the poor working conditions.

Note: As the head of Toomim Bros, I am a competitor of ASICSPACE, and clearly have a conflict of interest in this case. Apply salt liberally before hashing.

Did Toomim Brothers already give a timeline when first, or the complete, results will be available?

Thanks for the advice and the heads up !

You're also better off sticking with what you know and not arguing with Jared. I wouldn't put his working knowledge of creating mines too far above the people he left behind who cooked your units, and I think even you can see he's here more to argue than to solve anything. Circular logic and speculation, are exactly that. Good for you for having the balls to stand up to these folks. Of course the twits here are going to poke all kinds of holes in what you're doing. Fair point though, you won't see a criminal prosecution out of this, so go for civil. Good luck.

So since i did not want to remove the NEUTRAL rating for you i now get you going after me?

Its clear why i rated you since i linked the problem with the rating. And you still werent able to explain how someone would have to be refunded for a miner that he rated as perfectly fine, ran it for months until it was worthless, and then got the idea that this miner was so broken that it could have not ever been sold. Then starting a campaign and tries everything to move dogie for a refund.

Whatever, i dont want to go about this in this thread. Maybe you should consider how much worth is in going after others you blame for your own errors. You look like you have enough time to create something constructive instead.

Yep, and this is virtually impossible, at least with the given evidence at hand and the limited availability of other customers willing to sue (they have intends, but fear the costs which would add on to their losses).

For any criminal prosecution you need to prove intent, and no matter how incompetent they may have proven to be, you will never ever get any D.A to file criminal charges in this type of dispute. To think otherwise is criminal in and of itself.. IMHO

Quite the thread...

In case it wasn't already said... Criminal and civil law are not mutually exclusive..

Is it a civil/contract dispute, without a doubt and the limited facts I have seen from the OP, and another customer I have crossed paths with show there may be a case..

Is it criminal, perhaps if a pattern of willful misconduct can be shown but that is a much higher bar to cross and it will be tough to claim without hard evidence, which in itself might be hard to come by..

I never said it was theirs, I only said it was an intent and means to deliver.  Asicspace is contractually bound to that space and it to them, which definitely passes the bar for an intent and means to deliver on contracts.  If you think being pedantic about it matters, I don't think a single big Bitcoin mine operator owns their land, at least none in central WA.  Amazon.com doesn't even own the buildings they are headquartered in, none of the entire lake union campus, and they don't manufacture any of the Amazon products they sell.  Microsoft didn't even have an operating system when they sold one to IBM.  None of that has ANYTHING to do with whether someone intends to or is capable of delivering.

I know far more than you have been told.  Others you esteem highly are not the angels you think they are.  But I'm sure they didn't tell you that part of the story.

** Going back to clarify for someone reading this later, I'm not saying Asicspace did right or good here or that I support them or don't.  All I'm saying is I know they don't have criminal intent, and I know they have not broken any laws.  I'm almost certain that the temperature didn't get hotter (or equal to) other functional mines I've seen(big and small).  I know they have cooling improvements coming.  I also know they have had network outages.  I have no opinion on whether they did or didn't actually damage or contribute to the damage of the hardware. **

This is not ASICSPACEs property. They are managing this, but it is not theirs. They own jack shit and their net value is close to 0.

You know far less than you should, if your claims are true and you know them so well.

Lets agree to disagree and be done with it.

Judge: "So you are claiming that these individuals are not actual hosters for these servers?"
You: "Yes your honor, as we have shown, these individuals clearly made an agreement with the full knowledge that they were and are incapable of fulfilling it, and -"
Judge: "Wait, hold on.  They were hosting for others before you, right?"
You: "Well yes, but they-"
Judge: "And today, months later, they are still hosting for others, right?"
You: "Well, yes but they-"
Judge: "Case dismissed.  Next plaintiff."


Jeez, you are contradicting yourself in two sentences: First, you claim both sides are at fault, then you state, that you are in no position to judge at all. Stick to the latter, please!