I have had 39 SP30s running 2.5.12 so far for about 12 hours 16 minutes with no apparent issues. If I see no problems with 2.5.12 by next morning, I will probably move the rest of our machines over to it. Otherwise, I will probably move everything back to 2.4.23.
I've had a plethora of "disabled (bist cant start, stuck X)" errors on the remaining 100 or so SP30s we have running 2.5.6 in the mean time. It seems every few hours, I have to restart cgminer on 5% or 10% of our machines to get them back to full hashrate. Very annoying, but so far the issue has always been resolved by restarting minergate.
I've also had several reports of machines on 2.5.6 showing half hashrate at the pool even though I see full hashrate. These issues have always been restarted by a minergate restart.
I saw some much more serious problems on 2.5.7 during the 36 hours we had machines using that version. We saw about 10 machines get "Loop serial failed x" error messages (where x is a loop number, between 0 and 9 inclusive) and fail to start until one or sometimes two ASICs on that loop were disabled. After a few hours, many of these ASICs could be enabled again, but about 25% of the ones that failed are still non-operational. Currently, I think there are 9 ASICs which we lost on 2.5.7 which have not recovered.
It's possible that the potential ASIC-damaging bug in 2.5.7 might still be present in 2.5.12. I haven't seen any indications of it yet, but I also haven't had enough machines running on it for long enough to be certain. On 2.5.7, I saw the loop serial failed ASIC damage thing with about 5% of our machines per day. At that rate, there's only about a 63% chance that I would have seen it if it were present at the same rate in 2.5.12 with 39 machines running half a day.
We moved everything onto 2.5.12 about 5 hours ago (11am PST). Everything looks much better now. When we were running 2.5.6 on most machines, our hashrate would vary between 706 TH/s and around 722 TH/s, depending on how active I had been restarting machines with "bist cant start (stuck x)" errors. Now we're at 724 TH/s (pus another 4.5 TH/s that we added this morning), and everything seems a lot more stable and requires less effort to keep things going. I recommend anyone using an SP30 or SP31 with 2.5.6 to 2.5.10 firmware to upgrade to 2.5.12 immediately. I also recommend to Zvi to mark 2.5.6 as unstable/testing.