Topic: Strange problem with miners, help needed (Read 498 times)

Thank you for procedure on adjusting fan speed. 

Hm. Well, let's try this: Dunk the board in the fluid, and hook it up to real fans. See if it fails.

If so, it's something with the board. Could be the trimmer circuit at the end of the string. If not, it's something with the fan deletes, try running them out of the fluid with extension cables.

One step at a time.

Yeah, I've been trying to isolate the problem. I appreciate the reply. I've already tried switching the fans and fan deletes between two types of miners.

So I do have an update. I did the painstaking process over the past week of removing the miners from the fluid and replacing the fans on them. I then got them all to start mining, and reflashed the firmware.

Then, (this is risky, and I would not recommend trying this), I put the fan deletes on and ran them in the air. Guess what? They started mining right before I turned them off. So, now I have the miners that were originally resetting every so often running on the fan deletes in air.

I put them back into the tank, and they start resetting again. Could this be a tank issue?

This fluid isn't supposed to carry a charge though.

I'm still basically where I was at on day 1.

Sorry for the late replies, holidays are approaching and I have been busy.

Have you tried switching the fans between the two types of miners? Maybe the model not working with your device has a different way to determine RPM.

Try to isolate the problem.

well then start by running them at stock freq to eliminate that option. also check the pools. try using something different, switch PSUs !

also one thing I noticed about electronics in general that you most likely will never find 2 identical , let alone the poor quality control in these Asics, so everything could be really different for GOD knows why. I have some asics from same batches, same cooling, same fan speed and same about everything yet they still have different temps, different hashrate, some need restarts every now and then, some have been on for months with 0 problems. this is just the way it works !

Thank you for the reply. Yes, I have tried custom firmware based off Bitmain's own firmware. This custom firmware allows me to disable the fans, but it doesn't work. The miner still resets.

I could cut the fan blades, but the my main reason for immersion cooling is to cut on utility costs (especially right now lol).

I appreciate it though. I'm open to all suggestions.

---

***Update: Tried Braiins OS and the machine is still resetting.

I'm hoping this is a simple problem I'm overlooking. I just don't understand why the fan mod works on one miner and not this one.

i think you have tried every possible solution that i could suggest to you. now what's left to try a custom firmware like this one > https://bitcointalksearch.org/topic/antminer-s9-volt-rocket-ship-firmware-mod-time-to-max-out-your-bitmain-psu-4513567

please do your own research on this firmware, i have not tested it and i don't know anything about it except for the fact that the dev claims this can run without fans attached


also another thing that i remember one guy mentioned about was cutting the fan blades to reduce the resistance for the cooling fluid, you may keep this as last resort should everything else fails!

@SidSlobber

First off, thanks for the reply. I've flashed the fixed firmware, and have verified the correct firmware on all miners. I have also already tried the fixed duty cycle for the fans.

Seems that the vast majority of the miners not working are s9is (with only 2 S9s not working).

Since my last post, I have purchased a fan delete with a 555 timer and tried that. Sadly, the results were the same. The  miner constantly resets. I am currently in the process of installing Braiins OS. I'm running out of ideas, and since Braiins is a truly 3rd party OS it's worth a shot.

If you have any more logical steps to try I would really appreciate. I feel like I've tried everything.

Thank you.

To me there is nothing obviously wrong in your kernel log.  I've checked it against one of my miners with what I believe is the same hw/fw config as yours and the log is identical "line for line" including "bmminer not found, restart bmminer" and the chain temps etc - so I don't think either of those previous suggestions are your problem.   You've obviously done the logical troubleshooting - i.e. Miner-Bad runs fine and doesn't reset with original fans attached, but still resets even with a known working fan delete mod swapped from Miner-Good.   Frustrating.  It's possibly crashing when it gets to the "checking fans......" stage e.g. this is from one of my auto-tune S9is which occurs around the same time in the kernel log as when yours crashes:


As you have already trouble-shot your hardware mod, it's possible that there is some subtle difference with your Miner-Bad's compared to the Miner-Good's - but the issue only shows up when you attach the fan delete mod. You have probably done this already - but I'd start looking for what is different with these crashing miners - if anything.  E.g. you mention that you have S9's &  S9i's in 13, 13.5 and 14 variants - so is there any obvious pattern with the 7 in your tank that are not working e.g. it would give us more clues if, for example, they were all S9i's etc?   Also as per the quote below, you use an example of 2 x 13.5 S9i Miners, but your kernel log appears to be from a plain S9 - which is fine obviously as you only mentioned S9i's "as an example".    So as the "devil is in the detail" can you just confirm that the kernel log was from an S9 i.e. not from an S9i but with the "wrong" S9 asci-boost firmware applied?    


Until recently, there was only the fixed-freq and the auto-freq firmware for the S9 available.  But since Bitmain launched the S9i and S9j and then also released the asic-boost firmware's (very confusingly in different ways for the 3 S9 models  - i.e. loaded "on-top" of the previous firmware for the S9s, but a total new replacement firmware for the S9i and S9j)  - there are now 11 current "latest" firmware's for the S9 models available on the Bitmain website.   Combined with a lack of Bitmain documentation this has led to the people flashing the wrong firmware to their miners - e.g. S9 FW on an S9i and lots of confusion about the asic-boost update process. The miners may appear to work with wrong firmware in some cases, but it could cause "strange" issues or sub-optimal performance.

So assuming that you are using the standard auto-tune / auto-freq, to completely rule out any FW incompatibility issues, your miners should be as follows if they are up to date:

S9 - the asic-boost patch Antminer-S9-LPM-20181102.tar.gz  flashed on top of the latest auto-tune FW - Antminer-S9-all-201711171757-autofreq-user-Update2UBI-NF.tar.gz which was probably what your miners already had if they were previously up-to-date.  Unfortunately due to the inconsistent way Bitmain released the asic-boost for the S9s as a small patch file, once it is applied "on-top" there is no way of knowing what the "main" firmware version is - the miner's overview page just displays the asic boost version FW details i.e. - "File System Version   Sun Nov 2 11:55:42 UTC 2018"

S9i - just the asic-boost new FW - Antminer-S9i-all-201811071119-autofreq-user-Update2UBI-NF.tar.gz   (the very latest S9i version released a few days ago drops both the power and hashrate by around 40% - which Bitmain either didn't intend or haven't documented - so I guess you don't want to use that).

Apologies if the above is obvious to you and you are certain that you have the correct up-to-date FW on all your different S9 models.  I only mention it as the Bitmain documentation is not clear and some experienced miners have therefore flashed the wrong firmware as can be seen in other sections of this forum.

Therefore if you are 100% certain of your fan delete mods, all the FW is up-to-date and the correct version for the S9 variant and there are no obvious patterns with the "bad miners", then I guess it is probably going to have to be trial and error to get them working without the fans.   A couple of suggestions which are quick to try are:

a. Try flashing a "bad" S9 with the appropriate fixed-freq firmware: e.g. Antminer-S9-all-201705031838-650M-user-Update2UBI-NF.tar.gz  which is for a 14 Th/s S9 but should be fine for all of them.   The fixed-freq FW does a much quicker "boot" process and skips all of the auto-tune and auto-freq processes - so that may help to get round the "fan speed checking" issue.  Some of my miners are using fixed-freq FW and unlike my normal auto-tune miners the are no "fan checking" lines in the log - so it might work for your set-up - but no guarantee.   If it works then you can apply the LPM asic-boost FW on-top which will leave you with a boosted fixed-freq miner still running with less power, but now also faster than before.  Your 13 and 13.5 Th/s S9's may be fine running with the 14 TH/s fixed-freq FW - but if not (i.e. unstable or too many HW errors) just use  http://192.168.1.xx/cgi-bin/minerAdvanced.cgi to get to the hidden config page and set the freq manually to whatever is stable (or back to what the miner originally was e.g. freq=631.25 for 13.5Th/s etc).   Obviously you can't do this for your S9i's as Bitmain doesn't offer any fixed-freq firmware for this model.

b. Another suggestion - it may not work - but it only takes a few seconds to try - is to manually fix the fan speed - or notional fan speed in your case.  I don't know if a fixed "dummy" fan speed is compatible with your "fan delete hw mod".   This should just end up with an entry as follows as the very last lines of the kernel log and hopefully will mean that the fan speed does not get checked during restart - and hence hopefully stop your miners from crashing.  
  

As you are doing immersion cooling, you are probably far more expert than me with tweaking fans etc, however below is the method I use to manually over-ride the auto-tune fan speed - only takes a few seconds + a restart (it's only temporarily revealing the hidden options in the standard Bitmain miner web-server - so it's not changing any code etc):

The November firmware is well known to cause many problems including the bmminer not found issue. I would NEVER recommend this firmware to anyone.

According to your kernel logs "bmminer not found, restart bmminer ..."
This might be the reason why the miner keeps rebooting. The only way that I know you can fix this is to reflash it with other firmware.

Can you try to flash it with this firmware Antminer-S9-all-201711171757-autofreq-user-Update2UBI-NF.tar.gz
Or downgrade it to "November firmware".

Edit:
It looks like its unable to detect the chain 5,6 and 7 temps. There is something wrong reading the temp, not a software it looks like a faulty sensor.

Thank you for the reply:


That's how far it gets before it resets.

The miner works fine with the air cooled fans it came with.

With a fan delete or firmware which overrides the fans it just constantly resets.

I've tried using one of the fan deletes (linked earlier) that works on a good miner on one of the bad miners and it still just resets.

Can you post the logs of the machines that keep restarting? That would help immensely in trying to troubleshoot the issue.

I posted about this problem recently, and was given a solution to download firmware. I tried that with the same problem. So i'm back again to explain in more detail. Thanks for the replies.

So, I'm going to try and keep this short and simple. I run a small mining setup (100 miners) using immersion cooling. Currently, I have a mixture of S9-S9i from 13 to 14 TH/S.

I purchased some fan delete mods (These but at a cheaper price from someone else: https://www.amazon.com/Wave-Pulse-Generator-Adjustable-1Hz-150KHz/dp/B01MA1M7Y9 ). I've been soldering and attaching these to the miners.

This is where the problem starts. Just remember, these miners work perfectly with the out of the box air cooled fans attached.

So let's say I have 2 13.5 TH S9i miners (I'll call them Miner-Good and Miner-Bad). I flash them both with the new AsicBoost firmware recently released by Bitmain. I then attach the fan deletes. Miner-Good works flawlessly. Miner-B, however, starts resetting itself every 15-30 seconds.

So, I decide to replace the fan delete on Miner-Good, and place it on Miner-Bad. Miner-Bad still just resets.

I tried downloading custom firmware to disable the fan check. Miner-Good works fine but Miner-Bad still resets.

The only way I can get Miner-Bad to stop resetting is to reattach the original air cooled fans. I just don't understand why. They are both s9i's at 13.5 TH. They have the same firmware.

And this isn't an issue with just one miner, in one tank I have 17 miners that work, and 7 that do not work (all resetting unless I attach original fans).

Does anyone have any other ideas or solutions? I'm lost.

Thank you. I hope I was clear, but I'm not much of a writer