I could't see which hashboard detect over temp would you mind to try to run the miner with one hashboard? I mean test them one by one disconnect the two hashboard power and data cable and test again to find what hashboard is broken.
Also, try to hard reset the unit by holding the IP report button for 5 seconds or more then release then test it again and check for changes.
This is interesting. So I got the middle board identified as the error. the log is below.
I did do a hard reset this morning, and was still having the issue. I did it again after identifying the board and now now it works.
Error log with just the middle board running:
On node 0 totalpages: 61440
free_area_init_node: node 0, pgdat c0b3c040, node_mem_map cde10000
Normal zone: 480 pages used for memmap
Normal zone: 0 pages reserved
Normal zone: 61440 pages, LIFO batch:15
percpu: Embedded 12 pages/cpu @cddf0000 s19916 r8192 d21044 u49152
pcpu-alloc: s19916 r8192 d21044 u49152 alloc=12*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 60960
Kernel command line: mem=240M console=ttyPS0,115200 ramdisk_size=33554432 root=/dev/ram rw earlyprintk
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 209672K/245760K available (6317K kernel code, 243K rwdata, 1932K rodata, 1024K init, 232K bss, 19704K reserved, 16384K cma-reserved, 0K highmem)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xcf800000 - 0xff800000 ( 768 MB)
lowmem : 0xc0000000 - 0xcf000000 ( 240 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc090e410 (9242 kB)
.init : 0xc0a00000 - 0xc0b00000 (1024 kB)
.data : 0xc0b00000 - 0xc0b3cda0 ( 244 kB)
.bss : 0xc0b3cda0 - 0xc0b77024 ( 233 kB)
Preemptible hierarchical RCU implementation.
Build-time adjustment of leaf fanout to 32.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
efuse mapped to cf800000
ps7-slcr mapped to cf802000
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at cf802100
Zynq clock init
sched_clock: 64 bits at 333MHz, resolution 3ns, wraps every 4398046511103ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x4ce07af025, max_idle_ns: 440795209040 ns
Switching to timer-based delay loop, resolution 3ns
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 537538477 ns
ps7-ttc #0 at cf80a000, irq=18
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 666.66 BogoMIPS (lpj=3333333)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100058
CPU1: failed to boot: -1
Brought up 1 CPUs
SMP: Total of 1 processors activated (666.66 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor menu
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
zynq-ocm f800c000.ps7-ocmc: ZYNQ OCM pool: 256 KiB @ 0xcf880000
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
media: Linux media interface: v0.10
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti
PTP clock support registered
EDAC MC: Ver: 3.0.0
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm_global_timer
NET: Registered protocol family 2
TCP established hash table entries: 2048 (order: 1, 8192 bytes)
TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
PCI: CLS 0 bytes, default 64
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (no cpio magic); looks like an initrd
Freeing initrd memory: 6632K (cd480000 - cdafa000)
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
futex hash table entries: 512 (order: 3, 32768 bytes)
workingset: timestamp_bits=28 max_order=16 bucket_order=0
jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc.
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
dma-pl330 f8003000.ps7-dma: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.ps7-dma: DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
e0000000.serial: ttyPS0 at MMIO 0xe0000000 (irq = 159, base_baud = 6249999) is a xuartps
console [ttyPS0] enabled
xdevcfg f8007000.ps7-dev-cfg: ioremap 0xf8007000 to cf86e000
[drm] Initialized drm 1.1.0 20060810
brd: module loaded
loop: module loaded
CAN device driver interface
gpiod_set_value: invalid GPIO
libphy: MACB_mii_bus: probed
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 31 (00:0a:35:00:00:00)
Generic PHY e000b000.etherne:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=e000b000.etherne:00, irq=-1)
e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
i2c /dev entries driver
cdns-i2c e0005000.ps7_i2c: 100 kHz mmio e0005000 irq 154
Xilinx Zynq CpuIdle Driver started
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
mmc0: SDHCI controller on e0100000.ps7-sdio [e0100000.ps7-sdio] using ADMA
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nand: disable subpage write
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xda
nand: Micron MT29F2G08ABAEAWP
nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
nand: NAND_ECC_HW
nand: NAND_ECC_HW_SYNDROME
mtd->writesize = 2048
ecc->strength = 1
ecc->size = 2048
mtd->writesize = 2048
chip->ecc_strength_ds = 4
chip->ecc_step_ds = 512
nand: WARNING: pl35x-nand: the ECC used on your system is too weak compared to the one required by the NAND chip
Bad block table found at page 131008, version 0x01
Bad block table found at page 130944, version 0x01
8 ofpart partitions found on MTD device pl35x-nand
Creating 8 MTD partitions on "pl35x-nand":
0x000000000000-0x000002800000 : "BOOT.bin-dts-marker-kernel"
0x000002800000-0x000004800000 : "ramfs"
0x000004800000-0x000005000000 : "configs"
0x000005000000-0x000005200000 : "sig"
0x000005200000-0x000006000000 : "reserve1"
0x000006000000-0x000007000000 : "upgrade-ramfs"
0x000007000000-0x00000a800000 : "upgrade-file"
0x00000a800000-0x000010000000 : "reserve2"
nf_conntrack version 0.5.0 (3635 buckets, 14540 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 10
ip6_tables: (C) 2000-2006 Netfilter Core Team
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
can: controller area network core (rev 20120528 abi 9)
NET: Registered protocol family 29
can: raw protocol (rev 20120528)
can: broadcast manager protocol (rev 20120528 t)
can: netlink gateway (rev 20130117) max_hops=1
zynq_pm_ioremap: no compatible node found for 'xlnx,zynq-ddrc-a05'
zynq_pm_late_init: Unable to map DDRC IO memory.
Registering SWP/SWPB emulation handler
hctosys: unable to open rtc device (rtc0)
ALSA device list:
No soundcards found.
RAMDISK: gzip image found at block 0
EXT4-fs (ram0): couldn't mount as ext3 due to feature incompatibilities
EXT4-fs warning (device ram0): ext4_update_dynamic_rev:746: updating to rev 1 because of new feature flag, running e2fsck is recommended
EXT4-fs (ram0): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 1024K (c0a00000 - c0b00000)
EXT4-fs (ram0): re-mounted. Opts: block_validity,delalloc,barrier,user_xattr,errors=remount-ro
devpts: called with bogus options
ubi0: attaching mtd2
ubi0: scanning is finished
ubi0: attached mtd2 (name "configs", size 8 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 64, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 12/4, WL threshold: 4096, image sequence number: 606679770
ubi0: available PEBs: 0, total reserved PEBs: 64, PEBs reserved for bad PEB handling: 4
ubi0: background thread "ubi_bgt0d" started, PID 729
UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 733
UBIFS (ubi0:0): recovery needed
UBIFS (ubi0:0): recovery completed
UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "config_data"
UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi0:0): FS size: 5840896 bytes (5 MiB, 46 LEBs), journal size 1015809 bytes (0 MiB, 6 LEBs)
UBIFS (ubi0:0): reserved for root: 275879 bytes (269 KiB)
UBIFS (ubi0:0): media format: w4/r0 (latest is w4/r0), UUID 40C2E32E-4292-42D3-B3A6-4455C3E2D40A, small LPT model
ubi2: attaching mtd4
ubi2: scanning is finished
ubi2: attached mtd4 (name "reserve1", size 14 MiB)
ubi2: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi2: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi2: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi2: good PEBs: 112, bad PEBs: 0, corrupted PEBs: 0
ubi2: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi2: max/mean erase counter: 11/5, WL threshold: 4096, image sequence number: 3248766587
ubi2: available PEBs: 0, total reserved PEBs: 112, PEBs reserved for bad PEB handling: 4
ubi2: background thread "ubi_bgt2d" started, PID 740
UBIFS (ubi2:0): background thread "ubifs_bgt2_0" started, PID 744
UBIFS (ubi2:0): recovery needed
UBIFS (ubi2:0): recovery completed
UBIFS (ubi2:0): UBIFS: mounted UBI device 2, volume 0, name "misc"
UBIFS (ubi2:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi2:0): FS size: 11935744 bytes (11 MiB, 94 LEBs), journal size 1015809 bytes (0 MiB, 6 LEBs)
UBIFS (ubi2:0): reserved for root: 563754 bytes (550 KiB)
UBIFS (ubi2:0): media format: w4/r0 (latest is w4/r0), UUID A9EF88C2-A577-4085-8DC6-BB5676C496A1, small LPT model
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
random: avahi-daemon urandom read with 2 bits of entropy available
macb e000b000.ethernet eth0: unable to generate target frequency: 25000000 Hz
macb e000b000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
In axi fpga driver!
request_mem_region OK!
AXI fpga dev virtual address is 0xcfbc8000
*base_vir_addr = 0xb031
In fpga mem driver!
request_mem_region OK!
fpga mem virtual address is 0xd2000000
===========================================Miner log===========================================
1970-01-01 00:00:09 Miner compile time: Fri Dec 11 11:23:44 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:10 This is fix-freq version
1970-01-01 00:00:10 Miner compile time: Fri Dec 11 11:23:44 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:10 commit version: 1821c90 2020-11-16 16:05:37, build by: jenkins 2020-12-11 11:35:43
1970-01-01 00:00:10 opt_multi_version = 1
1970-01-01 00:00:10 opt_bitmain_ab = 1
1970-01-01 00:00:10 mid_auto_gen = 0
1970-01-01 00:00:10 opt_bitmain_work_mode = 0
1970-01-01 00:00:10 mmap fpga_mem_addr_hal = 0xb5700000
1970-01-01 00:00:10 HASH_ON_PLUG V9 = 0x4
1970-01-01 00:00:10 Note: front fan is power on!
1970-01-01 00:00:10 Note: rear fan is power on!
1970-01-01 00:00:10 start the http log.
1970-01-01 00:00:10 httpListenThread start ret=0
1970-01-01 00:00:10 start listen on 6060 ...
1970-01-01 00:00:10 load machine NBP1901 conf
1970-01-01 00:00:10 machine : NBP1901
1970-01-01 00:00:10 chain_num 4, chain_domain_num 38, chain_asic_num 114, domain_asic_num 3
1970-01-01 00:00:16 miner ID : 800aed042b10481c
1970-01-01 00:00:16 FPGA Version = 0xB031
1970-01-01 00:00:16 HASH_ON_PLUG V9 = 0x4
1970-01-01 00:00:16 ==========================capability start==========================
1970-01-01 00:00:17 board num = 1
1970-01-01 00:00:17 board id = 2, chain num = 1
1970-01-01 00:00:17 chain id = 2
1970-01-01 00:00:17 ==========================capability end============================
1970-01-01 00:00:17 chain num = 1
1970-01-01 00:00:18 [chain 2]
1970-01-01 00:00:18 0x0000 11 42 84 E6 07 D3 6D B7 45 BE D3 05 4F 26 99 A6
1970-01-01 00:00:18 0x0010 95 99 40 22 CD CD B4 58 5E 7B 33 FE 4C 06 CD 92
1970-01-01 00:00:18 0x0020 E1 EA B3 D4 BE 5E 05 B9 54 E6 E6 7C B0 A4 F5 13
1970-01-01 00:00:18 0x0030 29 1D FC 20 06 55 C9 CB D9 B3 1A 7C C3 C0 C0 C8
1970-01-01 00:00:18 0x0040 54 39 FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x0050 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x0060 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x0070 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x0080 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x0090 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00A0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00B0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00C0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00D0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00E0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
1970-01-01 00:00:18 0x00F0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 5A
1970-01-01 00:00:18
1970-01-01 00:00:18 fan_eft : 1 fan_pwm : 100
1970-01-01 00:00:18 create thread
1970-01-01 00:00:18 fixed working voltage = 1260
1970-01-01 00:00:18 Chain [2] PCB Version: 0x0100
1970-01-01 00:00:18 Chain [2] BOM Version: 0x0100
2022-11-07 00:19:44 Fan check passed.
2022-11-07 00:19:45 chain[2] PIC jump to app
2022-11-07 00:19:47 Check chain[2] PIC fw version=0x89
2022-11-07 00:19:47 create thread
2022-11-07 00:19:47 max sensor num = 4
2022-11-07 00:19:47 temperature_monitor_thread start...
2022-11-07 00:19:51 power type version: 0x0071
2022-11-07 00:19:51 Enter sleep to make sure power release finish.
2022-11-07 00:20:22 Slept 30 seconds, diff = 0.
2022-11-07 00:20:22 set_voltage_by_steps to 1500.
2022-11-07 00:20:25 chain avg vol rise to 15.33
2022-11-07 00:20:27 start up min temp by 75a = 0
2022-11-07 00:20:27 set UART baud to 115200
2022-11-07 00:20:34 Chain[2]: find 114 asic, times 0
2022-11-07 00:20:34 pulse_mode = 1, ccdly_sel = 1, pwth_sel = 1
2022-11-07 00:20:36 min freq in eeprom = 525
2022-11-07 00:20:36 fixed frequency is 525
2022-11-07 00:20:36 Bring up temperature is 0
2022-11-07 00:20:36 set UART baud to 12000000
2022-11-07 00:20:36 set_voltage_by_steps to 1310.
2022-11-07 00:20:43 STATUS_INITED: soc init done!
2022-11-07 00:20:43 create thread
2022-11-07 00:20:43 create thread
2022-11-07 00:20:44 fan_etf: Set fixed fan speed=100
2022-11-07 00:20:46 start to init...
2022-11-07 00:20:54 Init done!
2022-11-07 00:20:54 STATUS_OKAY
2022-11-07 00:20:54 start the cached job
2022-11-07 00:20:54 Version num 8
2022-11-07 00:20:54 Mask num 0xe000
2022-11-07 00:20:54 freq = 525, percent = 90, hcn = 12480, timeout = 449
2022-11-07 00:20:54 set_start_time_point total_tv_start_sys=97 total_tv_end_sys=98
2022-11-07 00:20:54 set_voltage_by_steps to 1335.
2022-11-07 00:21:03 over max temp, pcb temp 0 (max 80), chip temp 15(max 95) pcb temp rise 0 chip temp rise 0, total_exit_failure 0
2022-11-07 00:21:03 ERROR_TEMP_TOO_HIGH: over max temp
2022-11-07 00:21:03 stop_mining: over max temp
2022-11-07 00:21:03 uninit_temp_info
2022-11-07 00:21:03 do not read temp anymore...
2022-11-07 00:21:03 cancel thread
2022-11-07 00:21:03 cancel thread
2022-11-07 00:21:03 cancel thread
2022-11-07 00:21:03 ****power off hashboard****
2022-11-07 00:21:04 temp monitor thread exit
