So ... ... fuck, crap, $^*# *(#$&*(&$(* (*#&$(* *(&$ #( (*#&*( $&(*#, someone find something I can kill ... ... ... ...
OK, now for the details.
The CN node did indeed do it's work properly.
But that still wasn't good enough.
Every single relay shows our block before the f2pool block, even the CN relay.
Every single relay shows our block sent back to us before the f2pool block, even the CN relay.
... the us relays don't show us sending it to the relay coz the solo pool did that faster.
Our time to process the block was 280ms.
I've shortened all the block hashes below to 4 zeros and 4 hex since that's enough to see which is which.
00003e80 is 410418 BTCC in china - all pools worked on this block
*00002b07* is
410419 our block
-00001f67- is
410419 FUPool
00003007 is 410420 FUPool confirming their block
bjs [2016-05-05 23:35:17.374+00] 00003e80 recv'd, size 989168 with 6717 bytes on the wire
bjs [2016-05-05 23:38:01.073+00] *00002b07* sent, size 985717 with 37638 bytes on the wire
bjs [2016-05-05 23:38:01.209+00] *00002b07* recv'd, size 985693 with 37546 bytes on the wire
bjs [2016-05-05 23:38:02.227+00] -00001f67- recv'd, size 989649 with 935353 bytes on the wire
bjs [2016-05-05 23:44:40.138+00] 00003007 recv'd, size 755724 with 4191 bytes on the wire
jpy [2016-05-05 23:35:17.561+00] 00003e80 recv'd, size 989168 with 6717 bytes on the wire
jpy [2016-05-05 23:38:01.122+00] *00002b07* sent, size 985717 with 37638 bytes on the wire
jpy [2016-05-05 23:38:01.291+00] *00002b07* recv'd, size 985693 with 37546 bytes on the wire
jpy [2016-05-05 23:38:08.164+00] -00001f67- recv'd, size 989649 with 935353 bytes on the wire
jpy [2016-05-05 23:44:40.200+00] 00003007 recv'd, size 755724 with 4191 bytes on the wire
eu [2016-05-05 23:35:17.641+00] 00003e80 recv'd, size 989168 with 6717 bytes on the wire
eu [2016-05-05 23:38:01.118+00] *00002b07* sent, size 985717 with 37638 bytes on the wire
eu [2016-05-05 23:38:01.163+00] *00002b07* recv'd, size 985693 with 37546 bytes on the wire
eu [2016-05-05 23:38:05.960+00] -00001f67- recv'd, size 989649 with 935353 bytes on the wire
eu [2016-05-05 23:44:40.240+00] 00003007 recv'd, size 755724 with 4191 bytes on the wire
us-east [2016-05-05 23:35:17.741+00] 00003e80 recv'd, size 989168 with 6717 bytes on the wire
us-east [2016-05-05 23:38:01.601+00] *00002b07* recv'd, size 985693 with 37546 bytes on the wire
us-east [2016-05-05 23:38:08.310+00] -00001f67- recv'd, size 989649 with 935353 bytes on the wire
us-east [2016-05-05 23:44:40.268+00] 00003007 recv'd, size 755724 with 4191 bytes on the wire
us-west [2016-05-05 23:35:17.741+00] 00003e80 recv'd, size 989168 with 6717 bytes on the wire
us-west [2016-05-05 23:38:01.601+00] *00002b07* recv'd, size 985693 with 37546 bytes on the wire
us-west [2016-05-05 23:38:08.310+00] -00001f67- recv'd, size 989649 with 935353 bytes on the wire
us-west [2016-05-05 23:44:40.268+00] 00003007 recv'd, size 755724 with 4191 bytes on the wire
The winner was of course decided by the next block (00003007), but the problem is: why was FUPool not working on our block?
So basically the cause was one or more of the following:
FUPool is REAL slow getting their blocks to the CN relay.
FUPool works on their own blocks after they see other pool blocks if they are found soon after.
FUPool had most of their 410419 block transactions not in the relay so was really slow getting their block into the relay (compression was almost zero as you can see) Edit: but I don't think this is it coz it was prolly the relay did that coz it already had our block using the transactions.