Author

Topic: BiblePay - New Coin Launch - Official Thread - page 127. (Read 119833 times)

hero member
Activity: 714
Merit: 500
"blocks": 2723,
  "currentblocksize": 1000,
  "currentblocktx": 0,
  "difficulty": 0.007120199528557721,
  "errors": "",
  "genproclimit": 24,
  "network_khashps": 5863.713043883206,
  "hashps": 281426.3337108216,
  "minerstarttime": "08-12-2017 07:26:08",
  "pooledtx": 0,
  "testnet": false,
  "chain": "main",
  "biblepay-generate": true,
  "poolinfo1": "",
  "poolinfo2": "",
  "poolinfo3": "",
  "poolmining": false


Am i mine right chain?

Current block is 2824, so you're a ways behind. Are you synced?

Yes im synced, why im behind? But i find 2 blocks with this? Am i forked?

Dev never mentioned about new fork, your case is really weird. Now at 2825 block.
full member
Activity: 154
Merit: 100
"blocks": 2723,
  "currentblocksize": 1000,
  "currentblocktx": 0,
  "difficulty": 0.007120199528557721,
  "errors": "",
  "genproclimit": 24,
  "network_khashps": 5863.713043883206,
  "hashps": 281426.3337108216,
  "minerstarttime": "08-12-2017 07:26:08",
  "pooledtx": 0,
  "testnet": false,
  "chain": "main",
  "biblepay-generate": true,
  "poolinfo1": "",
  "poolinfo2": "",
  "poolinfo3": "",
  "poolmining": false


Am i mine right chain?

Current block is 2824, so you're a ways behind. Are you synced?

Yes im synced, why im behind? But i find 2 blocks with this? Am i forked?
member
Activity: 70
Merit: 10
"blocks": 2723,
  "currentblocksize": 1000,
  "currentblocktx": 0,
  "difficulty": 0.007120199528557721,
  "errors": "",
  "genproclimit": 24,
  "network_khashps": 5863.713043883206,
  "hashps": 281426.3337108216,
  "minerstarttime": "08-12-2017 07:26:08",
  "pooledtx": 0,
  "testnet": false,
  "chain": "main",
  "biblepay-generate": true,
  "poolinfo1": "",
  "poolinfo2": "",
  "poolinfo3": "",
  "poolmining": false


Am i mine right chain?

Current block is 2824, so you're a ways behind. Are you synced?
full member
Activity: 154
Merit: 100
"blocks": 2723,
  "currentblocksize": 1000,
  "currentblocktx": 0,
  "difficulty": 0.007120199528557721,
  "errors": "",
  "genproclimit": 24,
  "network_khashps": 5863.713043883206,
  "hashps": 281426.3337108216,
  "minerstarttime": "08-12-2017 07:26:08",
  "pooledtx": 0,
  "testnet": false,
  "chain": "main",
  "biblepay-generate": true,
  "poolinfo1": "",
  "poolinfo2": "",
  "poolinfo3": "",
  "poolmining": false


Am i mine right chain?
member
Activity: 70
Merit: 10
Oh, by the way, I temporarily decreased the maximum number of connections to the biblepay.inspect.network node. When the number of connections got too high, biblepayd would eventually die which would stop the block explorer from updating since it relies on that node to get info on the blockchain. The error log wouldn't be very informative, a lot of network timeouts over a period of time before silently crashing:

Code:
2017-08-11 16:03:35 socket recv error Connection timed out (110)
2017-08-11 16:06:10 socket recv error Connection timed out (110)
2017-08-11 16:08:03 socket recv error Connection reset by peer (104)
2017-08-11 16:09:25 socket recv error Connection reset by peer (104)
2017-08-11 16:09:27 socket recv error Connection reset by peer (104)
2017-08-11 16:09:48 socket recv error Connection timed out (110)
2017-08-11 16:20:53 socket recv error Connection timed out (110)
2017-08-11 16:28:15 socket recv error Connection timed out (110)
2017-08-11 16:33:19 socket recv error Connection timed out (110)
2017-08-11 16:37:25 socket recv error Connection timed out (110)
2017-08-11 16:39:42 socket recv error Connection timed out (110)
2017-08-11 16:44:23 socket recv error Connection timed out (110)
2017-08-11 16:48:12 socket recv error Connection reset by peer (104)
2017-08-11 16:48:23 socket recv error Connection timed out (110)
2017-08-11 16:51:25 socket recv error Connection timed out (110)
2017-08-11 16:54:39 socket recv error Connection reset by peer (104)
2017-08-11 16:54:54 socket recv error Connection reset by peer (104)
2017-08-11 16:59:11 socket recv error Connection timed out (110)

Does node.biblepay.org ever have the same issue? I'm running 1.0.1.9. It's possible that I just don't have the server configured properly, I never really set up a node for 300+ connections before.

Yeah, but not as many as id think comparing to nethash. I am wondering if nethash is displayed wrong? As we can be 500+ miners that dont add up to nethash unless it is alot of rasperry pi or people with old miner / 1 thread

I think the 500+ number is mostly coming from the number of connections to the biblepay.org node and the node I mentioned above. The thing about the number of connections is that it doesn't necessarily tell you how many of those are actively mining. If you have the wallet open, you're running a node whether you're mining or not. I also wouldn't be surprised if there are a lot of people just running 1 or 2 threads so it doesn't slow down whatever else they're doing, CPU mining tends to attract a lot of casual miners. There's probably not really any good way to put an actual number on this.
full member
Activity: 462
Merit: 118
Did anyone find a block with the updated miner? Mining with 1/100 of nethash and haven't seen a block in 600 blocks.

Yeah, but not as many as id think comparing to nethash. I am wondering if nethash is displayed wrong? As we can be 500+ miners that dont add up to nethash unless it is alot of rasperry pi or people with old miner / 1 thread

And yeah, the price ppl are selling for is just funny. Hope we get to a bigger exchange soon Smiley and when updates are ready for the coin i guess the current buyers are the winners Smiley
sr. member
Activity: 546
Merit: 257
Have you found the Yellow Sign?
Did anyone find a block with the updated miner? Mining with 1/100 of nethash and haven't seen a block in 600 blocks.

Nothing yet. I used to get blocks every couple days.

At this point it's almost cheaper just to buy the BBP people are deciding to dump for basically nothing.
full member
Activity: 462
Merit: 103
Mining the next block starts as soon as the current block is found. Block 2782 was found at 08:39:48. It immediately creates the next block, then starts hashing it. A hash that satisfies PoBh was then found at 08:59:25.

You might be confused about the difference between creating the block and mining it. Really basically, a block is a collection of transactions and header information. For a block to be mined, it has to produce a hash that meets the difficulty requirements when ran through the mining algorithm. So, the block needs to exist before you can start hashing it. Obviously, the same data will always produce the same hash, so there' a value in the header called the nonce that starts at 0 and is incremented after each hash attempt until a solution is found.
Thanks for the explanation, I didn't know blocks were created before being mined.

18 days is not a long enough sample time to make the assumption that the parameters are failing.
I believe there was a discussion early on in litecoins thread that it took about a year for the diff retarget to average out.

However, I do want to ensure we start retargeting our diff every single block (instead of 4 as it is currently), and decrease our stuck blocks down to 30 mins during the next mandatory.  We have quite a few slow blocks, and that will drag this short sample period up significantly.

Yeah, I was thinking about that a little afterwards, and for most of that time the number of miners were so few that one or two people dropping in or out of the pool would be a significant portion of the network hash which would mess up the difficulty. So, trying to draw a conclusion from that might not actually be as accurate as you'd think just given the sample size. I might try running some daily averages and see how things progress. The network hash is increasing pretty rapidly, we've gone up like 20% in a day.
Hmm, but why would people dropping in or out be so important if the difficulty changes every 4 blocks? It's not every block (yet), but still, if the diff readjusts ~35 times a day, isn't it enough to not allow for any wild fluctuations?

Especially considering that the trend still continues - yesterday there was only 135 blocks, but nethash is much bigger and more stable now. And it's never more than 205 blocks per day, not even more than 155 per day. It's always less. One would think it would fluctuate to both directions if the early instability was an issue.
legendary
Activity: 1453
Merit: 1030
Did anyone find a block with the updated miner? Mining with 1/100 of nethash and haven't seen a block in 600 blocks.
member
Activity: 70
Merit: 10
18 days is not a long enough sample time to make the assumption that the parameters are failing.
I believe there was a discussion early on in litecoins thread that it took about a year for the diff retarget to average out.

However, I do want to ensure we start retargeting our diff every single block (instead of 4 as it is currently), and decrease our stuck blocks down to 30 mins during the next mandatory.  We have quite a few slow blocks, and that will drag this short sample period up significantly.

Yeah, I was thinking about that a little afterwards, and for most of that time the number of miners were so few that one or two people dropping in or out of the pool would be a significant portion of the network hash which would mess up the difficulty. So, trying to draw a conclusion from that might not actually be as accurate as you'd think just given the sample size. I might try running some daily averages and see how things progress. The network hash is increasing pretty rapidly, we've gone up like 20% in a day.
member
Activity: 70
Merit: 10
So I don't understand what does block 2783 have to do with the error which appeared in the yellow timeline, because that block was found in the green timeline 20 minutes later. Unless you mean it was supposed to be found by the miner back then, but the error always appears in the same second as the UpdateTip for the previous block.

Mining the next block starts as soon as the current block is found. Block 2782 was found at 08:39:48. It immediately creates the next block, then starts hashing it. A hash that satisfies PoBh was then found at 08:59:25.

You might be confused about the difference between creating the block and mining it. Really basically, a block is a collection of transactions and header information. For a block to be mined, it has to produce a hash that meets the difficulty requirements when ran through the mining algorithm. So, the block needs to exist before you can start hashing it. Obviously, the same data will always produce the same hash, so there' a value in the header called the nonce that starts at 0 and is incremented after each hash attempt until a solution is found.
full member
Activity: 1176
Merit: 215
Jesus is the King of Kings and Lord of Lords
Correct, but in reality there are about 144 blocks every day which is 10 minutes mean block time.

Edit: Here's happy_merchant's post if you missed it:

@bible_pay: Could you maybe explain the fact that only ~70% of blocks are found each day?

26928 minutes / 2675 blocks = 10.0665 minutes/block
18 days is not a long enough sample time to make the assumption that the parameters are failing.
I believe there was a discussion early on in litecoins thread that it took about a year for the diff retarget to average out.

However, I do want to ensure we start retargeting our diff every single block (instead of 4 as it is currently), and decrease our stuck blocks down to 30 mins during the next mandatory.  We have quite a few slow blocks, and that will drag this short sample period up significantly.
full member
Activity: 1176
Merit: 215
Jesus is the King of Kings and Lord of Lords
I'm not sure if I understood that correctly, but here I pasted the same log for block 2782, only this time with the surrounding blocks so you can see more info. I highlighted the time stamps which I think are related to the block 2782 in yellow, and the ones which are related to the block 2783 in green.
Quote
2017-08-12 08:30:45 UpdateTip: new best=000001631b73838c0fbb1c77726900fca1326f7ed687722ba68664da2b20e66e  height=2781  log2_work=32.422993  tx=3307  date=2017-08-12 08:30:33 progress=0.999996  cache=0.0MiB(116tx)
2017-08-12 08:30:45 ProcessNewBlock : ACCEPTED
2017-08-12 08:30:51 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done
2017-08-12 08:39:48 UpdateTip: new best=00000024a3327afa5e3ff620607e48841ec35c6a92a554a132e0dbb616b5526d  height=2782  log2_work=32.427115  tx=3308  date=2017-08-12 08:39:47 progress=1.000000  cache=0.0MiB(117tx)
2017-08-12 08:39:48

** TestBlockValidity FAILED - pindexNew->pprev != chainActive.Tip() (assert(pindexNew->pprev == chainActive.Tip())); **

2017-08-12 08:39:48

BiblepayMiner -- runtime error: CreateNewBlock: TestBlockValidity failed:  (code 0)
2017-08-12 08:39:48 ProcessNewBlock : ACCEPTED
2017-08-12 08:39:55 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done
2017-08-12 08:59:25 UpdateTip: new best=000000018734cbb6422e82aeaa2f52162760f95ea4d485a2a5c8de3a46bc47b9  height=2783  log2_work=32.43188  tx=3309  date=2017-08-12 08:59:25 progress=1.000000  cache=0.0MiB(118tx)
2017-08-12 08:59:25 ProcessNewBlock : ACCEPTED
2017-08-12 08:59:31 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done
2017-08-12 09:57:59 UpdateTip: new best=00000156f9db92abe88a4de4462004df12893c6b4718e8daca291ef06e765eb1  height=2784  log2_work=32.434599  tx=3311  date=2017-08-12 09:57:59 progress=1.000000  cache=0.0MiB(7tx)
2017-08-12 09:57:59 ProcessNewBlock : ACCEPTED
2017-08-12 09:58:05 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done
So I don't understand what does block 2783 have to do with the error which appeared in the yellow timeline, because that block was found in the green timeline 20 minutes later. Unless you mean it was supposed to be found by the miner back then, but the error always appears in the same second as the UpdateTip for the previous block.



Hi Inblue,

Looking at 2782 and 2783 and the timeline, it is OK if 2783 was actually solved 20 minutes later, what appears to be happening is this (I can say this based on the TestBlockValidity context):

2782 is accepted at 2017-08-12 08:30:45
2783 is accepted at 2017-08-12 08:59:25

In between these timestamps, at 08:39 AM, One of your mining threads throws a ** TestBlockValidity FAILED.

So you are alluding to the possibility that the miner could not create a new block, but yet the tip had no reason to be updated as a block didnt arrive until 20 minutes later.  Its possible createnewblock is affected by many competing threads,
 possibly a mutex issue causing a missing pindexprev pointer.  We can add more logging in createnewblock and try to find out why occasionally, the miner can not create a new block, and that results in testblockvalidy failing.  It looks like it is unrelated to network activity, orphans, reorganizations, and only the actual activeChain.tip, because the tip was not changing on the node.  So it really appears the createnewblock is sometimes in a blue moon returning a new block with a missing pindexprev pointer, or something to that effect.  It might be revealing for us to put in a little busy wait loop right when that occurs for 7 seconds and log info about the chainactive.tip before throwing the error, just to see if the node recovers its tip by itself.
I will add that to my todo list.


full member
Activity: 462
Merit: 103
Correct, but in reality there are about 144 blocks every day which is 10 minutes mean block time.

Edit: Here's happy_merchant's post if you missed it:

@bible_pay: Could you maybe explain the fact that only ~70% of blocks are found each day?

26928 minutes / 2675 blocks = 10.0665 minutes/block
full member
Activity: 1176
Merit: 215
Jesus is the King of Kings and Lord of Lords
Quote
I also found the 205 blocks per day parameters in the code, so I am very worried about this, there's definitely something strange here and I hope it's not very complex to fix.

Real quick on the 205 blocks per day, nothing to be worried about that is correct:
60/7*24
 205.7

HH/7Min Blocks * 24 Hrs per day = 205.7.  The integer is fine, its just an input for diff retargets, and lookback functions such as the gather prayers per week, etc.  Nothing looks awry.

I am still checking the rest of your post Smiley
full member
Activity: 462
Merit: 103
So, (assuming I'm reading this right) the error isn't related to block 2782, but rather the following block. Block 2782 was received by the network and accepted by your node. Your node then calls CreateNewBlock where it assembles the transactions and headers to create block 2783 before sending the block on to be hashed (which is the actual mining process).
I'm not sure if I understood that correctly, but here I pasted the same log for block 2782, only this time with the surrounding blocks so you can see more info. I highlighted the time stamps which I think are related to the block 2782 in yellow, and the ones which are related to the block 2783 in green.
Oh, and looking through some of my really old logs, this is what an error looks like when a peer sends you an invalid block:
Code:
2017-08-07 09:09:28 ERROR: CheckProofOfWork(): BibleHash does not meet POW level
2017-08-07 09:09:28 ERROR: CheckBlockHeader(): proof of work failed
2017-08-07 09:09:28 Misbehaving: 97.99.69.33:40000 (0 -> 50)
2017-08-07 09:09:28 ERROR: invalid header received 0000024a49a386eee621d31124cf1dea0131b0b0233ec168f8dbcc8ea22cca0e
2017-08-07 09:09:28 ProcessMessages(headers, 82 bytes) FAILED peer=1
Wow, I guess 5 days is really old in the crypto world. Grin

So this means the TestBlockValidity error is not caused by a peer sending an invalid block?

Also, I looked through the chain parameters and they appear to be configured correctly for a 7 minute block time, so whatever is affecting that is more complex than just a configuration error.
I also found the 205 blocks per day parameters in the code, so I am very worried about this, there's definitely something strange here and I hope it's not very complex to fix.



OK, bible_pay answered while I was making this post.

This part of it is absolutely normal and as long as it is happening about once every 4 hours or less, no harm is being done and the miner is not slowing down.
Somewhere I found two of these errors for the two blocks right next to each other and somewhere I found it to occur 3 times in about half an hour. But majority of the time it's at least a few hours apart.

The wallet Is set up with the correct parameters, and handling this condition, but
The main stress I want to make on this is that we ensure these things:
1. If the wallet is set up with 10 mining threads to begin with, each thread is handling that particular error and still mining at the end of the day. (Check this).
2.  The wallet does not crash.
3.  The HashPs of the Box itself should bear out a similar value as when the day started.

If #3 especially is true, then everything is fine.  I would suggest grabbing a baseline hashps from the box 5 min after start, and then checking on it 24 hours later.  If the hashps is the same, then dont worry about the error as long as its very rare in the log.
For me all three are check. I find that hash rate is the same after many hours, maybe even 1% higher than in the first few minutes.
full member
Activity: 1176
Merit: 215
Jesus is the King of Kings and Lord of Lords
Quote

 ** TestBlockValidity FAILED - pindexNew->pprev != chainActive.Tip() (assert(pindexNew->pprev == chainActive.Tip())); **


Regarding mining in general and seeing this error in the log:
First, it is normal for the miner to be stuck in a busy-wait loop busy hashing the current block, while the chainActive.Tip() moves on to a new block, and occasionally, for some reason in AcceptBlock it is possible right at a microsecond instant that the main wallet thread is assigning the pindexNew->pprev to something new, pindexNew->pprev to the miner is NULL, and this results in the miner throwing an error and having to go start over and CreateNewBlock.

This part of it is absolutely normal and as long as it is happening about once every 4 hours or less, no harm is being done and the miner is not slowing down.

The wallet Is set up with the correct parameters, and handling this condition, but
The main stress I want to make on this is that we ensure these things:
1. If the wallet is set up with 10 mining threads to begin with, each thread is handling that particular error and still mining at the end of the day. (Check this).
2.  The wallet does not crash.
3.  The HashPs of the Box itself should bear out a similar value as when the day started.


If #3 especially is true, then everything is fine.  I would suggest grabbing a baseline hashps from the box 5 min after start, and then checking on it 24 hours later.  If the hashps is the same, then dont worry about the error as long as its very rare in the log.

One additional note, if you turn on too many debug switches, the box gets busy just writing the Logs!  So after we are stable we need to focus on not logging everything in non-debug mode as that can slow down the miner.

Thanks.

member
Activity: 70
Merit: 10
Quote
2017-08-12 08:39:48 UpdateTip: new best=00000024a3327afa5e3ff620607e48841ec35c6a92a554a132e0dbb616b5526d  height=2782  log2_work=32.427115  tx=3308  date=2017-08-12 08:39:47 progress=1.000000  cache=0.0MiB(117tx)
2017-08-12 08:39:48  

 ** TestBlockValidity FAILED - pindexNew->pprev != chainActive.Tip() (assert(pindexNew->pprev == chainActive.Tip())); **

2017-08-12 08:39:48

BiblepayMiner -- runtime error: CreateNewBlock: TestBlockValidity failed:  (code 0)
2017-08-12 08:39:48 ProcessNewBlock : ACCEPTED
2017-08-12 08:39:55 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done

So, (assuming I'm reading this right) the error isn't related to block 2782, but rather the following block. Block 2782 was received by the network and accepted by your node. Your node then calls CreateNewBlock where it assembles the transactions and headers to create block 2783 before sending the block on to be hashed (which is the actual mining process).

At the top of the CreateNewBlock method, pindexPrev is defined as a pointer to the block currently at the top of the chain (2782 in this case):
Code:
CBlockIndex* pindexPrev = chainActive.Tip();
That's referenced throughout the block creation to get information necessary for creating the new block. Once the block is finished being created, TestBlockValidity is called with pindexPrev as one of the arguments:
Code:
if (!TestBlockValidity(state, chainparams, *pblock, pindexPrev, false, false)) {
        throw std::runtime_error(strprintf("%s: TestBlockValidity failed: %s", __func__, FormatStateMessage(state)));
}
and in that method it's failing this test:
Code:
if (!(pindexPrev && pindexPrev == chainActive.Tip()))
{
LogPrintf(" \r\n ** TestBlockValidity FAILED - pindexNew->pprev != chainActive.Tip() (assert(pindexNew->pprev == chainActive.Tip())); ** \r\n");
return false;
}
then returns and throws the runtime error seen in the error log.

pindexPrev is never reassigned, so in order to fail the test pindexPrev == chainActive.Tip() that means that the value returned by chainActive.Tip() changed. I don't see anything in CreateNewBlock() that looks like it's changing the blockchain, so it's probably caused by another thread. That's all I can gather from reading the code, finding out more would require actual debugging.

I'm not sure where the thrown exception gets handled, so I don't know how the node responds to this. The entire thing doesn't crash so something takes care of it, but I don't know if it attempts to build a new block or what. Mining doesn't seem to just stop since blocks continue to be mined after the error. I'm also not sure how much of this is Biblepay-specific code and how much is carried over from Dash.

Oh, and looking through some of my really old logs, this is what an error looks like when a peer sends you an invalid block:
Code:
2017-08-07 09:09:28 ERROR: CheckProofOfWork(): BibleHash does not meet POW level
2017-08-07 09:09:28 ERROR: CheckBlockHeader(): proof of work failed
2017-08-07 09:09:28 Misbehaving: 97.99.69.33:40000 (0 -> 50)
2017-08-07 09:09:28 ERROR: invalid header received 0000024a49a386eee621d31124cf1dea0131b0b0233ec168f8dbcc8ea22cca0e
2017-08-07 09:09:28 ProcessMessages(headers, 82 bytes) FAILED peer=1

Also, I looked through the chain parameters and they appear to be configured correctly for a 7 minute block time, so whatever is affecting that is more complex than just a configuration error.
full member
Activity: 462
Merit: 103
Unfortunately, I don't think that necessarily means much. A block your node discovers is only broadcast to peers that you're directly connected to. I think that defaults to 8 connections? So, say a node sends out a bad block to the 8 peers it's connected to (that node might be mining on a fork or something). Unless those 8 peers are mining on the same fork, they would reject the block and it wouldn't be propagated across the network. So, unless multiple of your machines shared that same peer, the rejected block would be unlikely to show up more than once.
Good thinking. Yes, I have exactly 8 connections on all instances. I just checked your theory and between the computers only 1 peer is consistent and always at the top of the list and that's 97.99.69.33:40000 (node.biblepay.org). Aside from that one, two computers generally share between themselves 1 or 2 more peers. Other 5-6 peers are different.

I'll try looking through the source and see if I can figure anything out about that error since I get it a lot too, but I can't really promise much.
Maybe here would be a good place to start.

since I get it a lot too
That reminded me - I wanted to ask you if on your hash rate you get about the same number of those errors or proportionally to the hash rate? Per one 140k computer, I got between 10 and 16 of those for 24 hours of 2017-08-11. Actually now that I think of it, it would be way too much to find that amount of blocks for that hash rate. So it's probably something else.
member
Activity: 70
Merit: 10
One very important thing to note is that the invalid block error on some block (for example block 2782 in the above paste) only appears on that machine. On all 6 others it's normal, like this:
Quote
2017-08-12 08:39:49 UpdateTip: new best=00000024a3327afa5e3ff620607e48841ec35c6a92a554a132e0dbb616b5526d  height=2782  log2_work=32.427115  tx=3308  date=2017-08-12 08:39:47 progress=0.999999  cache=0.0MiB(233tx)
2017-08-12 08:39:49 ProcessNewBlock : ACCEPTED
2017-08-12 08:39:55 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done
2017-08-12 08:59:27 UpdateTip: new best=000000018734cbb6422e82aeaa2f52162760f95ea4d485a2a5c8de3a46bc47b9  height=2783  log2_work=32.43188  tx=3309  date=2017-08-12 08:59:25 progress=0.999999  cache=0.0MiB(234tx)
2017-08-12 08:59:27 ProcessNewBlock : ACCEPTED
2017-08-12 08:59:33 CMasternodeSync::IsBlockchainSynced -- found enough peers on the same height as we are, done

Then some other block will be invalid again only on one machine (6 of them are identical btw), so that leads me to believe those are found blocks which are invalid for some reason.

Unfortunately, I don't think that necessarily means much. A block your node discovers is only broadcast to peers that you're directly connected to. I think that defaults to 8 connections? So, say a node sends out a bad block to the 8 peers it's connected to (that node might be mining on a fork or something). Unless those 8 peers are mining on the same fork, they would reject the block and it wouldn't be propagated across the network. So, unless multiple of your machines shared that same peer, the rejected block would be unlikely to show up more than once.

I'm completely assuming here that rejected blocks sent from the network show up in the debug log, by the way. I have no idea if they do or not, but I'd think they would.

That doesn't actually seem to be a rejected block, though. The hash matches up with the blockchain. I'll try looking through the source and see if I can figure anything out about that error since I get it a lot too, but I can't really promise much.
Jump to: