The problem was that sometimes a block (which includes a work creation transaction) did not propagate quick enough while unconfirmed POW/Bounty submissions were already received from the mempool of different nodes. Since the node did not know the work yet, it assumed that the POW submission is malicious/malformed and blacklisted the node.
Suggested fix for blacklisting issue:
https://github.com/OrdinaryDude/elastic-reference-client/commit/df3177877687b79ef4a63bdb85c55c3afbd9619c
Problem:
This of course opens up a wide surface for mempool flooding attacks with made-up work submissions (where the software can not yet decide if they are malicious or not). NXT suffers from the same problem. We will have to find a solution for that here.
Important:
IF PROBLEMS PERSIST: You need to "rm -rf nxt_text_db/" and redownload the blockchain entirely since forks may have screwed up a lot in the last days.
Pending:
I still suspect one little issue in the block reorganization routine, the "flooding problem" from above as well as block reorganization is the next point on the agenda.
Topic: [ANN][XEL] Elastic Project - The Decentralized Supercomputer - page 264. (Read 450524 times)
September 07, 2016, 09:11:06 AM
September 07, 2016, 07:31:50 AM
Hi,
forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
I think the network is split up big time, I've just forged every block for the last 1-2 hours 
@EK is working on it. I think that even if you delete db and resync sooner or later you'll be on a fork again. We need to wait to see what great mind of @EK came up with.
September 07, 2016, 07:15:01 AM
A restart of the node helps - now the blocks are distributed again ..
regards
regards
September 07, 2016, 07:12:55 AM
I've thrown up a VPS to hopefully stabilize the network a little bit more, as all the others you can access it here http://188.166.162.17:6876/index.html,
September 07, 2016, 06:58:09 AM
Hi,
forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
I think the network is split up big time, I've just forged every block for the last 1-2 hours forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
September 07, 2016, 06:49:17 AM
Hi,
forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
forging works here but I just forged 15+ blocks in a row - Did I cause a fork or are the other forging nodes down ?
Regards
September 07, 2016, 05:27:19 AM
@EK can you send me your bitmessage address? I'll send you passphrase to problem account. Every time I login to it I'm banned (or my node is banning) in 5-100 blocks so you'll have closer look at the problem.
I can't remember it, I would have to start the other machine first.
But I have downloaded your nxt_test_chain, logged in with my account and started blacklisting as well. I think its not connected to a specific account but more to the blockchain in nxt_test_chain itself.
I am inspecting now what is different from a working chain. I'll be back to you soon!
September 07, 2016, 05:24:20 AM
@EK can you send me your bitmessage address? I'll send you passphrase to problem account. Every time I login to it I'm banned (or my node is banning) in 5-100 blocks so you'll have closer look at the problem.
September 07, 2016, 05:23:08 AM
My miner isn't working for two days already (maybe you launched yours?). So it's strange that there are some PoW submissions since I think I was the only one who had working miner.
Well I have just started a miner half an hour ago! This was the time when your node started blacklisting. At the same time there was a 2-block-height fork!
I suspect it must have something with a closed work, that was rolled back and not correctly reopened. Others then submit POW submissions, and your node thinks that the work is still closed while on the (new) fork it should be open. Something like this I think!
September 07, 2016, 05:20:44 AM
Just tell him you participated is the not-yet-officially-existing bug bounty program and the consensus in the thread was 2 BTC per testnet crash ;and what he thinks about it :-)
Regarding your chain state! This is exactly it! This is the problem we have been seeing in the last days! Perfect, I think I will need a hour or two but I think we will identify whats going on soon ...
Regarding your chain state! This is exactly it! This is the problem we have been seeing in the last days! Perfect, I think I will need a hour or two but I think we will identify whats going on soon ...
September 07, 2016, 05:19:27 AM
My miner isn't working for two days already (maybe you launched yours?). So it's strange that there are some PoW submissions since I think I was the only one who had working miner.
September 07, 2016, 05:15:23 AM
Do you have bitmessage? Or should I contact Lannister? I really think you deserve a couple BTC for your hard work and I am sure he will be more than happy to give you some.
I'll set up bitmessage today and contact @Lannister! Thanks.
September 07, 2016, 05:14:15 AM
And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.
Give me a min I'll upload log to some file hosting.
Give me a min I'll upload log to some file hosting.
What would be really cool is your nxt_test_db/ folder. Then we can set up your chain state on a different machine and check whats wrong.
nxt_test_db: http://s000.tinyupload.com/index.php?file_id=00301526502546875780
nxt.log: http://s000.tinyupload.com/index.php?file_id=38360208295476988967
September 07, 2016, 05:08:30 AM
And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.
Give me a min I'll upload log to some file hosting.
Give me a min I'll upload log to some file hosting.
What would be really cool is your nxt_test_db/ folder. Then we can set up your chain state on a different machine and check whats wrong. Maybe it's related to the 2 blocks height fork from a few blocks ago? At least your node does not accept transactions which are accepted by others.
I feeel we are very close. No worries, nxt_test_db/ does only include the public chain, no private stuff.
September 07, 2016, 05:07:09 AM
Welcome back @EK! Hope you had good time in Greece 
You're back and my node isn't blacklisted anymore
But today I set nxt.myAddress to my external IP address and it seems that ban problem went away. Could this be related to the problem? If my node is announcing with different IP address I could be banned after forging too many blocks. What do you think?
Aaaaand it's gone. I'm on a fork again.
You're back and my node isn't blacklisted anymore
Aaaaand it's gone. I'm on a fork again.
Really? Maybe we can reproduce that?
Actually, most importantly, did your node blacklist all others, or did others blacklist your node? And if so ... why?
I think what we are experiencing is the very problem that we try to identify for days.
Do you have bitmessage? Or should I contact Lannister? I really think you deserve a couple BTC for your hard work and I am sure he will be more than happy to give you some.
EDIT: The pull request was pulled
Thanks September 07, 2016, 05:05:51 AM
And finally I have nice exception in my logs! This should tell you a lot more @EK. Looks like it's my node that is banning.
Give me a min I'll upload log to some file hosting.
Also please take a look at my pull request to elastic-miner.
EDIT: My nxt.log http://s000.tinyupload.com/index.php?file_id=38360208295476988967
Give me a min I'll upload log to some file hosting.
Also please take a look at my pull request to elastic-miner.
EDIT: My nxt.log http://s000.tinyupload.com/index.php?file_id=38360208295476988967
September 07, 2016, 05:04:06 AM
hagie, yes forging is based on coins! If you give me your address I can send you some more so your chances will increase.
Hi and thanks,
another problem I encontered is the "Could not verify signature (server side)." I can't register an account and I can't create any work (get the same error).
my address is : XEL-48LM-K27E-ZXWE-FFFF2 and you can check the node under https://elastic.cryptonodes.site
the only changes I made is in the nxt.conf :
Code:
nxt.myAddress=elastic.cryptnodes.site
nxt.myPlatform=linuxDebian8
nxt.apiServerHost=0.0.0.0
nxt.allowedBotHosts=*
nxt.adminPassword=wonttellhere
regards
September 07, 2016, 04:59:22 AM
@EK we miss you
Arrived back home,
started to work (like properly working) now! Let me find out what's exactly happening and then I will prove a fix including a detailed explanation.
Now, it will be 12h daily for elastic again
At the moment, my local node looks fine! Could someone point me into the right direction if there is some issue right now?
If not, I will find the issue anyway, it will just take longer. Will try to stress test the whole thing with a couple machines I think!
Welcome back @EK! Hope you had good time in Greece
You're back and my node isn't blacklisted anymore
Aaaaand it's gone. I'm on a fork again.
September 07, 2016, 04:56:56 AM
hagie, yes forging is based on coins! If you give me your address I can send you some more so your chances will increase.
September 07, 2016, 04:49:55 AM
Looks like fork handling is working :
Received Block, My prevblockid: 8909713533085497795, their prevblockid: -3291626022194019608
Received Block, My prevblockid: 8909713533085497795, their prevblockid: 4224601319020872313
2016-09-07 11:46:06 FINE: BlockchainProcessorImpl$1.downloadBlockchain: Will process a fork of 2 blocks
2016-09-07 11:46:06 FINE: BlockchainProcessorImpl.popOffTo: Rollback from 7432 to 7431
regards
Received Block, My prevblockid: 8909713533085497795, their prevblockid: -3291626022194019608
Received Block, My prevblockid: 8909713533085497795, their prevblockid: 4224601319020872313
2016-09-07 11:46:06 FINE: BlockchainProcessorImpl$1.downloadBlockchain: Will process a fork of 2 blocks
2016-09-07 11:46:06 FINE: BlockchainProcessorImpl.popOffTo: Rollback from 7432 to 7431
regards
Jump to: