Here is some explanation about that bug that happened and how I understood what was the problem...
Q: Why was I killed, I had great PC / VPS, 16GB of ram and 6 core CPU?
A: The whole network stopped, no one killed anyone, the whole network "killed itself" in a way. I understand that, that kind of powerful node is one of those who participated in committee that made those big blocks, but your results were not in the first but in the second block that was made late.
If you participate in the committee for making blocks, approximately, I'm not sure how big that committee is, that is, how many nodes it consists of. All participating nodes in committee must process all transactions within 20sec of the block's duration. What happened was, too many transactions from too many participants.
3420 nodes performed validation. In the previous successful validation, 2300 nodes passed, which means that the network worked with 47.8% more nodes than last time, which means that the load is so much higher, that is. number of transactions. The block reached a maximum size, 1MB, other transactions were waiting in the mempool to be put in next block. Then the network tried to process that block in order for those transactions to become valid (ie. our answers on validation).
Then, a committee of random nodes is made.
And every node in that committee must process that, so far the largest block we have ever had.
The nodes in the committee were changing all the time trying to process that huge block. But were not able and network made empty blocks.
Until the network came up with stronger nodes (like VPS's and home PC's that had multicores and higher RAM), and they managed to process that block, f**in near the end of validation. Those transactions(validation answers) processed in that block have passed validation, because those transactions are on the processed block and are written to the blockchain.
The next committee is again a failure and an empty block.
After that, when it was already late, the second block was processed (also almost full 1MB block), which contained the other half of our answers/transactions.
Now... The developers will make sure that validation cannot be completed while there are still unprocessed validation transactions in mempool. The worst that can now happen is to wait for the validation result for few blocks longer. Additionally, since it will reduce the maximum block size to 300KB from 1MB, that means it will be 3x easier for all nodes to process blocks.
Q: So what now? Will they kick us out who have 1cpu/1gb or?
A: No, these nodes will work now as well, because they will be 3x less demanding blocks.
Q: And what will happen for the next validation when we have 3000 nodes again?
A: Since there will be less demanding blocks, all nodes should be able to process them and that's only... 10 blocks.
If we calculate that in the last validation we needed 2 blocks of 1MB for all answers. This means we need 2MB of data for all responses to be written. Let's think that in the next validation we will have not 2MB as we had now, let's consider 3MB. Since the block will be 300KB, this means that we will need to process 10 blocks in order for all transactions (all validation responses) to be successfully written on blockchain.
Since the block lasts 20 seconds and we need 10 blocks, that is, we need 3.5 minutes.
And we know that a long session lasts 30 minutes, which means we have 8x more time than we need.
With this principle, we can roughly calculate the following theoretical maximum of the network before validation sharding must be implemented.
Q: And now the question, why didn't they remember that before? But when did it break?
A: It broke because we reached the maximum with testing. If no one had tested, no one would have ever found out about this what we found out. They themselves said they did not expect this.
I guess the problem was also VPSs that have 1Core/1GB. When the 1GB of memory is full, then it switches to SWAP which is actually a slow hard drive. Then for that, the processor needs much more time to allocate memory to RAM and SWAP. And that leads to the impossibility to do everything properly, that is, does not have enough CPU power to process that large block of 1MB transactions.
I also figured out what our maximum transactions were. There yeah were 3966 transactions in that first big block, let's say 4000.
4000tx / 20sec block time, so 200 transactions per second, what is the speed tx/s in ETH?
On ETH the max is 15tx/s.
And do you know what ETH is working on? What are those nodes? They are certainly not a $5 VPS. They run on powerful Amazon instances.
And ETH has 7800 nodes.
Not even Idena can work on Tetris hardware.
After reducing the block size 3x, we will again be at 4x faster transaction speed than ETH.
At about 66 tx/s, even with tetris computers.
My recommendation is to temporarily upgrade each VPS for validation to 2GB.
Digital Ocean is great for that, they support temporary component changes. One can even make a script that Digital Ocean itself does upscale 2GB for validation and downscale 1GB afterwards, automatically via API. I remember that Anthrophobia and Crackowich told me that they were doing it. Validation means testing the theoretical maximum of the network each time.
Node operation during mining, while there are barely a couple of transactions in each block, is nothing compared with node operation during validation.
And this is what we deal in the network, this member is complaining that syncing is slow, and has 5/0.2 internet speed...
