You guys are awesome! Sorry for the long response times, but as I told before I am in my vacation and only online when I find an open wifi on the go.
Things will improve once I get back in 2 days. Then, I will be working full time again.
Regarding the testnet, seems like someone again identified an edge case where transactions end up in the mempool being broadcast even though they are invalid. This causes the two public nodes hosted on AWS to be blacklisted:
nxt.NxtException$NotValidException: Cannot cancel already cancelled or finished work
at nxt.TransactionType$WorkControl$3.validateAttachment(TransactionType.java:532)
at nxt.Attachment$AbstractAttachment.validate(Attachment.java:34)
at nxt.TransactionImpl.validate(TransactionImpl.java:835)
at nxt.TransactionProcessorImpl.processPeerTransactions(TransactionProcessorImpl.java:478)
at nxt.TransactionProcessorImpl.access$8(TransactionProcessorImpl.java:448)
at nxt.TransactionProcessorImpl$5.run(TransactionProcessorImpl.java:217)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-04 18:37:00 FINE: TransactionProcessorImpl.processPeerTransactions: Invalid transaction from peer: {"senderPublicKey":"ae0df06215caa34bf680a84b24330554d0af092b925b5bcf8704ac31cd489019","attachment":{"id":"1309865111079249537","version.WorkIdentifierCancellationRequest":1},"subtype":5,"amountNQT":0,"signature":"f7e891e4e4965e765a1b11e31ac289770b2f5f3994346c8bc0701a500aee180de53e87bdb782e0846c0477c86b6341b6d8627b079c1946efdcb3e387fffacca1","feeNQT":0,"ecBlockHeight":3941,"type":3,"deadline":1440,"version":1,"timestamp":21374693,"ecBlockId":"11763907846096637330"}
nxt.NxtException$NotValidException: Cannot cancel already cancelled or finished work
at nxt.TransactionType$WorkControl$3.validateAttachment(TransactionType.java:532)
at nxt.Attachment$AbstractAttachment.validate(Attachment.java:34)
at nxt.TransactionImpl.validate(TransactionImpl.java:835)
at nxt.TransactionProcessorImpl.processPeerTransactions(TransactionProcessorImpl.java:478)
at nxt.TransactionProcessorImpl.access$8(TransactionProcessorImpl.java:448)
at nxt.TransactionProcessorImpl$5.run(TransactionProcessorImpl.java:217)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-09-04 18:37:00 FINE: PeerImpl.blacklist: Blacklisting 52.57.31.137 because of: nxt.NxtException$NotValidException: Peer sends invalid transactions
If you set your "blacklisting time" to 1s or 0s or so, in net-default.conf, then you will be able to stay connected with these nodes.
I repeat, if someone can identify and fix this mistake before I do, I will give him 2 BTC of my personal savings. Thanks for stress testing! The testnet was the first test of Elastic and it was expected to have bugs. The more we identify the more stable we will eventually become! Thanks for everyone.
My feeling tells me, that we need a sophisticated mempool cleanup routine, which cleans up all bounties/work closures/POW submissions that are still pending when a work is being closed, and not broadcast regardless of the work's state!
... I will do some code reading today as well, and check out what kind of transactions are being broadcast by the AWS nodes and why!
when you have some time can you please take a look at problem i have since the start? i cant register account name and can't submit new work because i am getting this error:
lol 
