Try to update your system and Java:
Java is up to date
ii oracle-java9-installer 9.0.1-1~webupd8~0 amd64 Oracle Java(TM) Development Kit (JDK) 9
ii oracle-java9-set-default 9.0.1-1~webupd8~0 amd64 Set Oracle JDK 9 as default Java
09:55:57.804 [INFO] PeerClient Starting IP refresh thread
09:55:57.937 [INFO] NodeManager Node manager started
09:55:58.141 [INFO] PeerServer Starting peer server: address = 0.0.0.0:5161
09:55:58.202 [INFO] SemuxBFT Consensus started
09:55:58.204 [INFO] SemuxBFT Entered new_height: height = 1, # validators = 4
09:55:58.239 [INFO] SemuxP2pHandler Received DISCONNECT msg, reason = DUPLICATE_PEER_ID
09:55:58.608 [INFO] SemuxP2pHandler Received DISCONNECT msg, reason = INVALID_HANDSHAKE
09:55:59.363 [INFO] SemuxP2pHandler Received DISCONNECT msg, reason = INVALID_HANDSHAKE
09:56:00.097 [INFO] SemuxP2pHandler Received DISCONNECT msg, reason = INVALID_HANDSHAKE
09:56:00.587 [INFO] SemuxP2pHandler Received DISCONNECT msg, reason = INVALID_HANDSHAKE
As I'm saying RC5 is broken for me. RC3 works fine. I can go to my rc3 dir, start it and it works. When I stop it and go to RC5 dir, start it, then it's stuck with the above error.
Hm not sure how it works with Java9...
Works fine with version 8.
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
I think devs will have take a look at this.