I have just posted the security report on
http://boolesrings.org/jvanname/2017/11/09/security-report-for-r5-the-pow-problem-for-nebula. I now have the functions for R5 ready and coded in c++, but I need to get the thing launched (at this point, I am pretty sure that I will need some help from professional cryptocurrency developer in exchange for unobfuscated code access that will allow for quicker mining for problems 1-3, and a short period of pre-mining).
jukKas. I appreciate your interest in the Laver tables. They are truly unique and profound mathematical structures that arise from the highest levels of infinity (and they can be understood and calculated by just about anyone too).
I doubt that there will be any security issue for R5 that cannot be remedied by simply increasing the number of rounds since R5 is a symmetric cryptosystem rather than a public key cryptosystem (symmetric cryptography gains its security through the number of rounds while public key cryptography is another story since the Merkle-Hellman knapsack cryptosystem has been broken overnight and it cannot be improved by increasing the number of rounds.). Each problem in R5 employs many Toffoli gates and Fredkin gates and both the Toffoli gates and the Fredkin gates are universal for reversible computation. Furthermore, any permutation of {0,1}^n can be computed by Toffoli gates with at most 3 ancilla bits.
I have ultimately decided that having a mechanism in R5 that automatically increases the number of rounds in case there is any insecurity of R5 is unnecessary since the security requirements for R5 do not seem too demanding. For R5, recall that the POW is simply to find a 256 bit hash k along with 68 bit string x so that f_i(k#x)<1/d where d is the difficulty. A miner will have little control over k since a miner can only try many different hashes until he finds a good one. A miner can only control the 68 bit string x, and it will be difficult for an attacker to break R5 when he only has control over the 68 bit string x. By the nature of R5, the attacker does not have much room to work with. Therefore, since attackers do not have much room to work with, the functions f_1,...,f_5 only need to be OK randomizing functions in order for R5 to be secure. Most of the work has already been done for me since we already have secure hash functions. In essence, for all i, a one bit change to the input for f_i will result in a completely different output even after going through 1/4 of all rounds.
For option 2, the mechanism to increase the number of rounds will have an additional POW problem which for our purposes we shall call weak R5. Weak R5 will be like strong R5 except that weak R5 will only employ 1/3 as many rounds as strong R5, and weak R5 may employ other features that make it less secure. If weak Problem i is solved too often in proportion to strong Problem i, then the number of rounds in both weak and strong Problem i will automatically increase. Therefore, strong R5 will remain secure since an attacker cannot break strong R5 unless someone is able to break weak R5.
P.S. Since I have posted the original whitepaper on Nebula, I have changed each of the problems R5 slightly in order to increase the level of security.