difficulty for CPU block should calculates based ONLY on diff of previous CPU block.
difficulty for GPU blocks should calculates based ONLY on diff of previous GPU block.
I never seen before such... hm, weird implementation of multi-algo chain. Each kind of blocks (CPU, GPU, MN) should work on independent chain otherwise there will be mess. Or at least difficulty should calculates for each kind of blocks INDEPENDENTLY. Check out how other multi-algo altcoins works
For example, Unitus coin.
This coin contains five different hashing algorithms. Each of them works on independend chain with independent difficulty, with block target time 5 minutes, which in total results in 1 minute block target time for whole Unitus chain/network.

