Yes, as far as I can see, we're getting the right number of blocks overall - it's just we have high variability on the block times. I think the difficultly adjustment might be making that worse.
What I'm leaning towards is taking an average of three time periods - previous 10 blocks, previous 60 blocks, previous 240 blocks, and limiting the adjustment each block to a maximum of 5% up or down.
Are there any other coins that handle this well? I see Freicoin has an advanced diff adjustment - maybe too complex.
I've had a suggestion that we use the MegaCoin 'Gravity Well' -
https://forum.megacoin.co.nz/index.php?topic=893.msg5742#msg5742