Alright, I will let you know!
I was actually wrong and testnet crashes for me every time the first time (without any testnet3 folder). I have to try to run it a second time to have a chance for it to work.
One thing I noticed is that when running for the first time, it would start generating blocks very quickly without trying to sync to anyone else first? Is that normal?
I think it could actually explain one of my issues as I would start generating blocks very quickly and it be on heigh 40 in a matter of a few seconds.
2017-11-14 16:15:07 init message: Loading wallet... (101.00 %)
2017-11-14 16:15:07 CBlock(hash=6ff4f6a30b254ef8facae6a621ec0ea6853fcb115c320aefa2a2a5c6feb4bfe3, ver=536870912, hashPrevBlock=122f423f0912850a871c58f1533dd80be62154bb0c56dfb8cb9ae2b957d1ac10, hashMerkleRoot=25d601002eb50fb6475128a7bdd8986b159e2b354c84346d43d1d82fc6101d97, nTime=1510676107, nBits=207fffff, nNonce=0, vtx=1)
CTransaction(hash=25d601002e, ver=1, vin.size=1, vout.size=1, nLockTime=0)
CTxIn(COutPoint(0000000000000000000000000000000000000000000000000000000000000000, 4294967295), coinbase 510101)
CTxOut(nValue=20000.00000000, scriptPubKey=2102ababcffd56488a890ebb29eb4f)
2017-11-14 16:15:07
ProcessBlockFound::Generated 20000.00
2017-11-14 16:15:07 Pre-allocating up to position 0x100000 in rev00000.dat
2017-11-14 16:15:07 UpdateTip: new best=6ff4f6a30b254ef8facae6a621ec0ea6853fcb115c320aefa2a2a5c6feb4bfe3 height=1 log2_work=2 tx=2 date=2017-11-14 16:15:07 progress=0.000002 cache=0.0MiB(1tx)
2017-11-14 16:15:07 AddToWallet 25d601002eb50fb6475128a7bdd8986b159e2b354c84346d43d1d82fc6101d97 new
2017-11-14 16:15:07 ProcessNewBlock : ACCEPTED
2017-11-14 16:15:07 keypool keep 2
2017-11-14 16:15:07 CBlock(hash=a1c0bdae133d9acac4f3e8fc453999b4649698b3069ae3fa130aaae531e88fd4, ver=536870912, hashPrevBlock=6ff4f6a30b254ef8facae6a621ec0ea6853fcb115c320aefa2a2a5c6feb4bfe3, hashMerkleRoot=15ca5ca5c7045eb5f0be405828f7a8f8ce9b6c2c08c73871a01ec5ca2132684c, nTime=1510676108, nBits=207fffff, nNonce=0, vtx=1)
CTransaction(hash=15ca5ca5c7, ver=1, vin.size=1, vout.size=1, nLockTime=0)
CTxIn(COutPoint(0000000000000000000000000000000000000000000000000000000000000000, 4294967295), coinbase 520101)
CTxOut(nValue=20000.00000000, scriptPubKey=210293485085672a4273023a261874)
Then could it explain why it would reject blocks from other nodes at that time (after having already mined 40 blocks?). Then after 20-40s it would just crash with no explanation. I can then try to relaunch it (without deleting anything) and have it sync to the correct chain.
Sometimes it's sucessful and sometimes it crashes again and throws error such as:
Error: A fatal internal error occurred, see debug.log for details
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
The good news is that I still have no new error on the node still connected to testnet right now.