Is there a hard-coded minimum difficulty then?
Yes, the lowest possible target is hardcoded into the client.[1]
[1] https://github.com/bitcoin/bitcoin/blob/1b046603b30ebfab6199a2f92015d507b248b590/src/pow.cpp#L82
That's it! I could read in the code that the minimum difficulty target is 0x00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff. That is transformed into 486604799 when compressed. Then that's why the difficulty never went lower, even when the blocks were produced slower than 2,016 blocks per 2 weeks.
Thanks!