In addition, if someone figures out your address, they can simple come by and take your safe now (probably not worth the effort, unless you hold a huge amount of coins or the value skyrockets).
Keep the things that are required together seperated and keep the 24 words safe and you are good to go.
Lol I like your way of thinking but let me tell you it has some flaws in it from which to improve it later .
First they have to find out my real identity, address, house, city, road and everything to try and to come to steal my safe.
Second, I don't see any possible way of someone doing this without me giving them some hints.
I have done everything as suggested by Ledger support and so far no problems, my 3.37 BTC are safe there as of yet. If any hacker had any possibility I think they would be gone by now.