That's because they have no idea what private keys I possess.
You have to tell us something more to give you an answer. Tell us something about the way you get your income and the way you spend your money. For example, are you paid for your work in BTC? How do you convert BTC in USD for anything you need to pay that does not accept BTC? How do you pay rent, electricity, internet, etc?
There may be a loophole in your process.
For example, let's say you provide web design services. I hire you and I pay you with BTC. Your part of the transaction is transparent to IRS but I will deduct your payment to the IRS as my business' expenses. If I am audited, I will tell the IRS that I paid you in BTC and the IRS will ask you where those BTC are and it's your burden to give them this info.
Another example. You have a bunch of BTC but you have to pay your rent in USD. You need to convert those BTC in USD. Of course, if you use an exchange platform such as Coinbase, they will get you. Maybe you will get cash selling BTC to someone for USD. That someone may be another loophole for you. Also, you pay rent and your landlord puts that rent in his income. If he is audited, he will show IRS that he received cash from you and IRS will then ask you where you got that cash from.
In summary, as long as your whole life is not in BTC, you may get caught because of third parties you have interacted with.