On second thought: I don't think the bold part should be the default: if you use multiple wallets, or resync your main wallet, you'd want your spent inputs to show up too.
Here's a quick thought: When creating your wallet, scan for all transactions that include a Taproot output. When opening your wallet afterwards, look only for unspent inputs. This way, it'll take some more time to find your transaction history only during setup. For faster scanning, silent payment software could give you a wallet block height for when recovering it in the future.
Silent payments would not be very silent if they could be easily identified in the blockchain (hence chain analysis also identifies them) - so therefore it's a good thing that there's crap like Ordinals and Runes for it to blend in with, "obfuscation" if you may call it that.
On the other hand when the blocks are full of Taproot outputs, it becomes more computationally expensive to search for them. The process is parallelizable to a degree, but only for parsing the transactions in a block. Unless you are querying from a node that carries block undo history.
So maybe we will see more applications of taproot in the future.
