For example sometimes when people say "wallet" they actually mean "account". Something they have with a centralized service like Coinbase or blockchain.com. In this case it depends on what you mean by "you"? A regular person can not find your wallet from your IP address but obviously these services keep a record of your IP and can associate it with your wallet.
Another case off the top of my head is the wallet type you use. For example it is a lot harder to associate wallet (addresses) with IP addresses if you were running a full node like bitcoin core; however, it is easier to do that if you were using a SPV wallet like Electrum and far more easier in case of server dependent SPV wallets (where you can only connect to a single centralized server to sync).
In this case again it is harder for a "regular person" to make that association but the servers, the network "crawlers", the SPV nodes, etc. can do that.
There are other criteria involved too. So you have to clarify what exactly you have in mind regarding IP and wallets relationship.