If you're trying to connect to 127.0.0.1 from a DIFFERENT machine, you're connecting to said port on the local machine, and not the one running your node. You'll have to find the ip address associated to theactual network interface of the host running your node (in a lan, most of the time these ip's are in the range 192.168.0.0/16).
Agreed. If your hardware wallet is connected to software that is running on a different computer than the bitcoin node, and the two are connected to the same router, check what ports your Bitcoin node is listening on by running netstat -tpla on that machine. Normally, Bitcoin Core listens on all of the available network interfaces, so just look for the IP address that starts with 192.168 and use that one for connecting.