The wireless is probably the culprit here. Mining doesn't take much bandwidth, but latency can be a big problem. Not sure if there are any settings you can use to improve it. Encryption and authentication probably adds quite a few unnecessary milliseconds, especially with normal consumer adapters and routers, but turning that off is of course limited by what else the network is used for.
In any case, if you can move the rig to test it with a physical connection I'd recommend trying it out.