In the two PCs I have version 0.7.1 of bitcoind, one is build 0.7.1-precise1 this is what gives 4s latency over right now and the other is the build 0.7.1-oneiric1 works well but not I think the problem comes from the collection, I end the ideas ...
Try editing your bitcoin conf with the below settings. The reasoning behind them is with less connections and less transactions to process your bitcoind will respond faster. It did help a lot for me. Also I've noticed that anyone compiling their own bitcoin from git has a big advantage too since the new db system for the next bitcoin release is way faster. If you can't compile your own keep an eye out for a 0.8RC1 when it comes out.
Code:
maxconnections=10
#Maximum size, in bytes, of blocks you create:
blockmaxsize=50000
#How many bytes of the block should be dedicated to high-priority transactions,
#included regardless of the fees they pay
blockprioritysize=2000
#Minimum block size you want to create; block will be filled with free transactions
#until there are no more or the block reaches this size:
blockminsize=0
#Fee-per-kilobyte amount (in BTC) considered the same as "free"
#Be careful setting this: if you set it to zero then
#a transaction spammer can cheaply fill blocks using
#1-satoshi-fee transactions. It should be set above the real
#cost to you of processing a transaction.
mintxfee=0.0005
263MB res just after start on my 64bit VM 
