When the seller is willing to use the localbitcoins.com transaction service, there is no need to wait for any confirmations at all, but that service costs the seller 1% of the transaction amount. If the seller is in a hurry and doesn't want to wait for the number of confirmations you need to feel secure, suggest that they cover the cost and use the transaction service. That allows them to buy the ability to leave quickly after the transaction.
For amounts less than $20, I'm almost always willing to accept with 0 confirmations.
For amounts less than $200, I'm almost always willing to accept with 1 confirmation.
For amounts less than $2000, I'm almost always willing to accept with 3 confirmations.
I don't think I'd personally be concerned about waiting for 6 confirmations in a face-to-face exchange unless the amount was over $10,000.
That's my personal choice though. A zero confirmation transaction is pretty risky, but it would take some significant technical knowledge, to pull off a scam after 1 confirmation, especially if you verify that confirmation from multiple sources (your wallet, blockchain.info, and blockexplorer.com).
The only concern I've had so far in a face-to-face transaction was when someone tried to look over my shoulder as I typed in my wallet password. I don't think they were trying to memorize the password (which would've been extremely difficult for them anyhow), I think they were just naturally curious about what I was doing on my computer, but as soon as they left, I changed the password anyhow, just in case.
I've had a similiar situation with some random person looking over my shoulder just as I was entering my password. I asked him kindly to look away, and later changed my password just to make sure