There might not be any anonymity, but that still doesn't help you prove that a specified person owns the private key associated with a newly generated Bitcoin address, whose only transaction so far was to receive some coins.
Unless the person shows you the private/public key; for all you know, the Bitcoin address was a DUMMY address, and the coins are essentially destroyed --- since there is no known public/private key pair in existence that would have the address that the coins were transferred to.
Incorrect. You had to transmit your public key to the sender who sent you BTC.
Edit: actually the public key is hashed and not public until you spend. But unless it was sent encrypted to the sender, this could be intercepted. And frankly most of the time you are not obtaining coins from such a covert sender. When you buy on an exchange, the authorities can get this information.
Also once you spend your coins then your public key is revealed. If you can't spend your coins without revealing your identity, then what is your point?