Stake out a claim, defend it from all comers until everyone decides that it belongs to you --- aka the cost of invading you is considered too high because you keep winning. Historically this is how wars get started. Family vs Family - Tribe vs Tribe - State vs State.
I think the very idea of ownership goes against the NAP. One more reason that ancap would never survive for long.
It could also depend on verbal agreements. You keep this land, because you know how to use it best, I'll keep this land because I know how to use it best, and we'll both agree to be good neighbors, just because life will be easier for both of us.
Seriously, it's like you think the only thing stopping people from turning into marauding bands of murderous thieves is laws on some books. People can be decent and courteous human beings just because. This whole thing reminds me of how perplexed and stupified some christians are when you try to explain to them that atheists can still be moral even without some book telling them what to do.