Libertarianism: everyone is a rational actor, always. We shot all the people with mental illnesses.
Not so. Mentally ill people already take medication, and those who commit crime are treated like most any other person; this does not change, no matter what system you live under. Though Libertarians may generally be rational, they should not assume all people of the world will always be rational. Are you one of those irrational people? If not, don't worry about them; they already figured out how to get by now, they'll figure it out any other way. If you are, you should already know how you're getting along now.
The only way I could see this possibly working is if there was a standard rules agreement and people had to list where there rules deviated from that.
We already have that; just list everything you don't like to happen to you, and then don't do that to other people. Ta-da; it may seem surprising, but most people generally don't like the same things. If you want a really simplified version of this:
1. Don't fuck with people.
2. Don't get fucked.
But on the upside, the Mormons and Jehovah's Witnesses will probably leave you alone.