Outside your property, or on your property, outside the building? There's a difference, you know.
Outside your property, no, there's no foundation to insist he leave. You can ask, but you can't force him to go. Of course, he's probably on someone else's property, and you could probably get them to make him leave.
On your property, outside the building, yes, you can insist he leave, just as you could with anyone. It is, after all, your property.