Agreed. It doesn't seem like it if you watch the news but the human race does actually get more peaceful as we advance. If there is a civilization that is so advanced as to be able to do interstellar travel a reasonable assumption is that they would either be peaceful or avoid detection altogether so as to not interfere with our history.
There is some reason to fear emerging species from the interstellar viewpoint. Suppose you are happily existing around Alpha Centauri, and you get word of this massive heap of junk headed your way from Sol. The remains of an interstellar spaceship sent on a trip to your solar system which disintegrated en route killing everyone on their 250 year voyage.
All that junk is coming in at a significant fraction of light speed......
That's way, WAY more scary than a big rock hitting the Earth...I guess depending on the size of the rock, but energy increases as the square of the speed...