Yes. Why not? If you steal my bicycle, I can't use it anymore. If I steal your idea, you can still use the idea. Only one of us can use the same oven to bake a cake but an infinite number of people can use the same cake recipe.
If i write a book, I spend time, effort and resources on this process. I need to recoup those expenses + profit. If you copy the book, I get no return on investment. If you copy the book and sell it for profit, I can't compete against your price, because you didn't participate in the original expenses so you can reduce your margins. Think about it with a movie. Lots of money spent on it, jobs to pay, places to book, stories to write, film to edit and god knows what else. If you take it for free, I am losing money in the process, and I certainly didn't make that movie to lose money. If you steal my movie, I can't pay for food or rent anymore, how's that?
There is no 'aggression' in copying, as it is independent of the primary party (the author). No interaction between the 'author' and the 'copier' needs to occur. If no interaction, no violence can have taken place.
A good thief usually makes sure no interactions will ever occur between you and him. He's only after your property after all. You are confusing physical violence with the concept of violence, which is to take actions with consequences to external parties with no care for such parties' stance on the matter. And once again, you take the point of view of the aggressor. You are taking something from me while you refuse to aknowledge it as something that can be taken. You need to look at it from the author's stand point. If you think you aren't taking something from me you are profoundly wrong.
But let's discuss the fundamentals: do you think intellect can consist in property?