How did we go to talking about from responsibility to influence? If your friends dare you to throw a rock through a window are you no longer responsible for your actions?
Of course, unless you have some form of mental defect that makes you compulsively act out on your aggression. I covered that already.
Environmental factors do give rise to mental defects. It's not just lead poisoning, though that is one of the most obvious. It's also malnutrition, fetal drug/alcohol exposure, even chronic harassment or witnessing violence. You were the first one to bring up influence in the OP, when you said "what anyone else does has absolutely nothing to do with your behavior." This can be taken two ways: as an empirical statement about reality or as a normative statement of your moral attitudes. On the first count it is false. On the second count, just like any normative statement it cannot be proven or falsified, but I disagree vigorously. Many crimes and disasters have multiple causes. We localize responsibility on one or a handful of people for criminal justice purposes because it's impractical to do otherwise, but that isn't the end of the story.
I think that you're trying too hard to atomize responsibility. There can be more than one party responsible for a problem. If an arsonist sets fire to a night club whose fire doors were chained shut to discourage cover charge avoidance, there are several parties responsible for the ensuing deaths. Ordered by (my opinion) level of culpability: the arsonist, the club operator who disabled safety features, the local enforcement agency who failed to catch the safety violation, and the cheapskate patrons who used working fire doors to dodge payment.
If we wish to minimize bad behavior rather than simply condemn it after the fact, it takes collective action: agencies to fund epidemiological research and uncover significant but non-obvious links between environmental factors and health/behavior. An EPA and FDA to ensure that people aren't unwillingly breathing, eating, and drinking chronic poisons. Social services to ensure that children are being fed and not abused. If you think that equivalent services have been and will be provided in the absence of government action, I will happily discuss examples.