No, it wouldn't make much difference if the IRC network was compromised.
It is just used to get a list of hostnames to try to connect to, but you only need to connect to one real node in order to be able to receive the majority block chain. It is just one of many methods used to find other nodes - the address messages broadcast on the bitcoin network are the primary means, and once you've been connected to the network the client keeps a local cache of the addresses it has seen. The ones it sees on IRC are just added to the list.
You'd still be able to hijack new users
That would be true of any bootstrapping mechanism you can think of. At least with IRC it's somewhat transparent in that people can log in and see what's going on.. I don't really see any way around this though..
Also, users would probably notice that they weren't on the 'real' bitcoin network since they wouldn't be receiving any payments made to them or be able to send payments and also if they ever connect to a single real node by any of the bootstrapping mechanisms then they'd 'break out' of the illusion.