I went to my bank yesterday and made a withdrawal of almost all my balance. Was that an illusion, period?
(Of course I know the bank is loaning my money and eventually I will be unable to withdraw it. However, this has not happened so far.)
If the bank loaned my money, then it forgot to subtract any such loan from my balance. If my money has indeed "moved into someone else's pocket," then the bank, for not letting me know about it by subtracting the moved money from my account balance, is defrauding me---which you said not long ago it is not doing, remember?
So it is not true all I have is an IOU. What I have is an IOU that I must take as money (since this is what my account balance is telling me).
It is not my choice to treat this IOU as money: it is the only money around. If I don't treat it as money, then I will have no money to pay for things (at least while Bitcoin does not go mainstream).
What you are failing to understand is that even if I am eventually unable to withdraw my money, I am still entitled to do so, which is precisely why fractional-reserve banking is a flawed monetary system.
Under the current system with the FDIC and FED you have the illusion of no responsibility because you are protected by them and can count on always getting your IOU repaid which is the huge problem banking has today. But that doesn't mean you aren't actually holding an IOU when depositing into a demand deposit account and that you aren't personally responsible to pick a bank that will prudently loan out your money and keep a prudent reserve ratio. You are, read your contract with the bank, it tells you are when they tell you that under certain circumstances you wont get your money on demand.
One thing is my responsibility for the monetary system society adopts. In this sense, I have the responsibility to fight for a better monetary system, for me and for all. However, in the current monetary system I am not responsible for my bank's loans. This is not a matter of considering myself responsible or not: this is a formal contract between me and my bank. Whatever that contract says, my money remains mine and I can always withdraw it.