One problem that no-one has mentioned here is that in many respects it doesn't matter how large the theoretical block size limit really is, it matters what the block makers choose to accept. Right now, for example, most will only mine up to a maximum of 750,000 bytes even though the protocol allows 1,000,000 bytes (the default is 750,000 but can easily be changed). Some block makers are still using much smaller values, however.
The reality is that if pretty-much all of the major mining groups (many of which are privately held and therefore cannot be influenced anywhere near as easily as public pools) decide to keep 750,000 bytes, or adopt some other arbitrary smaller limit, then it doesn't matter if Gavin or anyone else changes the protocol to allow 20M bytes or 20G bytes, blocks will still be handled exactly as they are right now.
In practice mining 20M blocks isn't going to make any sort of sense for miners without something like the IBLT change that's also under consideration as otherwise the risks of losing an orphan race when announcing a very large block compared with another miner announcing a much smaller one will dramatically outweigh any theoretical advantage in terms of fees (which are already too low to be of any real interest anyway; larger blocks will push them down).