Who determines that?
The community and the developers. If you decide you want to use larger blocks, there are plenty of altcoins you can pick from which do so.Why do they change it frequently?
They don't. Block size has only been changed once, from a limit of 1 MB to an effective limit of 4 MB, as hatshepsut93 has explained above.If you want to check it yourself, go to any block explorer and look at block information, you will notice that all blocks have the weight of 4 MB, meaning they have hit the limit, but their size is between 1 and 2 MB.
That's not quite accurate. First, block weight is measured in weight units, not in MB. Secondly, not all blocks have a maximum weight. It is certainly the case at the moment, because the mempool is full and therefore every block is full of transactions, but if you look back a week or so to when blocks weren't full, then you will find plenty of blocks with a weight far below the 4 million limit. For example, block 629169 had a weight of only 577 kWU. 
