The biggest reason against raising the block size limit is the risk of Denial of Service attacks. The idea of "just change one constant" comes from ignorant and naive people who do not understand how Bitcoin works on a technical level.
The issue is that of signature hashing for signature operations (sigops). Right now, sigops-to-verification-time is a quadratic relationship, meaning that as you increase the number of sigops in a transaction, the time to verify that transaction squares. For example, doubling the number of sigops in a transaction (as with a 2 Mb increase) will require quadruple the amount of time to verify the transaction. Bump up block size limit to 4 Mb (thus bumping the sigops limit to 4 times of what it is now) means that a transaction with the max sigops will require 16 times as long to verify as a transaction with max sigops now.
This sigops issue is one of the biggest problems when it comes to scaling as it becomes possible to cause nodes and miners to have to verify huge transactions and waste time, time in which another miner could find another block. Furthermore, to solve the issue becomes a major pain. There could be a maximum transaction size, but that isn't really ideal (although AFAIK that is what Classic decided to do). Alternatively the signature algorithm could be changed, but in a hard fork scenario, that is just a nightmare to deploy as any unconfirmed transactions at the time of deployment suddenly become invalid.
Edit: The 1 Mb limit was added because Satoshi was worried about spam attacks.
Because the main developers in charge of the core repository refuse
to raise it.
Some of the most senior members also work for a
corporation called Blockstream who wants to develop scaling
solutions off the main chain, so therefore many of us believe,
unfortunately, that there is a major conflict of interest at work here.
While blockstream is developing sidechains as an alternative scaling solution, the idea that they are going to somehow force everyone to use their sidechains is false. The sidechains are completely open source licensed under the MIT License which means that literally anyone can take the code, use it, and start their own business with it. They are literally allowing anyone to make money off of something that they created. Secondly there is also the lightning project, which is not developed by blockstream and is also open source with a few implementations of it already existing.
Additionally, while there are several members of blockstream who are Core devs, they are also the ones who founded the company. Only one of those members even has commit access so it cannot be said that blockstream controls Bitcoin Core. Granted many of those members are senior Core developers who have been actively working on the project for several years, long before blockstream was even founded, and their opinions do carry significant weight.
Lastly, those developers are not actively trying to undermine Bitcoin as some would lead you to believe. They are all doing what they think is good for Bitcoin, as are the developers of XT, Classic, and Unlimited.
Also, increasing the block size limit ad infinitum is not a viable solution either and, at some point, we will need to use off chain scaling solutions.