You are asking two different questions (and I'm not even sure if you realize it). Some people are responding to one of those questions, and other people are responding to the other question. This is creating confusion and miscommunications. I suppose, we need to start by figuring out which question you are trying to understand.
Question 1.
Why does a transaction need to include an output specifically for sending the change from the transaction back into the wallet? This could also be phrased as "Why was the protocol designed to spend previously spent outputs in their entirety?"
The answer to this question is that it is the most efficient and reliable way that Satoshi could come up with to create a trustless distributed system. If you have a better way, go ahead and suggest it, but you'll almost certainly find that it won't work without a centralized trusted source of authority.
I don't "buy" that part - but I'm not necessarily implying you are "selling" it. The fact that Bitcoin is trustless is related to the PoW that makes it possible for the algorithm to determine the validity of transactions through network consensus - not because it uses change addresses.
I haven't done extensive research on other types of blockchains (PoW / PoS) that are written from scratch - perhaps someone that has a greater familiarity with such blockchains can tell us whether they are emulating Bitcoin's choice or if it is unique in Bitcoin.
You don't "buy" it because you want there to be a simpler way. I've not heard of any better ways in the 28 months that I've been studying cryptocurrencies. Wanting something doesn't make it so. If you know of any better ways, please suggest them, but anytime I've seen anyone try to present something that they thing is simpler it's been clear that they haven't really thought the process through and their idea is full of holes.
Question 2.
Why does the Bitcoin Core wallet choose to create a brand new address to send this change back to with every transaction sent, rather then sending to one of the existing "receiving addresses" in the wallet?
There are several answers to this question:
- It slightly increases anonymity and privacy
- It slightly increases security by maintaining 3 levels of cryptographic functions between the private key and the address
- It allows a user to track where all the payments to their wallet came from, since they can give out a new receiving address for every transaction.
The first part seems slightly futile when you do a common spend and things get linked. But it is a slight increase, I agree.
Yes, a slight increase. As you've mentioned, the increase in privacy and anonymity is rather insignificant unless the spender is using coin control and is very careful about how they structure their transactions.
The second part, IMO, can be a small factor or a large factor, depending the point of time. In other words: Has a QC been developed (whether the public knows it or not) at a specific time? If the answer is yes, then money are far better protected.
Even without QC, there is a distinct possibility that in the near future mathematicians could discover previously unknown weaknesses in ECDSA. Such weaknesses might not result in the ability to calculate a private key from a public key in a few minutes, but even if it reduced the time to calculate a private key down to a few years (or months, or days, or hours) your bitcoins would be safe as long as they were associated with an address that had never had its private key used to sign any previous transactions.
The third part would be ok in theory but it creates more confusion for the average user due to all those tiny amounts that end up being an entire list. A visual representation tool would be, IMO, better for that purpose.
It only causes confusion if you are trying to understand the technical details or use the wallet in a way that it isn't intended to be used.
If you create receiving addresses to receive bitcoins from people, and use the wallet to send bitcoins to people, and maintain a reasonable backup schedule, there isn't anything confusing about it.