First off, what "the government" are you talking about. There are some 200 worldwide and every one of them can trade gold. Now factor in the millions of gold owners who are fee to sell at whatever price they choose. A bank may be able to manipulate THEIR valuation on gold, but how would they convince millions of others to sell at a price they choose?
Global control of gold price was not possible until the height of the British empire. (During the Dutch Golden Age the Dutch elites had to keep paper money tightly covered by specie since they still couldn't contain a capital flight out of their paper money into specie.) From the late 1860s Britain was able to co-ordinate with all major central banks to ensure convertibility of paper sterling to gold at a fixed price -- this was a total global suppression of gold market price. E.g. in 1890 the Bank of England borrowed gold from other central banks to contain a big financial crisis -- this meant, in practice, keeping the paper sterling price of gold the same as before.
After the collapse of the last remnants of the gold standard in 1971, it seems, manipulation has been done by co-ordination between the major central banks, bullion banks, and big investment banks. (The evidence provided by books like "The Gold War" and "The Gold Cartel" seems pretty convincing.)
Individual investors are price takers, not price setters, most of the time, in this system. Your gold dealer or coin shop will buy from or sell to you at a price based on the spot price "managed" by the elites via their "fix" system.
The borrowing of physical gold (ie the location of the physical metal, regardless of who owns it in theory) seems to play a key role. Most European countries' gold is held in their name in Britain and the US. When German voters forced Germany to ask for its gold to be moved to Germany, the process was strangely slow and tedious (and I think it will be done over many years and the final negotiated amount will be only part of Germany's gold.)
The only successful example of full relocation of gold was by Hugo Chavez of Venezuela. (You could also trust him to rebel against the system!) I think the process was also painfully slow. If they had trouble coming up with that small amount of gold, you can bet central banks' physical gold is highly leveraged for suppression purposes, and that the whole scheme can come crashing down. If you look into the fine print of any "paper gold" investments, you'll probably find that, in extremis, you have to accept paper currency.
Buy physical gold only.