Gold in part holds its value because of its limited supply. Unlike fiat where we keep on printing, more money supply, means it value becomes diluted. Thus in reality for the same amount of gold, essentially we need more and more of the paper currency as time goes back to purchase that very same amount of gold.
This is true, but there is a caveat:
it is not because something is in limited supply, that it is a monetary asset, or that it has value as a store of value.
The link between both properties (limited supply, and monetary asset) is in fact very weak, although limited supply is a basis for a "sound money" doctrine.
You can have things in limited supply which have no function as a monetary asset at all (are not a store of value), and you can have things of which there is a continuous supply, and which have nevertheless a monetary function.
My first kid's baby drawings are for instance an asset in limited supply. In fact, because my first kid is not a baby anymore, there is an absolute guarantee that there will never any produced any more. We actually threw away most of them, and only kept a handful as souvenir. I'm 100% certain that those drawings, even if my first born becomes a nobel prize winner, will never be a monetary asset :-)
The often-mentioned beanie babies are also on limited supply. They are not a monetary asset either.
In order for something to become a monetary asset, there has to be a sufficient number of people wanting to consider them as such.
As you mentioned, fiat money is in continuous supply, but nevertheless, it is a monetary asset. Too much supply can induce the loss of its status as monetary asset (hyperinflation), and that has happened to quite some fiat moneys, but the "big" fiat moneys are around for many decades or even a century or more, without them loosing their status as monetary asset (they loose in value, but not in status as monetary asset).
The problem with continuous supply of monetary asset is not so much in the risk of hyperinflation (although that can happen). The true problem of continuous supply is the seigniorage. The fact that the suppliers of the money get all the value of it (which is taken on the inflationary effect, paid by the holders of the asset). This unwanted transfer of value is problematic. This is why a sound money doctrine wants a finite supply.