Well, economic growth can also be attributed to the manifestation of new resources, such as discovering a coal deposit or an increase in the population, but all else being equal growth is an increase in efficiency.
What you do with the dirt matters, but this will be determined by prices in the market. A business will produce [resources out] based on whatever configuration yields the most profit, considering those prices. This process maximises the total subjective value of all resources.
People who lose their jobs due to their work being made redundant will get jobs in other fields. The money saved from the new efficiencies will be freed up for other things. These newly unemployed people will have to provide these other things. Having 100 men dig a hole when 1 man + machine can do it is just a waste. There is no point being inefficient just for the sake of jobs. Remember, we want stuff, not work.
The end result is that quality of life goes up for all. And of course the environment is completely destroyed due to edge effects, because the environment has 0 value ;-)
Well I like the idea that quality of life could come into play in an economic indicator, but this is tough to quantify. Some indicators of quality of life such as people "not being depressed" or "not having cancer" correlate inversely with the counterfeit-money-distribution index (GDP). There are also reasons to believe that your quality of life will be better if you get out and dig a ditch once in while rather than letting a machine do it every time. And as for "we want stuff".. I am hopeful we can move past that. Stuff is good for filling landfills, for example making giant piles of crushed automobiles seems to be an indicator to some of "economic growth". But are giant floating island of garbage really what we're after? For many an epiphany in their view of "stuff" comes from having a family member pass away and being in charge of "getting rid of the stuff". Hoarding is an instinct that does not always serve us well. As for the environment, we are the environment. Quality of life is a nice phrase but I think survival is even more basic, (QOL requires L), and makes it more obvious that our extended bodies (the environment) are crucial.
Quality of life just means the things you value have a lower price. You don't have to "hoard". You could just work less and achieve the same basic necessities. Materialism is an orthogonal issue.
Imagine you can buy all the food and shelter you need and basic utilities, while working 1 day a week. Think of the Jettsons, where George had a nice place, family, robot maid and worked 2 days a week just pressing a button. This is a society with high efficiency. So optimised that pressing a button generates wealth to support a family.
The problem we have now is that the vampire squid called the state is sucking the life out of the economy as fast as it can grow. Why are first world nations experiencing stagnant growth, with all our technological advancements? Because the government is constantly piling on new taxes and regulations and stealing through inflation.
