My question is, as at when my portfolio was zero, where did my money go? And when it started adding again, where was it coming from?
Now, let me understand your question differently. Others have already responded to the possible technical side of your confusion.
Anyway, if you are asking about the dollar value to your portfolio, it is actually just paper value. Meaning to say, you don't actually have it. Those are just numbers. If your Bitcoin portfolio's value, for example, changes from $10,000 to $1 and then from $1 to $50,000, it doesn't mean there is actually money in that amount which moved from somewhere to somewhere else. No money changed hands.
Those numbers will only be realized if you sell. By then, there is someone else on the other side who would provide you the real amount for your Bitcoin's fiat worth.