As long as you can remember that you have done nothing. First things first with these government matters is that they're going to send you a letter or someone will step on your doors knocking informing you this kind of matters.
It's not limited to government calls that are being done by these scammers but even something simple that they'll try to scare you out so that you'll pay their scam. Don't be too scared with them and always have a presence of mind.
Some people are too vulnerable and believe in these
kinds of things. They panic and they don’t think properly.
Check the details of the caller/sender. Ask if they are
affiliated with any particular organization or government agency.
Do not act impulsively and keep a clear mind.