I've finished re-writing the code for the top-navigation-bar. It now "only" takes 6 hours to update instead of 2 days, and from now on the required processing power only increases at a linear instead of exponential rate.
This is much longer than it should. You should consider using jinja templates versus having millions of individual files, each with their own HTML. This would result in your server/computer generating the HTML at the time a user accesses a page instead of when you update your data. It would also mean you can update your data ~instantly. I don't know how heavily your site is used, but my guess is 95%+ of your HTML files are never accessed by anyone. If someone accesses a particular page n times, your server would need to generate the HTML n times. I think most likely, this will result in total less processing time, but even if not the case, the computational power required to generate one HTML page is nominal, so you will likely not ever have a backlog of things to compute.
