Topic: [130 weeks] [Updated Jul 10] LoyceV's Trust list viewer - Create your own! - page 2. (Read 22615 times)

Full disclosure: the list of DT1-users in week 264 is wrong. It uses the previous week's DT1-settings, instead of the latest one.

Update:

---

---

Two months ago, theymos made large changes to his Trust list that I missed until now:

Update:

New and wiped Trust lists in the past week:

Update:
Sorry for the delay, a name change messed up my script. I also got some error output, but can't pinpoint it yet. Let me know if anything is missing.

No Trust lists were created or wiped in the past week.

Update:

The following Trust lists were created or wiped in the past week:

Update:

No Trust lists were created or wiped in the past week.

Update:

In the past week, one new Trust list was created:

Update:

New Trust lists created in the last week:

Update:
Sorry, I forgot to post here earlier. The update was completed hours ago.

Update:

New Trust lists created in the past week:

Update:

New Trust lists created in the past week:

Update:
Back on schedule

New and removed Trust lists in the past week:

Update:
Another update, another delay In the past week, one changed username messed up my scraping. And it didn't help that the forum was down for almost 2 hours this morning.
Everything should be okay again

Note: I scraped the Trust scores before theymos updated DT1, so they might be slightly outdated already.

New and wiped Trust lists in the past week:

Update:
After some delays, the update was finished yesterday already, but I wasn't at my desk so couldn't post it.

New Trust lists created in the past week:

So... in the past week, Admin(s) changed at least 4 usernames. I caught 2 on time, the remaining 2 messed up my update (because theymos' Trust data dump shows usernames instead of userIDs). That means I'll delete incomplete data again, and rerun my script. Sorry for the delay, this will add many hours to completing this week's Trust list viewer.

Update:
Sorry for the delay. I've had 2 earlier versions today, the first included incorrect data because of a changed username, the second try went wrong because I didn't clean all data. Now it seems to be correct.

New and wiped Trust lists in the past week:

Update:
The top-navigation-bar is finished already! That's how fast it is now (and I took much longer to post here).

New and wiped Trust lists in the past week:

Update:
Starting point to avoid loading the large index page: Trust list for: The Sceptical Chymist (Trust: +30 / =3 / -0) (5620 Merit earned) (Trust list) (BPIP).

New and wiped Trust lists in the past week:

---

---

The top-navigation-bar is working already, and the server is chewing through updates on the older weeks. Everything seems to work just fine now, but let me know if there's an issue.

So... there's an update on this: I think I fixed it

When I started this project, I had Trust data from a few weeks. So when I added the navigation bar, creating them all from scratch every week didn't take too long. By the time I had about 50 weeks and about 600,000 pages to update, I started caching temporary files to speed things up. When I got the first Xeon server (4 cores), it got a bit faster again.
But, by now I have data from 250 weeks, and close to 3.5 million pages, and there were still some old "loops" left to go through all of them: that meant that one loop had almost 1 billion operations. The newer and faster Xeon server (6 cores) should have been faster per core, but my script got slower. It now took almost 4 full days to update the top-navigation-bar on all pages, and it took the better part of Saturday to update navigation on the latest week.
So... time to overhaul this old spaghetti code. I got rid of the recursive loops, and now it scales more or less linear instead of taking exponentially more computing time. I also added multithreading, so it uses all CPU cores. There are still some time delays because all this happens on a different server than the one serving loyce.club, but I expect the navigation bar to be added to the latest week within an hour. At the same time it updates the navigation on the 6 most recent weeks. After that, the real fun starts and it crunches through the older data in 4 hours.

Part of my update means I no longer need to cache weekly temp files (it got to 1.9 GB and growing), which makes things easier but came with a small performance penalty. All in all I expect it to take about 4 hours now instead of 4 days, and most users shouldn't even notice it anymore as it only means "week 5" will only link up to "week 250" and not "week 251" until the update is done.

If anything fails, let me know. I'll use my new setup for the first time starting on Saturday.

---

---

Update: it's not working yet
Update 2: I think I got it now

Update:

New Trust lists created in the past week:

Update:
Starting point to avoid loading the large index page: Trust list for: Zaguru12 (Trust: awaiting update) (549 Merit earned) (Trust list) (BPIP).

Only one new Trust list was created in the past week: