Topic: [130 weeks] [Updated Jul 10] LoyceV's Trust list viewer - Create your own! - page 2. (Read 22309 times)

Update:
Another update, another delay In the past week, one changed username messed up my scraping. And it didn't help that the forum was down for almost 2 hours this morning.
Everything should be okay again

Note: I scraped the Trust scores before theymos updated DT1, so they might be slightly outdated already.

New and wiped Trust lists in the past week:

Update:
After some delays, the update was finished yesterday already, but I wasn't at my desk so couldn't post it.

New Trust lists created in the past week:

So... in the past week, Admin(s) changed at least 4 usernames. I caught 2 on time, the remaining 2 messed up my update (because theymos' Trust data dump shows usernames instead of userIDs). That means I'll delete incomplete data again, and rerun my script. Sorry for the delay, this will add many hours to completing this week's Trust list viewer.

Update:
Sorry for the delay. I've had 2 earlier versions today, the first included incorrect data because of a changed username, the second try went wrong because I didn't clean all data. Now it seems to be correct.

New and wiped Trust lists in the past week:

Update:
The top-navigation-bar is finished already! That's how fast it is now (and I took much longer to post here).

New and wiped Trust lists in the past week:

Update:
Starting point to avoid loading the large index page: Trust list for: The Sceptical Chymist (Trust: +30 / =3 / -0) (5620 Merit earned) (Trust list) (BPIP).

New and wiped Trust lists in the past week:

---

---

The top-navigation-bar is working already, and the server is chewing through updates on the older weeks. Everything seems to work just fine now, but let me know if there's an issue.

So... there's an update on this: I think I fixed it

When I started this project, I had Trust data from a few weeks. So when I added the navigation bar, creating them all from scratch every week didn't take too long. By the time I had about 50 weeks and about 600,000 pages to update, I started caching temporary files to speed things up. When I got the first Xeon server (4 cores), it got a bit faster again.
But, by now I have data from 250 weeks, and close to 3.5 million pages, and there were still some old "loops" left to go through all of them: that meant that one loop had almost 1 billion operations. The newer and faster Xeon server (6 cores) should have been faster per core, but my script got slower. It now took almost 4 full days to update the top-navigation-bar on all pages, and it took the better part of Saturday to update navigation on the latest week.
So... time to overhaul this old spaghetti code. I got rid of the recursive loops, and now it scales more or less linear instead of taking exponentially more computing time. I also added multithreading, so it uses all CPU cores. There are still some time delays because all this happens on a different server than the one serving loyce.club, but I expect the navigation bar to be added to the latest week within an hour. At the same time it updates the navigation on the 6 most recent weeks. After that, the real fun starts and it crunches through the older data in 4 hours.

Part of my update means I no longer need to cache weekly temp files (it got to 1.9 GB and growing), which makes things easier but came with a small performance penalty. All in all I expect it to take about 4 hours now instead of 4 days, and most users shouldn't even notice it anymore as it only means "week 5" will only link up to "week 250" and not "week 251" until the update is done.

If anything fails, let me know. I'll use my new setup for the first time starting on Saturday.

---

---

Update: it's not working yet
Update 2: I think I got it now

Update:

New Trust lists created in the past week:

Update:
Starting point to avoid loading the large index page: Trust list for: Zaguru12 (Trust: awaiting update) (549 Merit earned) (Trust list) (BPIP).

Only one new Trust list was created in the past week:

Bump!

New Trust lists created in the past week:

---

---

Amazingly, since my upgrade to a faster server, updating the top-navigation-bar takes a lot longer. It's as if it only uses half a CPU core, it now causes the server load to go to 0.50 instead of 1. I'm still ill, but once I'm okay again (which takes a lot longer than expected), I'll add multithreading to the navigation-update.

Update:

New Trust lists created in the past week:

Update:

My (donated) server got an upgrade, so creating the top-navigation-bar should be a bit faster now. It's still a lot of data to edit though.

New Trust lists created in the past week:

Update:

New Trust lists created in the past week:

Update:

No new or wiped Trust lists in the past week.

Update:

New Trust lists created in the past week:

Update:

New and wiped Trust lists in the past week:

Update:

New and wiped Trust lists created in the past week:
The last user got Nuked, chances are he didn't wipe his Trust list, but theymos' data dump only shows data from users with at least one post.

Update:

New Trust lists created in the past week:

Update:
Starting point to avoid loading the large index page: Trust list for: LoyceV (Trust: +34 / =0 / -0) (DT1! (60) 13709 Merit earned) (Trust list) (BPIP).

New Trust lists created in the past week:

Update:

New Trust list created in the past week: