Topic: [BPIP] Bitcointalk Public Information Project [Back in Action] - page 10. (Read 21833 times)

OK, /TrustLog should be flowing again and the ranks have been updated for most posts deleted.
To summarize the changes recently:

-Added 2023 option to Most Earned Merit report (by time)
-Adjusted column header for Most Earned Merit report from "Times Merited" to "Merit Received" to clear up confusion.
-Modified UpdateRanks SP to fix deleted posts report & ranks not updating
-Re-seeded one of the trust tables to get the /TrustLog operational again
Enjoy the music.

Long story short, one of the tables involved with the TrustLog has max'd out its identity column.. because, for some reason, the table appears to have started out with a high seed value which has caused it to max out after only ~1.3M 11.3M rows, when it should be able to handle ~2.1B rows as an int. Possibly at some point SM had to wipe this table and the seed started off high at that point, but I'm not 100% sure.

Once I feel comfortable I won't screw anything else up here by resetting these IDs, I'll need to reseed the table, which should get this flowing again.

edit: After re-counting the rows in the latest re-seeded table, I realized I miscounted digits when calculating the original table, which was actually ~11.3M rows, not 1.3M like was in this post, just updating to keep things accurate.

Timelord2067 posted this in my topic, but it belongs here:
The last entry in BPIP's DT Change Log is from May 20.

Deleted posts from mods value & rank:
To provide some follow-up on my prior post:
Looking into the counts & ranks closer here for this particular report, I was able to narrow down what was happening, but still need to fix it.

As it stands now:
a) The profile page is getting its value/count from a table/field which is being updated by parsers.
b) The report page is getting its value/count from a table/field which is not being used or updated by parsers.
c) Both pages are pulling their rank from the same table/field, which unfortunately has not been part of the stored procedure which calculates and updates ranks.

Because of this, the visible result is:
The report page for deleted posts by mods hasn't been updated at all (in terms of value/rank), for some time they've been static.
You can compare an archive from 2020 with a recent archive from the end of March and see the value & ranks haven't changed.

For the profile page, the value has been changing, as can be shown checking the archives of someone like JollyGood (current & archive from Feb 2021 and June 2022).
But the rank has remained static on the profile page as well, again due to the current table not being included in the rank calculations.

All this said, there is a chance for potential small gaps in capturing the modlog/seclog in the past, if there were parser issues or downtime occurring, which occasionally happened more in the earlier days of development then it would now, but I'd expect the count shown on the profile page is close to what it should be, based on what has been collected from the parsing since 2018.

---

Here is what I'm seeing:
Seclog on BPIP goes back to March 16th, 2018.
Modlog on BPIP has ~1.3M saved records with an accurate date/time which ranges from August 3rd, 2018 until today. The modlog also has 177,836 records with a (default) unix timestamp from 1970. Based on the ID's of these records without a timestamp, they were all created (assumingly in order) before the range of time above. I didn't go back and look but the lack of a timestamp on these were likely related to a bug which was ultimately fixed in August of 2018, when it started capturing the date/time correctly.

Currently, you'd only be able to see:
Full sent: https://bpip.org/smerit.aspx?from=Avirunes
Full received: https://bpip.org/smerit.aspx?to=Avirunes

But it might not be a bad idea to have a view showing them merged together for a profile, which shouldn't be too difficult. I'll add this to my TODO list.

Ah LoyceV's tools. That will for sure do. Thanks a lot 

If it helps: LoyceV's seclog log.

There's loyce.club/Merit/history/175302.html.

I've some questions regarding BPIP security/moderator log. Does it shows the full list of security and modlog till date? or is there a point of date? I am checking a profile and just trying to make sure of something.

Also is there a way where I can see full merit history of a profile?

I've seen BPIP overlook deleted posts from modlog, so this counter could be too low.

BPIP only counts posts deleted by Mods, I scrape all posts including the ones you delete by yourself later on. Judging by my own data, I have deleted about 2000 of my own posts (mostly bumps).

Correct.

Hey thanks for that. I had a feeling it was going to be a long shot.

---

---

I see a couple of people have been discussing the disparigy between bpip and other sites' information (in the above example numbers of posts deleted).  I recently had someone suggest I'd only had 33 posts deleted, but the thread was locked before I could point out both loyce and ninja's sites have my post counts hundreds of posts higher than what bpip says with the Forum's user profile page only showing the number of posts that *haven't been* deleted.

The other two sites are much closer to the count, but probably don't include those posts of mine that were deleted in the first four or five years after registration.

Depends on the information you are looking for, but as far as I know. The log kept getting modified over time

The earliest records you will find are of most merited posts daily, and then Changed to Archived status, mostly because data would be displayed on the site even when Vod was still part of the project(I see some info as far as 2018)

When were the first entries recorded by BPIP.org ?

Thanks, I'm seeing inconsistencies for many users as I spot checked, not just MFahad. Some are off by a few (Fredomago, cvm, Alex-007, etc.), some are off by larger amounts (joshy23, Adriano2010, JollyGood, etc), and a lot seem to match up correctly.

Trying to wrap my head around this one as I haven't really looked at this job (and related procedures) recently in the past, but there does appear to be a problem here which will need some troubleshooting which I can't do at the moment, I'll try to take a closer look as soon as I can.

I saw this post:
And indeed, bpip.org/Profile?p=MFahad shows:
However, bpip.org/Report?r=postsdeleted shows:
I checked a few other profiles in the list, and for them, the data is consistent. How did MFahad go from 164 to 372?

If it helps: loyce.club/other/tmp/ibminer/ (scheduled to be deleted in 30 days) lists all userIDs who posted on the past 29-ish days. Note that the files contain duplicates if users posted more than once.

Thanks, and no not normal. As of yesterday the recent posts job had been locked for around ~26 days.. so the job hadn't been running on the parsers, picking up those posts, and updating the status of those users. I cleared the lock so it's been running correctly now for new posts, but I may need some time to figure out a way to either re-process posts through the parsers in that timeframe, or find some way to manually update. If any users continue to post, it should at least update the status on their next post.

Does this mean BPIP doesn't scrape all new posts? If that's the case, I can easily share a list of all users that posted "today". I could provide historic data too: I have this data starting May 23, 2019 already (I needed it to keep remove unbanned users from the list of banned users).

I don't know if I have not been so observant in the past but lately there seem to be so many profiles just woken up showing as "inactive" or "archived" several minutes or hours later after posting

Like this user: https://bpip.org/Profile?id=1286231



Is it normal?

Sorry for the delayed reply. I was not aware of how many users have same Trust score... wow!

Howeever, I don't think that was the reason for the change... I will explain:

- last time I posted here it was just after the DT change, when I was dropped to DT2. At that moment, my position on those ladders registered a fluctuation of 100 places (similar to current one)
- since then, my position on those two ladders faced only minor changes
- immediately after lattest DT change, when I returned to DT1, I noticed same huge fluctuation. So I believe that, when I checked my profile on bpip, the DT change was processed already by bpip.

Having these said, I think it would be a very big coincidence to have these huge fluctuations twice already, first time after being dropped from DT1 and second one after returning to DT1... So I believe this is the cause - the change from DT1 to DT2 and from DT2 to DT1 and not the Trust score change of other users having a Trust score close to mine.

I will still keep observing this at each DT change I'll meet, but who knows when I'll go again to DT2... In any case, I will pay attention each month, after DT users are reshuffled and, when I'll go to DT2 I'll come back to confirm if same huge fluctuation occurred. My curiosity was if such a big fluctuation is normal...

I think there are 2 causes:
1. BPIP doesn't update all profiles at the same time, so theymos' DT1-update doesn't instantly reflect in the correct new order.
2. Some people have quite a lot of inclusions, including self-scratching. So their own Trust score changes significantly depending on whether or not they're on DT1.

Scrolling through Most Trusted Profiles, there are many users with the same Trust score as yours. A small change is enough to bump you up or down quite a lot.

You can check a few users for yourself: check their Trust/Merit/Activity, and see if they're in the right position in the list.

I come back to this thread for a sequel of my earlier post. I have to confirm what ibminer said above. However, I still have one question...

So as you know, theymos made another reshuffle of DT members, on March 1st. By pure chance, I took a screenshot of my bpip profile before the reshuffle and one just after. Before the reshuffle I was in DT2 and after I reached back in DT1. Also have in mind that, during this time, my Trust score has not changed, nor my positions on "Most merit" and "Most earned merit" ladders (I received some merits between those pictures, yet I remained on same positions on both merit ladders). What I am trying to say is that, excepting the fact that I went back to DT1, nothing else changed (nothing which could have any implication on "Most trusted" and "Most recognized" ladders).

So the differences are the following:

Before the reshuffle:


After the reshuffle:


As you can see, there is a huge step forward on both "Most trusted" and "Most recognized" ladders. Last month, after the reshuffle, when I went to DT2 from DT1 I also noticed a huge difference on these ladders, but the huge step was back

Anyway, my question is the following: I understand that both of these ladders are influenced by the presence in DT1, respectively DT2; however, is it actually normal such a huge difference? There is a 69 positions difference on "Most recognized" ladder and 176 on "Most trusted"... I don't know but, from my perspective, this difference seems way too big to be natural...

ibminer, please don't take me wrong -- I am not questioning your algorithm here -- I posted all these just to ask if it would be a good idea to make a double-check, to ensure that all these scores are calculated correcly