Topic: [BPIP] Bitcointalk Public Information Project [Back in Action] - page 10. (Read 19631 times)

I noticed recently three inconsistencies on bpip.org. I'm not sure if these were ever discussed, but I don't remember seeing these issues ever raised before. If they were mentioned earlier I apologize.

1. For some reason, while looking at my profile, I see a Sent/Received merits ratio of 1.08. This is impossible, for two reasons: (1) I am not a merit source and I did not receive any airdropped merit, therefore I can't send more than 50% of the merits I received, thus my max ratio should be 0.5; (2) the merits shown are correct -- 1604 sent and 3335 received -- and, if we divide 1604 by 3335 we obtain the score of 0.48. However, the "S/R Ratio" shows 1.08.



2. If I click on "Refresh Profile" a small window pops for a short time, showing "Most merited posts in the last 24 hours". However, I don't think that the posts shown there are also correct. For example, let's look at the fifth post written -- the one made by Hellena Yu, which appears to have received 5 merits. It's hard to believe that this is the fifth most merited posts within last 24 hours.



3. Still at "Most merited posts in the last 24 hours" window, I also observed that the merits shown there are incorrect. For example, Hellena Yu's post appears as having received 5 merits while, in fact, the post has 13 merits. Ddmr's post appears as having 10 merits while, in fact, the post has 29 merits and so on.

I hope this helps.

The problem is that sorting is done by the Bitcointalk server, and our extension works entirely inside the user's browser. So there is no easy way to implement sorting by merits. We could sort the current page but that doesn't help and might even be misleading, since the threads on other pages might have more merits. A more robust option would be to keep a list of threads on our (BPIP) server or perhaps use TryNinja's site to grab all threads for the board, but then it's complicated to display the result with all the other columns (i.e. last post etc).

I don't know if this has been suggested or not. I was looking for threads with the most merits on some boards through the help of BPIP extension but at some point, I got bored as I can't sort it. I'm not also aware of any other working method. However, as the screenshot says, all the option marked has clickable link behind the text so that one can sort according to asc/desc order. It makes it easier to find out the most for example viewed topics on that board.
If the "merit" button next to "Views" can also have the same feature, I think it would be very much helpful. Of course, if the "Merit" button is from BPIP extension  

Scraping programs normally use an interval of at least 2 seconds by default.

For this type of statistics sites, I take into account that there is no need for an update every second, it is enough to define the collection of information from minute to minute for most cases. And there are still cases where some information has already been collected by other users and therefore can be used without having to scrape again.

In the midst of all this, the only thing that influences is the statistics of the number of views/pages viewed, many of which are done by bots. But the forum is already at a level that this information turns out to be of little relevance. As long as the numbers of bots remain minimally acceptable.

I use wget.

It's rate limited though: a maximum of 1 page per second on average.

Hello, I just discovered this website and have some questions, I'll appreciate if you answer me. P.S. I am not a developer but would love to know a thing.

1. As I have searched, bitcointalk doesn't have an official API, so how do you get data from the website? Are you scrapping?
2. If you are scrapping, doesn't it slow down the website? Doesn't it increase their server load? If yes, then why do admins allow you to do so?
3. What programming languages and frameworks (if any) are used on this project?

I think it's already quite useful information, and it should serve what you want.

I'm planning to do one more statistical analysis on the members here on the forum.

I just remembered Piggy made a data dump, but it ends 3 years ago.
Can I be curious: what are you looking for?

I may not have been clear in the question.
I'm asking for a list of all users and their registration data.

Is it possible to get a list of this?

For knowing the registration date of users, you don't need BPIP at all. It's shown on users profile.
Anyway, users registration date is shown on BPIP as well. It's under "General Information for Profile ID XXXXXX".

Maybe, I am missing something here?

Hi @ibminer and @suchmoon first thanks for the great work.

I would like to know if I have any way to obtain the information of the users registration date. Can you get this info?

No worries, you have really done an awesome job to keep the project running. We appreciate your contribution so much.

So after moving all of my BPIP stuff to another machine recently to accommodate another need I had (personally), the machine I moved everything to went down, and after a few days of troubleshooting I've just decided to use a different machine to get the parser running again, my parser is the only one doing flag log updates at the moment. Weekly parse of all flags will happen soon, once I can clear the lock on that job, but the checking of recent flags has resumed. My apologies for the gap in time the parser wasn't checking these.

It seems the flag log hasn't been updating too since 6/25/2022 as well

The profiles haven't been updated in a while, you can tell by the 'Last Parsed' column on that page. I'm guessing the signature itself was still on the profile at the time of the parser seeing the user ban, so the parsers would have archived the profile and never went back to check for the signature after that. Manually refreshing each profile would update them and clear them off the list.

edit: To clarify, I think most of these were parsed before the /TempBans page even existed (in 2021), so the page would have went by whatever signature was last parsed with the account. Nonetheless, 'refresh profile' on each page cleared them off.  

Just alerting you.
Under temporary bans, https://bpip.org/TempBans Some users seem to have initially received a signature ban, and then they were completely banned afterwards. They still show up as though they have temp bans despite being permanently banned except for creeps.
Maybe you forgot to update that part.

Thank you all, I believe I corrected this the other day when it was reported so it should be parsing the seclog as of 5/28/22.. there was a lingering 'lock' on the seclog job, which stops a parser from running the job.. it did appear to stop on 2/25 as per the dates in the image below. The locking system will be looked at to try and circumvent this issue moving forward.

Thank you.
Apparently, I didn't check all the profiles shown in the seclog and I may be wrong. But seems that BPIP stopped scraping seclog on February 25.
According to LoyceV's seclog log, gemstone woke up on February 25, 6:42:14 PM and changed the password on 6:53:51 PM.
BPIP only shows the wake up and doesn't show the password change.



LoyceV's seclog log

If it helps, I have a backup: LoyceV's seclog log

I just checked some other profiles that are shown in seclog. None of the recent activities is shown in BPIP.
It seems that BPIP security log hasn't been updated for at least a month.