[BPIP] Bitcointalk Public Information Project [Back in Action] - page 26.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on August 31, 2020, 05:54:33 AM

Can you give me some examples of users that you can't find? Do those users have DT ratings?

The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.

If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 31, 2020, 03:33:51 AM

because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it Cry

Can you give me some examples of users that you can't find? Do those users have DT ratings?

The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.

If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on August 30, 2020, 08:48:45 PM

Your bot will need a login, possibly deal with CloudFlare

Damn I forgot that to view profiles you need to be logged in

Quote from: suchmoon on August 30, 2020, 08:48:45 PM

Since you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are:

Code:

user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count

If this is good I can set it up to be updated once a day.

That format works for me! It would be great if you also added a "username" column so I can make it cross-compatible wether the user included bitcointalk profiles urls or usernames in the excel file.

Now, there is only ~27k users, is that what you meant by

Quote from: suchmoon on August 30, 2020, 09:33:09 AM

It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind.

because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it Cry

Hmm, maybe I should at first use this CSV to check the user's trust, if he's not in it, I will search for him in LoyceV's Custom trust list

Quote from: Vod on August 30, 2020, 11:04:37 PM

I assume you included me because I created it. thx! But I'm not part of the official team anymore.

Yes that's why I included you,I wasn't aware you left the team sorry!

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: Aveatrex on August 30, 2020, 04:59:12 AM

Hey BPIP team @ibminer @Vod @suchmoon

Hey Aveatrex

https://bpip.org/about.aspx

I assume you included me because I created it. thx! But I'm not part of the official team anymore.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 30, 2020, 08:24:37 PM

Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests Cry

but is a solution to be considered

You could do that but it's quite a bit of hassle. Your bot will need a login, possibly deal with CloudFlare, and of course scrape multiple pages.

Here's a file of all trust ratings that we have on BPIP - let me know if it works for you: https://bpip.org/all_trust.csv

Since you mentioned Excel I created it in a CSV format, which you should be able to import into a spreadsheet. If you prefer a different format, like JSON or XML - let me know. The columns are:

Code:

user_id, positive_score, negative_score, neutral_score, trust_color, dt_status, dt1_strength, dt2_strength, flag_count

If this is good I can set it up to be updated once a day.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: LoyceV on August 30, 2020, 09:24:12 AM

Quote from: Aveatrex on August 30, 2020, 09:18:25 AM

No, the part just under that, if it's Red, Black or Green.

I also have Trust score images, but it depends on how many you need.

Well if I'm going to release the software publicly, I can't predict how many users an user X will include in his excel file so that won't work.

Technically, I can use and scrape this page instead of bpip it would be much easier but the downside is that it updates every week which is a bit too long to my taste.

Quote from: suchmoon on August 30, 2020, 09:33:09 AM

We continuously scrape every user profile, prioritizing those who are most active.

Now that you say it, I wonder if i shouldn't make the bot do the same? i.e scrape directly bitcointalk profiles instead of using bpip as a middleman? That for sure wouldn't solve the false postives detected by 2 out of 68 anti viruses (using virus total) since it will still be making automated requests Cry

but is a solution to be considered

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 30, 2020, 09:18:25 AM

Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?

We continuously scrape every user profile, prioritizing those who are most active. It's technically possible that some trust ratings (colors) aren't updated quickly if e.g. a rarely active DT member posts a rating for another rarely active user, so keep that in mind.

I'll see how I can make this easier for you.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: Aveatrex on August 30, 2020, 09:18:25 AM

No, the part just under that, if it's Red, Black or Green.

I also have Trust score images, but it depends on how many you need.

Quote

Edit: I'm curious, if theymos trust dump updates every week, how blip updates the trust of users on demand?

Theymos only dumps Trust lists (only for users who have at least one post), not Trust (feedback) scores. BPIP has to scrape each profile for each update.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on August 30, 2020, 08:31:09 AM

What do you mean by trust - is it this part

No, the part just under that, if it's Red, Black or Green.

Quote from: hosseinimr93 on August 30, 2020, 08:57:48 AM

You can also use the tool created by LoyceV to see trust scores of all users in a single page.
Custom Trust list viewer, (Topic on bitcointalk)

Thank you for your suggestion I'll take a look at it

Edit: I'm curious, if theymos trust dump updates every week, how bpip updates the trust of users on demand?

hosseinimr93

legendary

Activity: 2380

Merit: 5213

Quote from: Aveatrex on August 30, 2020, 08:21:40 AM

...Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

I don't think you need to scrape BPIP.
You can use the following file provided by theymos to extract all trusts and distrusts. It is updated once per week.
trust.txt.xz

You can also use the tool created by LoyceV to see trust scores of all users in a single page.
Custom Trust list viewer, (Topic on bitcointalk)

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 30, 2020, 08:21:40 AM

I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

What do you mean by trust - is it this part?

Loading...

Edited 2020-11-30 to fix a broken image

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on August 30, 2020, 07:09:27 AM

Quote from: Aveatrex on August 30, 2020, 04:59:12 AM

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow

Let me know if you need additional information on the intended usage

Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it.

I'm scraping the trust of a set list of users from an excel file that the user selects and saves the results on the same excel file. The problem of this is that it's slow, is heavy on the server and triggers false positives from 1-2 anti-virus that thinks the software is performing a ddos attack or is part of some bot net. Is it possible to have a compressed file of the trust of each user updated like each 24 hours, or is that too much to ask for?

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 30, 2020, 04:59:12 AM

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow

Let me know if you need additional information on the intended usage

Let me know what you want to scrape (via PM if you don't want to make it public) and we might have a better solution for it.

Aveatrex

sr. member

Activity: 840

Merit: 375

Hey BPIP team @ibminer @Vod @suchmoon,

I'm currently developing a mini-software for the community, is web scraping your website allowed via automated requests? I prefer to ask before as I don't know if you can afford that with your hosting especially that right now, the website's response seem to be slow

Let me know if you need additional information on the intended usage

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Quote from: Timelord2067 on August 26, 2020, 02:47:13 AM

Quote from: TheBeardedBaby on August 26, 2020, 02:41:17 AM

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit"

Well I lost 2 positions due to inactivity the last few months but.. still in top 25 Tongue

Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information.

Hopefully it's just a glitch during the fortnightly update. (have gone back from 39^th to 37^th with my last fortnight's posts, so about the same as before. Wink

)

Are your house building works done yet?

Then mostmerited = mostmerit

Code:

https://bpip.org/report.aspx?r=mostmerit

Quote

Are your house building works done yet?

Renovating. 2 floors done, now is the basement this winter. Build some small things in the garden as well, almost finished with shed for wood. I'm doing all alone so it takes time.

Timelord2067

legendary

Activity: 3696

Merit: 2219

💲🏎️💨🚓

Quote from: TheBeardedBaby on August 26, 2020, 02:41:17 AM

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit"

Well I lost 2 positions due to inactivity the last few months but.. still in top 25 Tongue

Most merited is not the same as earned merit - the link is contained in the first shield of the five shields on each users' page information.

Hopefully it's just a glitch during the fortnightly update. (have gone back from 39^th to 37^th with my last fortnight's posts, so about the same as before. Wink

)

Are your house building works done yet?

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Quote from: Timelord2067 on August 26, 2020, 02:35:02 AM

Hi guys,

just a quick heads up, the link

Code:

https://bpip.org/report.aspx?r=mostmerited

returns a page:

Code:

https://bpip.org/?noreportname

Regards,

Just change the :

Code:

https://bpip.org/report.aspx?r=mostmerited

to:

Code:

https://bpip.org/report.aspx?r=earnedmerit

and it will be fine. I think there was an update, now I get also reward for the "Most earned merit"

Well I lost 2 positions due to inactivity the last few months but.. still in top 25 Tongue

Edit: wasn't mostmerit but earnedmerit, corrected now.

Timelord2067

legendary

Activity: 3696

Merit: 2219

💲🏎️💨🚓

Hi guys,

just a quick heads up, the link

Code:

https://bpip.org/report.aspx?r=mostmerited

returns a page:

Code:

https://bpip.org/?noreportname

Regards,

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Vod on August 25, 2020, 05:20:04 PM

https://epochtalk.org/reference/api-reference.html#get-recent-messages
That is only per user, but they do have a framework.

I know... it looks like it's just some autogenerated doc for the node.js crap. So basically what you see on the website. I don't think it has "get all new and updated trust lists since " or anything like that. Hopefully there will still be at least the weekly downloadable files.

Quote from: Vod on August 25, 2020, 05:20:04 PM

As long as the forum displays the last update, you can always recursively parse any top level category as needed.

Possibly. But on a busy forum like Bitcointalk you'd expect a lot of boards have activity so multiple pages would need to be reparsed every time. A patrol/recent posts page/API would be better and I don't think EpochTalk has that. Maybe putting all boards on a watchlist would solve it.

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: suchmoon on August 23, 2020, 07:43:14 PM

Ideally the forum should have a basic API, which would make it easier for everyone, but I'm not holding my breath.

https://epochtalk.org/reference/api-reference.html#get-recent-messages
That is only per user, but they do have a framework.

As long as the forum displays the last update, you can always recursively parse any top level category as needed.

Topic: [BPIP] Bitcointalk Public Information Project [Back in Action] - page 26. (Read 21862 times)