Can we have one small change on Recent Merits page?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: hatshepsut93 on October 02, 2019, 06:31:27 PM

Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement.

What's really going to get you, is if the Merited post was made on a hidden board (most likely the VIP or Staff board). It'll look like the post was deleted, and you'll have to correct it after theymos' weekly Merit data dump.

Quote from: hatshepsut93 on October 02, 2019, 07:44:32 PM

Did theymos ever said anything about scraping in general - what's his stance on it?

I haven't read anything about it, but if you go faster than 1 request per second, you'll hit a timeout.

hatshepsut93

legendary

Activity: 3024

Merit: 2148

Quote from: hosseinimr93 on October 02, 2019, 07:06:45 PM

http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly.

I know, my project is actually based on it - https://bitcointalksearch.org/topic/json-api-for-merit-data-5181082

Quote from: Vod on October 02, 2019, 07:28:20 PM

I sent this in a PM to Theymos when merit was released, but no reply. I don't think he wants to make it too easy to scrape the data.

I was under an impression that this kind of pages exists specifically to be scraped, am I wrong? Did theymos ever said anything about scraping in general - what's his stance on it?

Quote from: Vod on October 02, 2019, 07:28:20 PM

https://bpip.org/smerit.aspx?to=&from=&start=&end=&min=&max= ---> Real Time Historical sMerit Transfer. See link for instructions

Doesn't work, says "to or from is required", which is not what I need - I don't want to poll this page for every subscribed user, this wouldn't be scalable.

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: hatshepsut93 on October 02, 2019, 06:31:27 PM

I would like to scrape https://bitcointalk.org/index.php?action=merit;stats=recent to make real-time merit api and notifications, but this page lacks one crucial piece of data - the name and link to the profile of a user who received merit. Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement.

I sent this in a PM to Theymos when merit was released, but no reply. I don't think he wants to make it too easy to scrape the data.

Meanwhile...

https://bpip.org/report.aspx?r=meritratio ---> Merit sent/received ratio
https://bpip.org/report.aspx?r=meritcircle ---> Merit circle (number of distinct profiles merit sent to or received from)
https://bpip.org/report.aspx?r=mostmerit ---> Most overall merit (initial and received)
https://bpip.org/report.aspx?r=mostmerited ---> Most merit received
https://bpip.org/report.aspx?r=meritsent ---> Most merit sent
https://bpip.org/smerit.aspx?to=&from=&start=&end=&min=&max= ---> Real Time Historical sMerit Transfer. See link for instructions
https://bpip.org/r/mostmerited.aspx ---> List of most merited posts for a certain time frame

tranthidung

legendary

Activity: 2310

Merit: 4085

Farewell o_e_l_e_o

Quote from: hosseinimr93 on October 02, 2019, 07:06:45 PM

http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly.

It is weekly updated because the data source comes from weekly update of theymos

Quote from: theymos on March 01, 2018, 06:37:16 AM

Here you go: https://bitcointalk.org/merit.txt.xz

Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them.

I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources.

Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs.

hosseinimr93

legendary

Activity: 2380

Merit: 5213

http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly.

hatshepsut93

legendary

Activity: 3024

Merit: 2148

I would like to scrape https://bitcointalk.org/index.php?action=merit;stats=recent to make real-time merit api and notifications, but this page lacks one crucial piece of data - the name and link to the profile of a user who received merit. Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement.

Topic: Can we have one small change on Recent Merits page? (Read 166 times)