Author

Topic: JSON API for merit data (Read 457 times)

sr. member
Activity: 854
Merit: 424
I stand with Ukraine!
January 02, 2020, 06:24:47 PM
#14
OP has started to scrap recent merit transactions to improve the accuracy, and not only depends on weekly merit data dumps from LoyceV - that is what OP intitially depended upon.
hero member
Activity: 1659
Merit: 687
LoyceV on the road. Or couch.
January 02, 2020, 02:00:36 PM
#13
I think the data provided by the userscript is wrong for almost all users.
It's updated once a week, so the difference is the merit earned since last Friday morning.
sr. member
Activity: 1050
Merit: 416
Buy Bitcoin
January 02, 2020, 12:57:08 PM
#12
Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score.

Code:
-skip-
[img]https://i.imgur.com/wOHHo8P.png[/img]

I think the data provided by the userscript is wrong for almost all users. Take your profile for example:



It shows Merit = 1040, eMerit = 529, that means the airdropped merit is 1040-529=511, which is not possible.



My Profile shows Merit = 220, eMerit = 210, so Airdropped Merits = 10, but that's not the case, I started with 0 merits
legendary
Activity: 2310
Merit: 4085
Farewell o_e_l_e_o
January 01, 2020, 09:23:22 AM
#11
Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all.

As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.
I am deeply sorry to you. I mis-remembered what I saw in the image you posted with your script, in the second post. That shows eMerits, not sMerits. Once again, I am sorry because of my carelessness.  Cry


Ok, now I understood you do.

As being said, even if you provide sMerits with your script, there is nothing wrong from you. When people (users) want to use any tool, they can do it positively or negatively, you can control that.
legendary
Activity: 3024
Merit: 2148
January 01, 2020, 07:33:06 AM
#10
I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.

I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.

Anyway, thank you for the script and happy new year, @hatshepsut93.

Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all.

As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.
sr. member
Activity: 1288
Merit: 415
January 01, 2020, 02:49:46 AM
#9
I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

I agree, sMerits data scraping is Ok, but making that public to the level of viewing it under user's profile seems a bit off the track. Theymos would always appreciate keeping it as much as hidden as he can if you see.

I will not be posting a definitive list of merit sources (so that people don't bug them too much), though you'll soon figure out who they are if you pay attention.

Still a good peice of work, and I appreciate the efforts of the OP. But ya making more info easily accessible is not always profitable to the community overall. Merit Sources would just be fed up with begging requests.
legendary
Activity: 2310
Merit: 4085
Farewell o_e_l_e_o
December 31, 2019, 09:24:04 PM
#8
I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.

I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.

Anyway, thank you for the script and happy new year, @hatshepsut93.
legendary
Activity: 3024
Merit: 2148
December 31, 2019, 09:17:32 AM
#7
I have started scraping recent merit transactions, so the accuracy of my API will be significantly improved. I still don't guarantee it being 100% accurate, and at worst you can expect it to be only accurate up to the previous week.
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
September 17, 2019, 01:30:15 PM
#6
I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt?
As far as I know, it's been there since the introduction of Merit. But I'll keep doing my weekly updates anyway.
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
September 17, 2019, 12:55:38 PM
#5
<…> I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week <…>
The issue is probably more on the usability side, than the readiness of it being up-to-date right down to the most recent TXs. The profile and TX data references the inner forum IDs, which is fine, but anyone who is willing to exploit the data still needs to retrieve/scrape the mappings of the UserIDs to names and message IDs to the message hierarchy (i.e. forum path (section, subsection, etc.), post name).
legendary
Activity: 3024
Merit: 2148
September 17, 2019, 12:44:10 PM
#4
Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this.

I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt?

I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week  Embarrassed
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
September 17, 2019, 12:38:03 PM
#3
Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this.
legendary
Activity: 3024
Merit: 2148
September 17, 2019, 11:58:07 AM
#2
Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score.

Code:
// ==UserScript==
// @name     Display earned merit
// @version  1.0
// @grant    none
// @include        https://bitcointalk.org/index.php?topic=*
// @run-at         document-end
// ==/UserScript==

const getMeritsFromApi = async function(ids) {
    return await fetch(`https://bct-quote-notifier.herokuapp.com/api/merit/v1/profiles/?ids=${ids.join(",")}`).then(r =>
        r.json()
    )
}

const insertEmerit = function(meritNode, eMerit) {
    try {
        let eMeritNode = document.createElement("text")
        meritNode.after(eMeritNode)
        meritNode.after(document.createElement("br"))
        eMeritNode.textContent = `eMerit: ${eMerit}`
    } catch (e) {
        console.error(e)
    }
}

const main = async function() {
    let posters = new Map([])
    for (const node of document.querySelectorAll(".poster_info > .smalltext")) {
        const meritNode = [...node.childNodes].find(e => e.textContent.match(/Merit/))

        const userId = [...node.childNodes].find(e => e.href && e.href.match(/action=profile;u=/)).href.split(";u=")[1]

        if (meritNode && userId) {
            let poster = posters.get(userId)
            if (poster) {
                poster.meritNodes.push(meritNode)
            } else {
                posters.set(userId, { meritNodes: [meritNode] })
            }
        }
    }

    let earnedMerits = await getMeritsFromApi([...posters].map(poster => poster[0]))

    earnedMerits
        .filter(e => e !== null)
        .forEach(e => {
            let poster = posters.get(e.id.toString())
            if (poster) {
                poster.earned = e.earned
            }
        })

    for (let [, poster] of posters) {
        try {
            poster.meritNodes.forEach(node => {
                insertEmerit(node, poster.earned || 0)
            })
        } catch (e) {
            console.error(e)
        }
    }
}

main()


legendary
Activity: 3024
Merit: 2148
September 02, 2019, 11:27:21 AM
#1

I have created a public API for merit data, based on LoyceV's log of merit transactions, which is based on theymos's merit dumps. The data is updated once per week.

I didn't tune the rate limits yet, cause I don't know what kind of load can I expect, so refer to the headers if needed.
Jump to: