[BPIP] Bitcointalk Public Information Project [Back in Action] - page 25.

ibminer

legendary

Activity: 1789

Merit: 2535

Goonies never say die.

You make me feel so young.
After the physical and emotional stress I've endured so far in 2020, I'll estimate I will have aged 15+ years by 2021. lol

So what are the odds the active members drop is related to COVID-19?... or, maybe just someone's account farm went offline or something? Where's the poll?? lol Cheesy

.. BPIP should really get some historical tracking going with those stats ..

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Active members was hovering above 90k for a few days, but it just dropped below. :/

When I was your age, we didn't have these fancy websites to warn us things were failing. Sometimes they'd fail right when we were using 'em...

ibminer

legendary

Activity: 1789

Merit: 2535

Goonies never say die.

CloudWatch does seem like a bit more than is necessary, although I'm not familiar with it and have never used it. It probably makes sense to do what suchmoon is saying here, if I'm understanding correctly. We are already storing each job and a timestamp when it was last run, so maybe one page like 'modcheck?mod=flagparse' which checks the DB for timestamps on that particular job, and returns OK/Error based on the result?

I guess we could do the same thing with the logic we are using in the logo as well (like when DB is offline, the logo turns red, if the DB is up and the parsing has stopped, the logo is not spinning). We could just check the DB timestamps and if anything hasn't last run in... lets say 20 minutes to give buffer for any long-running jobs.. the logo spins but turns yellow, giving the indication something inside the parser is not running. Could get more specific here in the length of time and each job, but just a general thought.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: suchmoon on September 14, 2020, 08:47:08 PM

AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.

I'm guessing that BPIP is running on ASP.Net Core, from inspecting the request headers sent by it. If that's true then there's a way to directly wire heartbeats to it: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-3.1, though I admit that ASP.Net is not my strong suite so there isn't much else I can recommend.

You could also throw in some functionality that sends an email whenever a module goes down, so the error statuses don't go unnoticed.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: NotATether on September 14, 2020, 08:37:36 PM

The problem here is all of BPIP's updating is happening within programs inside rented servers (I guess?). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.

From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.

AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: Vod on September 14, 2020, 06:49:40 PM

Quote from: suchmoon on September 13, 2020, 11:58:54 PM

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

You use AWS services, right?
https://aws.amazon.com/cloudwatch/

The problem here is BPIP is running inside rented servers on AWS, and not across the über hyper-scaled AWS platform (correct me if I'm wrong). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.

From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: suchmoon on September 13, 2020, 11:58:54 PM

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

You use AWS services, right?
https://aws.amazon.com/cloudwatch/

ibminer

legendary

Activity: 1789

Merit: 2535

Goonies never say die.

Flaglog is running again, it has added new flags and is currently about 85% of the way through the weekly re-parsing of all flags... no big errors occurred here, just the parser losing connection to the DB and me not realizing it. Embarrassed

Quote from: LoyceV on September 14, 2020, 12:47:08 AM

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

Well, most of the reports pull into the profile page as general information/stats, so just because someone doesn't use the flaglog itself, they could be looking at the number of flags against a user... which will look normal in a situation where the parser has stopped parsing flags, so it might be good to at least have some sort of general alert that a particular parser has stopped, although, I agree it is probably not the highest priority right now... I'll try to check the parser log more often!

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Quote from: LoyceV on September 14, 2020, 12:47:08 AM

Quote from: suchmoon on September 13, 2020, 11:58:54 PM

we need to set up monitoring to alert us when stuff like that breaks

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

True, I'm keeping an eye on the site as I'm almost every day checking things so if something fails I will let you guys know as i did it many times before

Out of curiosity, how many unique views the site has per day?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: suchmoon on September 13, 2020, 11:58:54 PM

we need to set up monitoring to alert us when stuff like that breaks

The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: ?? on ??

What happened to the flags updates?

I've pinged ibminer, flag parsing is his baby. Something must've gone broken but I'm sure he'll pick it up and fill in missing flags.

Quote from: Vod on September 13, 2020, 10:02:53 PM

If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe.

We won't let terrorism define us

Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: ?? on ??

What happened to the flags updates?

If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe.

ibminer

legendary

Activity: 1789

Merit: 2535

Goonies never say die.

Quote from: LoyceV on September 12, 2020, 03:17:26 AM

but I don't think it gets processed.

I've still been manually processing this file.. need to get back with suchmoon on adding a task to the server to do this for me but I just hadn't gotten to the recent update yet. It's good now. Thanks for the reminder!

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Users with a newly created custom Trust list (such as protrader786) don't get a working link to my Trust list viewer yet.
Every week, I create a file user_ids.txt for this, but I don't think it gets processed.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: examplens on September 01, 2020, 06:41:20 PM

I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.

Good catch, thank you. I had a parser crash yesterday and didn't clean up the job queue properly so the merit update got stuck. Should be updated within the next few minutes.

Under normal circumstances there can be a slight discrepancy between sent merits and total merits because those two pieces of information are scraped from two different places. It shouldn't last more than a minute or so.

examplens

legendary

Activity: 3472

Merit: 3507

Crypto Swap Exchange

I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 31, 2020, 08:40:15 AM

Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.

I have added a column for user name, added column headers, and set it up to run around midnight UTC (give or take a few minutes depending on other tasks in the same batch).

https://bpip.org/all_trust.csv

Quote from: LoyceV on August 31, 2020, 01:20:31 PM

I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.

IIRC we parse from the receiving end. Your idea seems faster.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: Aveatrex on August 31, 2020, 03:33:51 AM

Damn I forgot that to view profiles you need to be logged in

Although I've had some troubles with it, it works quite easily by feeding your scraper your browser cookie. My scraper hasn't been bothered by Cloudflare in a long time.

Quote from: Aveatrex on August 31, 2020, 08:40:15 AM

That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other

I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on August 31, 2020, 07:05:28 AM

I can add a username to the file if it helps you. It would be UTF8.

Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.

Quote from: suchmoon on August 31, 2020, 07:05:28 AM

Quote from: Aveatrex on August 31, 2020, 06:52:19 AM

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.

That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on August 31, 2020, 06:52:19 AM

That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.

Thanks!

I can add a username to the file if it helps you. It would be UTF8.

I don't want to discourage you from double-checking with other sources, or having a backup if BPIP has an issue, but I think we have the same data that's in the trust dump or LoyceV's list, plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.

Topic: [BPIP] Bitcointalk Public Information Project [Back in Action] - page 25. (Read 21862 times)