CloudWatch does seem like a bit more than is necessary, although I'm not familiar with it and have never used it. It probably makes sense to do what suchmoon is saying here, if I'm understanding correctly. We are already storing each job and a timestamp when it was last run, so maybe one page like 'modcheck?mod=flagparse' which checks the DB for timestamps on that particular job, and returns OK/Error based on the result?
I guess we could do the same thing with the logic we are using in the logo as well (like when DB is offline, the logo turns red, if the DB is up and the parsing has stopped, the logo is not spinning). We could just check the DB timestamps and if anything hasn't last run in... lets say 20 minutes to give buffer for any long-running jobs.. the logo spins but turns yellow, giving the indication something inside the parser is not running. Could get more specific here in the length of time and each job, but just a general thought.
Topic: [BPIP] Bitcointalk Public Information Project [Back in Action] - page 25. (Read 21771 times)
September 15, 2020, 05:55:02 AM
September 15, 2020, 05:06:20 AM
AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.
I'm guessing that BPIP is running on ASP.Net Core, from inspecting the request headers sent by it. If that's true then there's a way to directly wire heartbeats to it: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-3.1, though I admit that ASP.Net is not my strong suite so there isn't much else I can recommend.
You could also throw in some functionality that sends an email whenever a module goes down, so the error statuses don't go unnoticed.
September 14, 2020, 08:47:08 PM
The problem here is all of BPIP's updating is happening within programs inside rented servers (I guess?). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.
From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.
From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.
AFAIK CloudWatch has agents to gather info from places like database servers and log files although to be honest I'd prefer to stay off Amazon-specific stuff in case they go full evil and hike the prices up and force us to leave. So for monitoring I was thinking to use a third-party uptime service that could check (among other things) web server's heartbeat and basic status (OK/ERROR) of every important module. That would require just a simple web page so perhaps easier to set up than CloudWatch.
September 14, 2020, 08:37:36 PM
Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
You use AWS services, right?
https://aws.amazon.com/cloudwatch/
The problem here is BPIP is running inside rented servers on AWS, and not across the über hyper-scaled AWS platform (correct me if I'm wrong). The Cloudwatch event somehow has to determine whether BPIP updating is broken and you can't do that without creating a middleware inside the server that collects BPIP diagnostics and send them somewhere like S3 where cloudwatch can access them.
From my development experience, large tasks like this are as laborious and boring to implement, especially if you're not getting paid to do it, as they are necessary and useful.
September 14, 2020, 06:49:40 PM
Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
You use AWS services, right?
https://aws.amazon.com/cloudwatch/
September 14, 2020, 05:09:46 AM
Flaglog is running again, it has added new flags and is currently about 85% of the way through the weekly re-parsing of all flags... no big errors occurred here, just the parser losing connection to the DB and me not realizing it. 
Well, most of the reports pull into the profile page as general information/stats, so just because someone doesn't use the flaglog itself, they could be looking at the number of flags against a user... which will look normal in a situation where the parser has stopped parsing flags, so it might be good to at least have some sort of general alert that a particular parser has stopped, although, I agree it is probably not the highest priority right now... I'll try to check the parser log more often!
The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed
Well, most of the reports pull into the profile page as general information/stats, so just because someone doesn't use the flaglog itself, they could be looking at the number of flags against a user... which will look normal in a situation where the parser has stopped parsing flags, so it might be good to at least have some sort of general alert that a particular parser has stopped, although, I agree it is probably not the highest priority right now... I'll try to check the parser log more often!
September 14, 2020, 03:50:00 AM
we need to set up monitoring to alert us when stuff like that breaks
The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed True, I'm keeping an eye on the site as I'm almost every day checking things so if something fails I will let you guys know as i did it many times before
Out of curiosity, how many unique views the site has per day?
September 14, 2020, 12:47:08 AM
we need to set up monitoring to alert us when stuff like that breaks
The monitoring is done by the users, in this case Ratimov. If it breaks and nobody notices, it's not needed September 13, 2020, 11:58:54 PM
What happened to the flags updates?
I've pinged ibminer, flag parsing is his baby. Something must've gone broken but I'm sure he'll pick it up and fill in missing flags.
If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe.
We won't let terrorism define us
Or to be more accurate, we need to set up monitoring to alert us when stuff like that breaks but that's item #214 on the TODO list.
September 13, 2020, 10:02:53 PM
What happened to the flags updates?
If I were Suchmoon, I would have locked everything down for the Sep 11 19th anniversary. After all, we expect another attack on the 20th, but this year would have surprised us. He was just being safe.
September 12, 2020, 08:37:24 AM
but I don't think it gets processed.
I've still been manually processing this file.. need to get back with suchmoon on adding a task to the server to do this for me but I just hadn't gotten to the recent update yet. It's good now. Thanks for the reminder! September 12, 2020, 03:17:26 AM
Users with a newly created custom Trust list (such as protrader786) don't get a working link to my Trust list viewer yet.
Every week, I create a file user_ids.txt for this, but I don't think it gets processed.
Every week, I create a file user_ids.txt for this, but I don't think it gets processed.
September 01, 2020, 07:04:18 PM
I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
Good catch, thank you. I had a parser crash yesterday and didn't clean up the job queue properly so the merit update got stuck. Should be updated within the next few minutes.
Under normal circumstances there can be a slight discrepancy between sent merits and total merits because those two pieces of information are scraped from two different places. It shouldn't last more than a minute or so.
September 01, 2020, 06:41:20 PM
I'm not sure you changed anything here, but it seems merit count again does not work well.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
For example, user @suchmoon has earned "Merit Score 5497" but in merit information table stay "Has received 4495 ". There is a difference of 2 latest earned merit. I don't count merit assigned at the beginning.
August 31, 2020, 07:02:17 PM
Yes please, it would make things much easier especially now that I need to get additional info from other sources users are referenced by username.
I have added a column for user name, added column headers, and set it up to run around midnight UTC (give or take a few minutes depending on other tasks in the same batch).
https://bpip.org/all_trust.csv
I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved.
IIRC we parse from the receiving end. Your idea seems faster.
August 31, 2020, 01:20:31 PM
Damn I forgot that to view profiles you need to be logged in
Although I've had some troubles with it, it works quite easily by feeding your scraper your browser cookie. My scraper hasn't been bothered by Cloudflare in a long time.That's the biggest advantage of bpip imho. So for getting DT ratings gonna use bpip and to confirm 0 feedback users gonna use LoyceV's trust list. You guys service complement each other
I didn't realize until now that it's relatively easy to get DT Trust scores for all users, and I'm not sure if BPIP uses it already: by checking the sent feedback for all DT-members, you know exactly who has received DT-feedback. It's probably less than 1000 pages to scrape for an update, but it's more work to set it up. I'm currently scraping 13651 profiles per week for an update, so this can be improved. 
August 31, 2020, 08:40:15 AM
I can add a username to the file if it helps you. It would be UTF8.
That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.
Thanks!
plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.Thanks!
August 31, 2020, 07:05:28 AM
That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.
Thanks!
Thanks!
I can add a username to the file if it helps you. It would be UTF8.
I don't want to discourage you from double-checking with other sources, or having a backup if BPIP has an issue, but I think we have the same data that's in the trust dump or LoyceV's list, plus more-frequent-than-weekly updates. At least as far as the DT ratings are concerned. LoyceV obviously maintains non-DT trust lists, which we don't.
August 31, 2020, 06:52:19 AM
Can you give me some examples of users that you can't find? Do those users have DT ratings?
The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.
If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.
The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.
If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.
That makes sense, I just checked some users that couldn't be found and all of them have 0 feedback. Which means that if an user can't be found in this csv it either means that the user has Black trust OR it's an incorrect username/bitcointalk profile url (I'm thinking of a scenario where a bounty manager have a list of users and a bounty hunter provides an incorrect url or username) so gotta differentiate the two cases. To do that I'll do additional checking using theymos trust dump or Loyce's viewer.
Thanks!
August 31, 2020, 05:54:33 AM
because if it's the case from my tests 27k is definitely not enough , like half the users I tested are not found in it 
Can you give me some examples of users that you can't find? Do those users have DT ratings?
The list includes only users who have at least one (negative, positive, or neutral) rating from a DT member, and/or have at least one DT-supported flag, and/or are DT1/2 members themselves.
If a user is not in the list then it should mean they have none of the above (i.e. trust color "black") but if you could get me some examples I can find out what's going on.
Jump to: