I've reduced my scraping frequency for new posts (to reduce unnecessary server load). I created my posts archive when there were many more posts per day, and the recent page only shows the last 10 so I have to be quick.
If I start missing more posts, that's either because it's spam and MindlessElectron deleted it before I scraped them, or because there were many posts in a short interval. I'll keep an eye on this.
Topic: Viewing unedited posts and deleted posts, view per post, per user or per topic - page 2. (Read 8900 times)
December 31, 2022, 02:37:17 AM
June 21, 2022, 03:01:10 AM
April 23, 2022, 03:01:59 AM
I've censored the following posts in my archive because of doxing (as requested here):
The first post has been removed from Bitcointalk, the other 3 posts have been edited by theymos. I didn't bother highlighting the changed, I just wiped them.
The first post has been removed from Bitcointalk, the other 3 posts have been edited by theymos. I didn't bother highlighting the changed, I just wiped them.
February 23, 2022, 05:50:41 AM
Quote
(Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)
You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there.Yeah, I figured already that multiple IPs would be unwelcome since it basically circumvents the 1sec restriction.
February 23, 2022, 03:01:47 AM
Just for clarification, that time is assuming there's a 1-second sleep between requests, right?
Yes.Quote
(Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)
You'll get a warning page if you go to fast. I've also seen bans for bots that use multiple IPs so don't go there. February 22, 2022, 08:28:18 PM
The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.
Just for clarification, that time is assuming there's a 1-second sleep between requests, right? (Theymos said he'd ban IP addresses that stress the server out with more frequent requests so that's what I have in my scraper)
February 22, 2022, 01:07:00 PM
Take for example "public key addition" https://ninjastic.space/search?content=public%20key%20addition there is only 1 result on the page with the exact words in order.
I get 2,376 results on that link. If I add double quotes, I get 8 results.Quote
For this example I was expecting some of my posts about public key addition to appear.
If I search my archive on your posts, I get only one match for "public key addition":https://loyce.club/archive/posts/5933/59330710.html
That's your post above. If I would download your post history, I may find it again if you edited those posts later.
The problem is we can't know which posts are edited, without checking all posts again. And downloading all posts takes about half a year.
February 22, 2022, 12:55:37 PM
So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.
That's exactly what I use it for (https://ninjastic.space/search). Maybe it doesn't suit your needs? No, unfortunately. The search is sometimes not specific just like Bitcointalk search. Take for example "public key addition" https://ninjastic.space/search?content=public%20key%20addition there is only 1 result on the page with the exact words in order. For this example I was expecting some of my posts about public key addition to appear.
February 22, 2022, 10:04:46 AM
So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.
That's exactly what I use it for (https://ninjastic.space/search). Maybe it doesn't suit your needs? February 22, 2022, 09:49:11 AM
I am trying to scrape posts from bitcointalk to load into my Elasticsearch cluster but I have no idea when to make my script check for new posts.
Any specific reason why you need them on an ES cluster? That's something I already do to run the ninjastic.space searching.So that I can make a better Bitcointalk search engine. I can't stand the built-in one on Simple Machines Forum - the search is slow (sometimes they take over a minute for results to load - if they even do) and the results are never specific enough.
February 21, 2022, 10:16:02 AM
I am trying to scrape posts from bitcointalk to load into my Elasticsearch cluster but I have no idea when to make my script check for new posts.
Any specific reason why you need them on an ES cluster? That's something I already do to run the ninjastic.space searching. February 21, 2022, 10:12:40 AM
@LoyceV how does your bot know when a new message is posted on bitcointalk?
Check Recent Posts often enough.Quote
I am trying to scrape posts from bitcointalk to load into my Elasticsearch cluster
Maybe @TryNinja can get you a copy of his posts archive. I could do it too, but mine is in 2 different formats so more work to figure out. February 21, 2022, 09:52:23 AM
@LoyceV how does your bot know when a new message is posted on bitcointalk? Last time I checked, you could only get a message ID a link with the topic ID inside it, but it is non-trivial to find the topic ID.
I am trying to scrape posts from bitcointalk to load into my Elasticsearch cluster but I have no idea when to make my script check for new posts.
I am trying to scrape posts from bitcointalk to load into my Elasticsearch cluster but I have no idea when to make my script check for new posts.
December 14, 2021, 03:50:55 AM
Full disclosure: I censored my own post in my archive, because I messed up quoting myself and didn't intend to share the address publicly.
October 30, 2021, 06:12:56 AM
Full disclosure: I censored 2 posts (as requested by witcher_sense) because of possible doxing:
https://loyce.club/archive/oldposts/459/459551xx.html#msg45955145
https://loyce.club/archive/oldposts/459/459487xx.html#msg45948759
https://loyce.club/archive/oldposts/459/459551xx.html#msg45955145
https://loyce.club/archive/oldposts/459/459487xx.html#msg45948759
August 04, 2021, 07:33:26 AM
Full disclosure: I temporarily censored post 57612715 in my archive. This was requested by the author (Royse777).
I don't like censoring data, and see no reason to permanently delete this. However, since it's part of an active scam investigation, it might help to keep it quiet for now. So I've removed it, and will restore the original archive in 90 days.
I don't like censoring data, and see no reason to permanently delete this. However, since it's part of an active scam investigation, it might help to keep it quiet for now. So I've removed it, and will restore the original archive in 90 days.
July 06, 2021, 04:07:30 AM
Theymos manually changed the ownership of 2 (deleted) posts:
Post 39334982 created by ChainEX now shows as created by 1miau.
Post 57307359 created by skillscreating now shows as created by Pmalek.
Done.
Posts are ordered in threads by message ID. What we do in these cases is that we repurpose the ID of a deleted post which lies in the appropriate range.
My records still show the original post.Posts are ordered in threads by message ID. What we do in these cases is that we repurpose the ID of a deleted post which lies in the appropriate range.
Post 39334982 created by ChainEX now shows as created by 1miau.
Post 57307359 created by skillscreating now shows as created by Pmalek.
June 12, 2021, 08:27:13 AM
I'm temporarily halting updates for the list per user and per topic, so I can merge the 50+ million older posts with the current version.
I've finished merging posts. Update are working again!Now all "users" and "topics" list have contain all posts, including the posts made long before I joined the forum.
June 01, 2021, 04:47:01 AM
I'm temporarily halting updates for the list per user and per topic, so I can merge the 50+ million older posts with the current version.
I don't know yet how long this is going to take.
Update: There are now some duplicates in my posts archive:
This is going to take a bit longer to fix.
Update: The "per user" archives are working again. The "per topic" archives have more problems. I'lll restart those from scratch in the weekend.
Update: At the rate this is going, it's going to take at least a week to complete.
Update (June 9): it's going to take another week.
Update: There are now some duplicates in my posts archive:
Code:
9118. Post 51903915 (scraped 2020-07-08_Wed_09.55h, might have been edited)
9119. Post 51903915 (by LoyceV) (scraped Sun Jul 21 20:37:55 CEST 2019)
9120. Post 51903915 (by LoyceV) (scraped Sun Jul 21 20:37:55 CEST 2019)
Update: The "per user" archives are working again. The "per topic" archives have more problems. I'lll restart those from scratch in the weekend.
Update: At the rate this is going, it's going to take at least a week to complete.
Update (June 9): it's going to take another week.
Jump to: