Viewing unedited posts and deleted posts, view per post, per user or per topic - page 7.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: alani123 on March 05, 2020, 10:10:01 AM

I see here that a thread was captured with just the error message for wrong BBcode:
http://loyce.club/archive/posts/5395/53951381.html

Any way to include the content in spite of wrong BBcode?

I don't do BBCode, I only read HTML. This error came from the forum, not from me. I just save the post as it was made before editing.

alani123

legendary

Activity: 2394

Merit: 1412

Leading Crypto Sports Betting & Casino Platform

I see here that a thread was captured with just the error message for wrong BBcode:
http://loyce.club/archive/posts/5395/53951381.html

Any way to include the content in spite of wrong BBcode?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: LoyceV on July 21, 2019, 01:08:59 PM

How to use it

Find the msgID, userID or topicID you need. Let's use msgID 51902990.
Remove the last 4 digits from the msgID to get the directory name (if there are less than 4 digits, use 0): 5190.
Put everything together behind the (above) URL and add ".html": http://loyce.club/archive/posts/5190/51902990.html.

This is an example of how I use it in practice: I copy the topicID (5229466), then type "topics" on my URL-bar. My browser suggests http://loyce.club/archive/topics/, which I select. Then, I paste the topicID, hit Backspace 4 times, type "/", paste the topicID again, type ".html" and hit enter. It takes some getting used to, but in just 5 seconds I have the page I was looking for: http://loyce.club/archive/topics/522/5229466.html.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: ~DefaultTrust on February 22, 2020, 09:22:26 AM

How fast does your parsing work?

See:

Quote from: LoyceV on February 02, 2020, 02:05:42 PM

I expect to complete this around August.

~DefaultTrust

copper member

Activity: 1540

Merit: 487

Stop the war!

Quote from: LoyceV on February 18, 2020, 04:34:49 PM

Quote from: LoyceV on February 02, 2020, 02:05:42 PM

Sneak preview: http://loyce.club/archive/oldposts/

I now have the first 6.1 million Bitcointalk posts archived. Data processing took longer than expected, but it's published now. See link above.

38 million left.
How fast does your parsing work?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: dkbit98 on February 21, 2020, 12:30:48 PM

If it is not a secret, how much data space is needed for all that millions of posts?

I'm currenly using 54 GB for loyce.club, and store 4.2 million files.

Quote from: LoyceV on July 21, 2019, 01:08:59 PM

Viewing unedited/deleted posts

See http://loyce.club/archive/posts/ for all posts (Working!)
New posts are archived within seconds after being created, and instantly available.
See http://loyce.club/archive/members/ for posts made by a certain user (Working!)
Updated every 5 minutes.
See http://loyce.club/archive/topics/ for posts made in a certain topic (Working!)
Updated every 5 minutes.

How to use it

Find the msgID, userID or topicID you need. Let's use msgID 51902990.
Remove the last 4 digits from the msgID to get the directory name (if there are less than 4 digits, use 0): 5190.
Put everything together behind the (above) URL and add ".html": http://loyce.club/archive/posts/5190/51902990.html.

Details

Files are stored with their msgID, userID or topicID as file name. I remove the last 4 digits to create the directory name. Each directory contains up to 10,000 HTML-files. Use CTRL-F to find what you're looking for.
I don't scrape hidden boards (such as Investigations).
I don't keep post titles
I save raw HTML, including quotes
If I run out of disk space, I might create compressed archives per 10,000 posts.
Although I plan to preserve all data, I make no guarantees. Feel free to archive posts.
My current (sponsored) webhost has enough storage space for years to come.
All scrape-times use Amsterdam time (CET).
Usually, I capture at least 99.95% of all posts. Server or internet connection problems can severely reduce this.

Examples

The unedited version of this post: http://loyce.club/archive/posts/5190/51902990.html
(the layout looks better in more recent archived posts)
All posts made by me: http://loyce.club/archive/members/45/459836.html
(obviously only since I started archiving posts)
All posts made in this topic: http://loyce.club/archive/topics/516/5167469.html
(the first posts aren't shown because of the slightly different format used for my archive)

dkbit98

legendary

Activity: 2212

Merit: 7064

Cashback 15%

If it is not a secret, how much data space is needed for all that millions of posts?
And is there a way to use some compression?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Update: I finally got to work on a topic-view. It'll be published at http://loyce.club/archive/topics/, and it's currently crunching data on 1.9 million posts. I'm downloading the thread titles that I don't have yet, and that part takes a lot of time.

This viewer should make it much easier to find back all posts made in a certain thread. Obviously this is also limited to posts made after I started scraping.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Bump!

Quote from: LoyceV on February 02, 2020, 02:05:42 PM

Sneak preview: http://loyce.club/archive/oldposts/

I now have the first 6.1 million posts, I'm currently still processing them for publication. Will be updated soon.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: LoyceV on February 02, 2020, 02:05:42 PM

Sneak preview: http://loyce.club/archive/oldposts/

I now have the first 6.1 million Bitcointalk posts archived. Data processing took longer than expected, but it's published now. See link above.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: Vod on February 09, 2020, 12:01:57 PM

LoyceV, are you saving the raw data, or just converting it?

I'm only saving the raw post (one line from the raw HTML), and my own header like this one (post number, link to post, link to my archive, by username and scraping time).

Quote from: nutildah on February 09, 2020, 12:06:33 PM

Yes, I'm all for it!

It's started already Cheesy

nutildah

legendary

Activity: 2940

Merit: 7892

Quote from: LoyceV on January 20, 2020, 09:02:51 AM

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet. It would be useful for a case like this. It requires scraping a couple million pages, and storing 50+ million posts. I can limit the number of files on the server by storing 10 or 100 posts per page. Would this be useful?

Yes, I'm all for it!

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

LoyceV, are you saving the raw data, or just converting it?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Something failed in the above scrape, the Wall Observer thread stopped scraping after page 2628. I thought I had the "last page detection" working, but there still seems to be a flaw.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Bump

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: LoyceV on January 20, 2020, 09:02:51 AM

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet.

An update: I have started this project! Measured in scraping time, it's the biggest project I ever started. In the past 9 days, I've scraped about 4% of all data, so I expect to complete this around August.
There's also a chance I'll run out of disk space because of the millions of large posts made by bounty spammers, but I'll deal with that when it happens.

Sneak preview: http://loyce.club/archive/oldposts/
How to use:

Find the msgID you need. Let's use 28228
Remove the last 5 digits from the msgID to get the directory name (if there are less than 5 digits, use 0): 0
Replace the last 2 digits of the msgID by xx, and add .html (if there are less than 5 digits, use 0xx): 282xx.html
Add "#msg" and the msgID: #msg28228
Put everything together and go to http://loyce.club/archive/oldposts/0/282xx.html#msg28228

Limitations

Currently, the first 2.1 million posts are available.
I'll scrape the first 5.21 million topics and all posts in there.
That means I'll archive 53.36 million posts, this partially overlaps with my scraper for new posts.
This is a one-time thing, I won't update it with newer posts (I scrape unedited versions for those).
The time "scraped on" is Amsterdam time.

If no username is mentioned, it's either "Anonymous" or "random". I forgot those exist when I started scraping, and it's not important enough to start over.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Bump

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

I've been thinking about expanding my archived posts to all posts that haven't been deleted yet. It would be useful for a case like this. It requires scraping a couple million pages, and storing 50+ million posts. I can limit the number of files on the server by storing 10 or 100 posts per page. Would this be useful?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: logfiles on January 11, 2020, 07:35:28 AM

I am trying to look for archived posts of a particular user from http://loyce.club/archive/members because his account was nuked but the Profile ID doesn't show up

The last update was on 2020-01-04 16:27 but on http://loyce.club/archive/posts the last update was just today, 2020-01-11 12:31. Could it be an irregularity or a technical error preventing an update on the members link.

I still haven't fully tested this part yet, so updates are only manually started. I'm running an update now, it shouldn't take too long to process a week's worth of posts.
Update: see http://loyce.club/archive/members/274/2743460.html

Quote from: TECSHARE on January 11, 2020, 07:42:49 AM

I think you keep merit even if a post is deleted? Some one correct me if I am wrong.

Correct. And TryNinja is also right, I just don't like my "Merit earned for deleted posts"-counter to go up.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TECSHARE on January 11, 2020, 07:42:49 AM

Quote from: LoyceV on December 30, 2019, 03:11:15 PM

Bump (I don't want to delete my previous bump, because it's merited).

I think you keep merit even if a post is deleted? Some one correct me if I am wrong.

You do. He’s not worried about that.

He probably wants to keep the reference for which post earned him his merit. Otherwise, it says it was for post “Deleted”.

Topic: Viewing unedited posts and deleted posts, view per post, per user or per topic - page 7. (Read 8635 times)