Topic: Viewing unedited posts and deleted posts, view per post, per user or per topic - page 10. (Read 8900 times)

If someone posts something on Facebook or Twitter, with privacy settings allowing anyone to see the post with the hashtag #crypto, anyone specifically looking at your profile can see your post, but it is not trivial for someone to obtain all posts containing a hashtag.

There are limitations as to how many tweets can be distributed to entity user per day and per month. The daily number of tweets that can be sent to a third party is large (50,000), but is a small percentage of the total tweets posted every day (500 million). If someone like DPR was posting on twitter instead of bitcointalk, they probably would still have gotten caught, while a HK protestor would probably be safe on twitter, while the HK government (a sockpuppet of the Chinese government) might be investigated similar to how legendster describes if they are posting on bitcointalk.

LoyceV is not the one who invented scraping forum posts, nor is he the only one to be actively scraping posts. There are many ways to download forum posts via automated means, and it is not difficult to get posts into a DataFrame that can later be analyzed.  

The the root cause problem is the administration allows too much access to posts. A straightforward solution is to have an hourly rate limit as to how many page views an individual IP address/range can access on an hourly and daily basis that is something above what a *person* would see in the normal course of reading, but well below the necessary amount of page views required to view all posts. The scraping of posts for non-academic use should also be explicitly prohibited by the administration.

So the Switzerland thing is just a CIA cover? Makes sense now.

You're missing the point. NSA or not NSA, any government agency that is interested in tracking their citizens' involvement in crypto could be doing what LoyceV does and scrape those posts without making an announcement thread here. They could be doing even more, e.g. scrape posts in Investigations.


You cannot "edit out" anything from the internet. The edit button is useful to fix spelling errors and such. It's utterly useless for hiding information, which could have been copied by anyone, could have ended in Bitcoinalk backups (and yes, likely to be handed over to law enforcement if theymos gets a subpoena), etc.

Anything you put on the internet, should be considered public information for the rest of eternity.

Agreed!
I don't scrape Investigations, which is the only place on Bitcointalk that allows DOXing. If someone gets DOXed anywhere else, posts can be reported and the user gets banned. If that happens, feel free to contact me to edit one of the archived posts. When I do that, I'll also edit the filename to make it very obvious it was edited by me, and I'll probably create a log here. Until now, I haven't done this.
Another thing I've been thinking about is if someone posts something that's not allowed by my webhost. If that would happen, I'll have to edit it too.
I'm not sure how Archive.org and the likes handle illegal stuff on their servers.

I'm not sure what that means

Not all posts are archived.

Google caches have an expiry date.

And there is, as far as I know, no way to grab info from Bitcointalk if the info is never quoted by someone else and deleted / edited out. And with Theymos' response rate I don't think any gov agency would have much luck getting him to dish out user data. He's pro anonymity as far as I can tell.

Btw. you seem to think that by gov. I implied only US. Obviously that's not the case.

In countries like Bangladesh the gov does not have the diplomatic resources to use the same tools that NSA offers to the US or any US allies. They employ low-level white hat / grey hat servicemen to do their online digging work. It is the same with many developing and underdeveloped nations.

Besides, I am sure NSA data is accessed in case of national security - NOT detecting / tracking crypto users for political purposes.

Information, shared by anyone on this forum is kind of made with the confidence that they can perhaps edit out any unrequired information in the future - hence the existence of the edit button.

But Loyce's tool is circumventing that basic forum function that the average user takes for granted.

idk guys - this is a big area of grey for me. This is dwelling into the areas of retaining user data, and user privacy - It is a bit uncomfortable for me to endorse it - even though this is great work and can be used to keep the forum clean. But it has a huge potential of misuse and abuse as well.

Loyce - maybe you should think this through and make your stance clear.

You're glowing..

It's not LoyceV's responsibility to solve it. If you post something you shouldn't have - it's your problem. The government doesn't need LoyceV's site, they can just grab the info directly from Bitcointalk, Google cache, archive sites, or NSA hard drives.

A great tool. But I have some thoughts about the possible use and misuse of this.

Let's talk about the positive first.

I am amidst manually collecting data from a few accounts that is operated by one individual. Now, I have an X number of accounts linked with this person but I am sure this person has 2X the number of accounts in total.

I am just being lucky to collect the ones where he has made some errors.

So if I post this half report of these linked accounts, then the person would be alerted and would, in theory, try to delete all his errors from all of his accounts' post history.

Your tool, could stop this. This basically enables you to get the posts made by anyone at the first instance they hit post. Which is great in this kind of example.


But now let's talk about the bad side.

Your tool is freely available for anyone to use. The very same person I am trying to bust for multi-accounting, is also using information scattered about me on this forum to track my identity and send me threatening messages on my telegram, phone etc.

This kind of tool would essentially be a weapon in the hands of such blackmailers and extortioners.

Furthermore,

In our country, our government is banning crypto, so if I posted my information here and if tomorrow some government agency wanted to track me down - then your tool is going to enable them to do that. And possibly land a person in jail for 10 years - just for being associated with crypto.

---

Now, I'd like to hear your thoughts about these 2 situations. What do you intend to do to avoid / solve such issues?

I just checked loyce.club/archive/posts/5233/: it shows 9994 files. That means my scraper missed 6 posts, or 0.06%.
This might have been caused by burst posting (faster than my scraper can handle), or those 6 posts could be in Investigation or some other hidden board that normal users can't access.

Last night, "recent" was broken for 8 hours.  It's the first time I've seen that, theymos didn't post yet what caused it.

As a result, I couldn't scrape anything between post 52295210 and post 52297852. That means I'm missing 2641 posts.

Thanks, I'll check back this time tomorrow (my free time this week is fairly limited)

Regards,

The scraping should be okay now, but I didn't fix the upload yet. I've uploaded an update for just this file.

I didn't notice before, but it shows many duplicate lines. The links to "scraped" don't work either, you'll have to manually change "/posts/posts/" in the URL to "/posts/".

Sorry for this, I haven't had the time to fix this yet.

Is this user's posts able to be updated at all?

There was a bug that overwrote files a few times. Some of the scraped posts were as old as a minute or more.
I fixed the bug, let's see how long it takes to post scrape this.

Update: http://loyce.club/archive/posts/5224/52243599.html was scraped 1 second after posting. Let me know if anything else fails.

This is currently uploading data from the past days. When done, new posts should be available online in less than a minute.
Update August 23: Upload will take a few more hours to catch up.

This part is still disabled.

Does it have anything to do with the recount theymos did? Not sure how that would affect the data LoyceV uses.


Edit: I guess not, just behind the scenes

Great! I was waiting for that

Now see if I can get it back up

Hmm... Website seems to be down, might want to take a look at it...

Well said. Quotable LoyceV.

Of course you are inadvertently insinuating that women never lie. 

I'm thinking eventually it will be handy in trying to compare writing styles between users, in addition to the usual "but you originally said this" type situations.

I know you could probably parse all the text from particular users from the forum itself, but in the format on your server its easier for me to attempt such a thing.

I encourage you to keep it up as long as you can.