Ninjastic.space - BitcoinTalk Post/Address archive + API - page 26.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: suchmoon on September 10, 2020, 05:18:39 PM

It's a simple HTTP request:

https://api.ninjastic.space/posts/55141939

Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.

Oh if it's just a simple HTTP request then I am familiar with that Grin

I thought it was some kind of special interface with mandatory authorization via an api key....

Quote from: TryNinja on September 10, 2020, 05:30:10 PM

You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload.

I'm still working on the bot right now, if I deem it useful enough to release it publicly one day, I will definitely let you know before so we can optimize it.

Quote from: TryNinja on September 10, 2020, 05:30:10 PM

I could also maybe implement some kind of authentication in the future.

I think you should. Otherwise you are just asking for a DoS attack Wink

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Vod on September 10, 2020, 06:15:27 PM

Nice job. I have some ideas that could help in scam busting. I left you merit, but even better than that, I've left you my trust.

Thanks, Vod.

Let me know and I'll implement them if possible.

Quote from: Vod on September 10, 2020, 06:15:27 PM

Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites? Some scammers have deleted hundreds of posts of illegal activities.

It's technically possible, but I'm not sure how hard that would be. I'm priorizing scraping all the live posts that are missing from the database. When everything is working and most features are done, I may think about doing that.

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: TryNinja on September 05, 2020, 09:07:12 PM

The new ninjastic.space website is out! Remade from scratch.

Nice job. I have some ideas that could help in scam busting. I left you merit, but even better than that, I've left you my trust.

Quote from: TryNinja on September 05, 2020, 09:07:12 PM

- The post archive is still incomplete as many posts from this year are missing. It has, however, a lot more posts than its previous version: 42,785,512 posts! Mostly from the previous years. (thanks to @LoyceV for his oldposts archive).

Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites? Some scammers have deleted hundreds of posts of illegal activities.

FatFork

legendary

Activity: 1568

Merit: 2581

Top Crypto Casino

Quote from: Aveatrex on September 10, 2020, 04:57:08 PM

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

You can find a useful online API testing tool here: https://reqbin.com/
Just send a request to one of the ninjastic.space endpoints (like these examples TryNinja gave) and watch the responses. You can also find code samples for popular programming languages. Try it. RESTful APIs are quite simple and easy to implement.

TryNinja

legendary

Activity: 2758

Merit: 6830

This ^

You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload. I could also maybe implement some kind of authentication in the future.

Some endpoints:

Post: https://api.ninjastic.space/posts/55141939
Posts: https://api.ninjastic.space/posts/46250414,50646799,55163653,50966531
Posts on topic: https://api.ninjastic.space/posts/topic/5273824

Address: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L
Address authors: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L/authors
Addresses on post: https://api.ninjastic.space/addresses/post/50966531

I will probably create a page to document them better.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on September 10, 2020, 04:57:08 PM

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

It's a simple HTTP request:

https://api.ninjastic.space/posts/55141939

Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: TryNinja on September 06, 2020, 02:24:59 PM

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go $\"Smiley\"$ . I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
   "Other",
   "Meta"
  ],
  "archive": false
}

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 10, 2020, 03:32:50 AM

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

Thank you!

That's a good idea. I'm indeed thinking about making a user's "profile page" where you will be able to see some stats about anyone's. Possibly stuff like where he posts and when he posts, his addresses, social links, possible connections, etc... all of that will be easier with the changes I'm making. Feel free to give more suggestions!

The number of posts without a title - shown as "(Unknown Title)" - has dropped from 42 millions to just 8k! Chances are that every post you check now has its title.

TheBeardedBaby

legendary

Activity: 2184

Merit: 3134

₿uy / $ell

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 09, 2020, 05:06:12 PM

Quote from: suchmoon on September 09, 2020, 01:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: suchmoon on September 09, 2020, 01:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Quote from: suchmoon on September 09, 2020, 01:04:57 PM

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

I'll fix that when I finish changing everything to search the new database layer (more below).

Quote from: LoyceV on September 09, 2020, 01:06:05 PM

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

Yes! You are absolutely right. I'll do that.

I found out that updating a lot of data in a database full of indices is a pain in the ass. The database needs to re-create every index, which takes a lot of time (mainly due to the reverse index that allows the "content" search).

I'm currently setting up a second database (using Elasticsearch) which will make searching and data analysing a shit-ton easier and faster. Because of that, the "search post" page will be unavailable for some time in a few hours.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 07, 2020, 03:01:36 PM

I'm also thinking about making more checks, like another one after 1 or 2 hours.

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 08, 2020, 11:42:06 AM

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

Pmalek

legendary

Activity: 2730

Merit: 7065

Farewell, Leo. You will be missed!

Quote from: TryNinja on September 08, 2020, 11:42:06 AM

Snip

Cool, thanks for the clarification.
I just translated your topic and posted it in our local Croatian forum. We have a few scam busters and private investigators who could find it useful. Although the majority speaks English, it's still a plus to have a reference in your own language.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Pmalek on September 08, 2020, 11:36:21 AM

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?

Each user gets a random color so you can scroll up through the list and differentiate them easier. It doesn't mean anything.

Quote from: Pmalek on September 08, 2020, 11:36:21 AM

2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

Because I don't know their titles.

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

Pmalek

legendary

Activity: 2730

Merit: 7065

Farewell, Leo. You will be missed!

This is getting bookmarked and definitely translated into my local forum for everyone to see when I get around to it.

I found the Addresses feature very interesting so I decided to give it a try.
I did a search on "site:bitcointalk.org avoiding ban" and found this thread by YOSHIE.
I copied the ETH address and ran a search. The results were instant! Very surprised by the speed. It showed all 5 users who posted that address and links to their posts.

I have 2 questions though.

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?
2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 08, 2020, 10:58:38 AM

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

It's ok, you didn't break anything (I think). Grin

I just messed up when importing suchmoon's file with the post titles. But it's now in progress. You just made your searches at the wrong time.

TheBeardedBaby

legendary

Activity: 2184

Merit: 3134

₿uy / $ell

Quote from: TryNinja on September 08, 2020, 09:04:38 AM

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

Thanks, i hate breaking things up.

TryNinja

legendary

Activity: 2758

Merit: 6830

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

bitmover

legendary

Activity: 2212

Merit: 5622

Non-custodial BTC Wallet

hello TryNinja

I tried to view your webpage but I couldn't pass the loading screen using Edge (a chromium browser) and firefox. I am using adblock and privacy badger.

Edit:
I clicked Search Post and Addresses and it worked. Just the first screen that is spinnig forever.

Topic: Ninjastic.space - BitcoinTalk Post/Address archive + API - page 26. (Read 15134 times)