Ninjastic.space - BitcoinTalk Post/Address archive + API - page 29.

FatFork

legendary

Activity: 1820

Merit: 2700

Crypto Swap Exchange

Quote from: Aveatrex on September 10, 2020, 03:57:08 PM

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

You can find a useful online API testing tool here: https://reqbin.com/
Just send a request to one of the ninjastic.space endpoints (like these examples TryNinja gave) and watch the responses. You can also find code samples for popular programming languages. Try it. RESTful APIs are quite simple and easy to implement.

TryNinja

legendary

Activity: 2758

Merit: 6830

This ^

You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload. I could also maybe implement some kind of authentication in the future.

Some endpoints:

Post: https://api.ninjastic.space/posts/55141939
Posts: https://api.ninjastic.space/posts/46250414,50646799,55163653,50966531
Posts on topic: https://api.ninjastic.space/posts/topic/5273824

Address: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L
Address authors: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L/authors
Addresses on post: https://api.ninjastic.space/addresses/post/50966531

I will probably create a page to document them better.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: Aveatrex on September 10, 2020, 03:57:08 PM

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

It's a simple HTTP request:

https://api.ninjastic.space/posts/55141939

Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: TryNinja on September 06, 2020, 01:24:59 PM

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go $\"Smiley\"$ . I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
   "Other",
   "Meta"
  ],
  "archive": false
}

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 10, 2020, 02:32:50 AM

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

Thank you!

That's a good idea. I'm indeed thinking about making a user's "profile page" where you will be able to see some stats about anyone's. Possibly stuff like where he posts and when he posts, his addresses, social links, possible connections, etc... all of that will be easier with the changes I'm making. Feel free to give more suggestions!

The number of posts without a title - shown as "(Unknown Title)" - has dropped from 42 millions to just 8k! Chances are that every post you check now has its title.

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 09, 2020, 04:06:12 PM

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

I'll fix that when I finish changing everything to search the new database layer (more below).

Quote from: LoyceV on September 09, 2020, 12:06:05 PM

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

Yes! You are absolutely right. I'll do that.

I found out that updating a lot of data in a database full of indices is a pain in the ass. The database needs to re-create every index, which takes a lot of time (mainly due to the reverse index that allows the "content" search).

I'm currently setting up a second database (using Elasticsearch) which will make searching and data analysing a shit-ton easier and faster. Because of that, the "search post" page will be unavailable for some time in a few hours.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 07, 2020, 02:01:36 PM

I'm also thinking about making more checks, like another one after 1 or 2 hours.

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 08, 2020, 10:42:06 AM

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

Pmalek

legendary

Activity: 2730

Merit: 7065

Quote from: TryNinja on September 08, 2020, 10:42:06 AM

Snip

Cool, thanks for the clarification.
I just translated your topic and posted it in our local Croatian forum. We have a few scam busters and private investigators who could find it useful. Although the majority speaks English, it's still a plus to have a reference in your own language.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Pmalek on September 08, 2020, 10:36:21 AM

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?

Each user gets a random color so you can scroll up through the list and differentiate them easier. It doesn't mean anything.

Quote from: Pmalek on September 08, 2020, 10:36:21 AM

2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

Because I don't know their titles.

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

Pmalek

legendary

Activity: 2730

Merit: 7065

This is getting bookmarked and definitely translated into my local forum for everyone to see when I get around to it.

I found the Addresses feature very interesting so I decided to give it a try.
I did a search on "site:bitcointalk.org avoiding ban" and found this thread by YOSHIE.
I copied the ETH address and ran a search. The results were instant! Very surprised by the speed. It showed all 5 users who posted that address and links to their posts.

I have 2 questions though.

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?
2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 08, 2020, 09:58:38 AM

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

It's ok, you didn't break anything (I think). Grin

I just messed up when importing suchmoon's file with the post titles. But it's now in progress. You just made your searches at the wrong time.

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Quote from: TryNinja on September 08, 2020, 08:04:38 AM

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

Thanks, i hate breaking things up.

TryNinja

legendary

Activity: 2758

Merit: 6830

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

hello TryNinja

I tried to view your webpage but I couldn't pass the loading screen using Edge (a chromium browser) and firefox. I am using adblock and privacy badger.

Edit:
I clicked Search Post and Addresses and it worked. Just the first screen that is spinnig forever.

Lafu

legendary

Activity: 3178

Merit: 3295

Awesome nice work and useful tool buddy Cool

Maybe you can add something in the search post section .
Would be helpful if there would be a Ignore field for Usernames so if you search for some content that the posts from this User dont show up in the results.

Dont know if that is possible and easy to code into the webpage and bot.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on September 07, 2020, 03:30:43 AM

Maybe you can include userIDs in the data dump? Or make a separate data dump for "userID,addresses".

I'll make some changes in the way I store the addresses which will probably allow me to get this data.

Quote from: LoyceV on September 07, 2020, 03:30:43 AM

When I search for a simple text ("test"), I quickly get 200 results. However, when I search for a rare text (such as "1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD") without Author or Topic ID, the Search button keeps spinning for what seems like indefinitely. I assume it's still searching, but this might cause a problem with server load at some point.

You're right. The database should have a timeout for long queries but it wasn't working. It is now. Tongue

You also made me realize that the query was taking too long in some cases when there were little results. For example, pretty much any address search would take forever. Now you can (mostly) search them and it is way faster.

Quote from: TryNinja on September 07, 2020, 01:49:56 AM

This is cool

Maybe make it 10 minutes though, that's the time limit after which the forum will show a post as edited.

I'll think about it. I'm also thinking about making more checks, like another one after 1 or 2 hours.

Quote from: TheBeardedBaby on September 07, 2020, 08:25:01 AM

I don't have enough free time, and sometimes I do research/post from the phone, while in bed, so having a bbcode format of the results from the search to copy from will be a very handy tool at least for me.

Will do! Thanks!

Quote from: mole0815 on September 07, 2020, 09:10:54 AM

I had already mentioned your old website in the German section. If it is desired I can translate the big new update and advertise it in our section.
Your effort and invested time should help as many users as possible

Thank you! I really appreciate that. Wink

Quote from: PrimeNumber7 on September 07, 2020, 11:21:29 AM

~

You gave me some ideas. I won't do it exactly like this since I would need to make some big chances, but I may have found a possible solution.

Edit: Just saw the bot/website received a big donation. Thanks however did it! That pretty much pays the bot for a whole year. Grin

PrimeNumber7

copper member

Activity: 1666

Merit: 1901

Amazon Prime Member #7

Quote from: TryNinja on September 06, 2020, 08:48:17 PM

Quote from: alani123 on September 06, 2020, 08:18:44 PM

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

Your 'posts' table should be as follows:
postID (P) - int
time
UID - int
posttext - str
addressposted - bool (this is optional, but may make searching for addresses easier)
(any other information that does not appear in your DB elsewhere)

You will have an 'addresses' table as follows:
addressID (P) - int (this is an arbitrary number)
address - str
(any other information that does not already appear elsewhere in your DB that you want to keep track of, such as address type, or coin)

postedaddresses table:
ID (P) - int (arbitrary number)
postID - int
addressID - int

Topic: Ninjastic.space - BitcoinTalk Post/Address archive + API - page 29. (Read 16824 times)