Ninjastic.space - BitcoinTalk Post/Address archive + API - page 29.

Aveatrex

sr. member

Activity: 840

Merit: 375

Quote from: TryNinja on September 06, 2020, 01:24:59 PM

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go $\"Smiley\"$ . I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
   "Other",
   "Meta"
  ],
  "archive": false
}

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 10, 2020, 02:32:50 AM

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

Thank you!

That's a good idea. I'm indeed thinking about making a user's "profile page" where you will be able to see some stats about anyone's. Possibly stuff like where he posts and when he posts, his addresses, social links, possible connections, etc... all of that will be easier with the changes I'm making. Feel free to give more suggestions!

The number of posts without a title - shown as "(Unknown Title)" - has dropped from 42 millions to just 8k! Chances are that every post you check now has its title.

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 09, 2020, 04:06:12 PM

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Quote from: suchmoon on September 09, 2020, 12:04:57 PM

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

I'll fix that when I finish changing everything to search the new database layer (more below).

Quote from: LoyceV on September 09, 2020, 12:06:05 PM

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

Yes! You are absolutely right. I'll do that.

I found out that updating a lot of data in a database full of indices is a pain in the ass. The database needs to re-create every index, which takes a lot of time (mainly due to the reverse index that allows the "content" search).

I'm currently setting up a second database (using Elasticsearch) which will make searching and data analysing a shit-ton easier and faster. Because of that, the "search post" page will be unavailable for some time in a few hours.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 07, 2020, 02:01:36 PM

I'm also thinking about making more checks, like another one after 1 or 2 hours.

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 08, 2020, 10:42:06 AM

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

Pmalek

legendary

Activity: 2730

Merit: 7065

Quote from: TryNinja on September 08, 2020, 10:42:06 AM

Snip

Cool, thanks for the clarification.
I just translated your topic and posted it in our local Croatian forum. We have a few scam busters and private investigators who could find it useful. Although the majority speaks English, it's still a plus to have a reference in your own language.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Pmalek on September 08, 2020, 10:36:21 AM

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?

Each user gets a random color so you can scroll up through the list and differentiate them easier. It doesn't mean anything.

Quote from: Pmalek on September 08, 2020, 10:36:21 AM

2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

Because I don't know their titles.

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

Pmalek

legendary

Activity: 2730

Merit: 7065

This is getting bookmarked and definitely translated into my local forum for everyone to see when I get around to it.

I found the Addresses feature very interesting so I decided to give it a try.
I did a search on "site:bitcointalk.org avoiding ban" and found this thread by YOSHIE.
I copied the ETH address and ran a search. The results were instant! Very surprised by the speed. It showed all 5 users who posted that address and links to their posts.

I have 2 questions though.

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?
2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: TheBeardedBaby on September 08, 2020, 09:58:38 AM

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

It's ok, you didn't break anything (I think). Grin

I just messed up when importing suchmoon's file with the post titles. But it's now in progress. You just made your searches at the wrong time.

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

Quote from: TryNinja on September 08, 2020, 08:04:38 AM

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

Thanks, i hate breaking things up.

TryNinja

legendary

Activity: 2758

Merit: 6830

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

bitmover

legendary

Activity: 2352

Merit: 6089

bitcoindata.science

hello TryNinja

I tried to view your webpage but I couldn't pass the loading screen using Edge (a chromium browser) and firefox. I am using adblock and privacy badger.

Edit:
I clicked Search Post and Addresses and it worked. Just the first screen that is spinnig forever.

Lafu

legendary

Activity: 3136

Merit: 3213

Awesome nice work and useful tool buddy Cool

Maybe you can add something in the search post section .
Would be helpful if there would be a Ignore field for Usernames so if you search for some content that the posts from this User dont show up in the results.

Dont know if that is possible and easy to code into the webpage and bot.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on September 07, 2020, 03:30:43 AM

Maybe you can include userIDs in the data dump? Or make a separate data dump for "userID,addresses".

I'll make some changes in the way I store the addresses which will probably allow me to get this data.

Quote from: LoyceV on September 07, 2020, 03:30:43 AM

When I search for a simple text ("test"), I quickly get 200 results. However, when I search for a rare text (such as "1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD") without Author or Topic ID, the Search button keeps spinning for what seems like indefinitely. I assume it's still searching, but this might cause a problem with server load at some point.

You're right. The database should have a timeout for long queries but it wasn't working. It is now. Tongue

You also made me realize that the query was taking too long in some cases when there were little results. For example, pretty much any address search would take forever. Now you can (mostly) search them and it is way faster.

Quote from: TryNinja on September 07, 2020, 01:49:56 AM

This is cool

Maybe make it 10 minutes though, that's the time limit after which the forum will show a post as edited.

I'll think about it. I'm also thinking about making more checks, like another one after 1 or 2 hours.

Quote from: TheBeardedBaby on September 07, 2020, 08:25:01 AM

I don't have enough free time, and sometimes I do research/post from the phone, while in bed, so having a bbcode format of the results from the search to copy from will be a very handy tool at least for me.

Will do! Thanks!

Quote from: mole0815 on September 07, 2020, 09:10:54 AM

I had already mentioned your old website in the German section. If it is desired I can translate the big new update and advertise it in our section.
Your effort and invested time should help as many users as possible

Thank you! I really appreciate that. Wink

Quote from: PrimeNumber7 on September 07, 2020, 11:21:29 AM

~

You gave me some ideas. I won't do it exactly like this since I would need to make some big chances, but I may have found a possible solution.

Edit: Just saw the bot/website received a big donation. Thanks however did it! That pretty much pays the bot for a whole year. Grin

PrimeNumber7

copper member

Activity: 1652

Merit: 1901

Amazon Prime Member #7

Quote from: TryNinja on September 06, 2020, 08:48:17 PM

Quote from: alani123 on September 06, 2020, 08:18:44 PM

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

Your 'posts' table should be as follows:
postID (P) - int
time
UID - int
posttext - str
addressposted - bool (this is optional, but may make searching for addresses easier)
(any other information that does not appear in your DB elsewhere)

You will have an 'addresses' table as follows:
addressID (P) - int (this is an arbitrary number)
address - str
(any other information that does not already appear elsewhere in your DB that you want to keep track of, such as address type, or coin)

postedaddresses table:
ID (P) - int (arbitrary number)
postID - int
addressID - int

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 06, 2020, 08:48:17 PM

Quote from: suchmoon on September 06, 2020, 07:37:06 PM

Let me know if you'd like some help with the unknown titles. I can give you a dump of post IDs and titles that could significantly reduce the number of posts you'd need to re-scrape.

That would be great.

https://bpip.org/titles_20200907.zip

Format is CSV because it's relatively compact for this volume of data. The zip file is ~800MB, uncompressed size ~3GB. It'd be possible to make it smaller by grouping the titles together ("Title",) since most replies typically have the same title but it would probably be a pain to import. Let me know if you prefer a different format.

Sample (note the double quote escapes):

Code:

"PostId","Title"
28,Welcome to the new Bitcoin forum!
29,Repost: Bitcoin Maturation
30,Repost: Request: Make this anonymous?
31,Re: Repost: Bitcoin Maturation
32,Re: Repost: Request: Make this anonymous?
33,Repost: How anonymous are bitcoins?
34,Re: Repost: How anonymous are bitcoins?
36,Repost: Linux/UNIX compile
37,Re: Repost: Linux/UNIX compile
38,[OLD THREAD] Bitcoin version 0.2 development status
40,A few suggestions
41,Re: A few suggestions
42,Re: A few suggestions
43,Questions about Bitcoin
44,Re: A few suggestions
45,Re: A few suggestions
46,Re: Questions about Bitcoin
47,Re: A few suggestions
48,Re: Questions about Bitcoin
49,Re: Questions about Bitcoin
50,Re: A few suggestions
51,Re: A few suggestions
52,Re: A few suggestions
53,Break on the supply's increase
54,Re: A few suggestions
55,Re: A few suggestions
56,Re: Break on the supply's increase
57,Re: A few suggestions
58,Re: A few suggestions
59,Re: A few suggestions
60,Re: A few suggestions
61,Re: A few suggestions
62,Re: A few suggestions
63,Re: A few suggestions
64,Re: A few suggestions
65,"New Exchange Service: ""BTC 2 PSC"""
66,Re: A few suggestions
67,Re: A few suggestions
68,Re: A few suggestions
69,Re: A few suggestions
70,Re: A few suggestions
71,Re: A few suggestions
72,Re: A few suggestions
73,Bitcoin 0.2 released!
74,Re: Bitcoin 0.2 released!
75,FreeBSD build patch
76,Re: A few suggestions
77,Re: A few suggestions
78,Re: A few suggestions
79,Re: A few suggestions
81,Re: A few suggestions
82,"Re: New Exchange Service: ""BTC 2 PSC"""
83,Is my second Transaction working correctly? +Transfer Question
84,Re: Is my second Transaction working correctly? +Transfer Question
85,Re: Is my second Transaction working correctly? +Transfer Question
86,Re: Is my second Transaction working correctly? +Transfer Question
87,"64bit support"
88,Re: Bitcoin 0.2 released!
90,Web UI ideas
91,Re: Web UI ideas
92,Re: Web UI ideas
93,Re: Web UI ideas
94,Re: Web UI ideas
95,"Re: New Exchange Service: ""BTC 2 PSC"""
96,"Re: New Exchange Service: ""BTC 2 PSC"""
97,Re: 64bit support
100,New exchange (Bitcoin Market)

mole0815

staff

Activity: 2548

Merit: 2709

Join the world-leading crypto sportsbook NOW!

I had already mentioned your old website in the German section. If it is desired I can translate the big new update and advertise it in our section.
Your effort and invested time should help as many users as possible

TheBeardedBaby

legendary

Activity: 2240

Merit: 3150

₿uy / $ell ..oeleo ;(

This is amazing tool, I love it.
Sorry, No more merit to give... and theymos is silent on my Merit Source Application Sad

I'll add as small suggestion.

I don't have enough free time, and sometimes I do research/post from the phone, while in bed, so having a bbcode format of the results from the search to copy from will be a very handy tool at least for me.

Topic: Ninjastic.space - BitcoinTalk Post/Address archive + API - page 29. (Read 16648 times)