Ninjastic.space - BitcoinTalk Post/Address archive + API - page 30.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 06, 2020, 08:48:17 PM

Quote from: alani123 on September 06, 2020, 08:18:44 PM

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

Maybe you can include userIDs in the data dump? Or make a separate data dump for "userID,addresses".

When I search for a simple text ("test"), I quickly get 200 results. However, when I search for a rare text (such as "1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD") without Author or Topic ID, the Search button keeps spinning for what seems like indefinitely. I assume it's still searching, but this might cause a problem with server load at some point.

Quote from: TryNinja on September 07, 2020, 01:49:56 AM

- Post edit history.

If you go to a post's page, you can see the original unedited post and its edited version (edited up to 5 minutes later).

This is cool

Maybe make it 10 minutes though, that's the time limit after which the forum will show a post as edited.

Stalker22

legendary

Activity: 1526

Merit: 1359

Quote from: ?? on ??

So your statement that searching for addresses is no different from searching for ordinary words is incorrect.

Not really. As TryNinja replied, it is probably possible to get some bounty profiles but the results will not be complete or completely accurate.
Anyone who has ever searched for alt accounts of ban evaders and bounty cheaters knows that it requires a bit of creativity as well. Wink

TryNinja

legendary

Activity: 2758

Merit: 6830

New Update!

- Post edit history.

If you go to a post's page, you can see the original unedited post and its edited version (edited up to 5 minutes later). Also, you will be notified by the bot if someone edits the post and mentions you/quotes you later (again, up to 5 min).

Example: https://ninjastic.space/post/55146574

It is possible to get the bounty profiles (at least some of them). I also save the post's board, so I can filter them to the "Bounties" board. I will see if I can take a look at this when I have more time.

Stalker22

legendary

Activity: 1526

Merit: 1359

Quote from: ?? on ??

Of course, it would be great if such a function worked not only on wallet addresses, but also on social media addresses (twitter, facebook, telegram, etc.)

I don't think it's that simple. There is no uniform format for social media addresses. For example, someone can post only their profile name as opposed to the full url to the profile page. Searching for such data is no different than searching for any other words.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: suchmoon on September 06, 2020, 07:37:06 PM

Maybe don't rush with that. Ignoring quotes could be a feature, at least an optional one. Most of the time I would probably want someone the user posted themselves, not when they quoted something.

I already plan to do that. The index I had created was not only removing the quotes but also parts of the original post, so changes are required anyways.

Quote from: suchmoon on September 06, 2020, 07:37:06 PM

However searching for partial words would be great. E.g. "bitcoin" should find "bitcoins". Or perhaps it should be an option too, for those cases where you don't want "ninja" to find "tryninja".

You do get results for "bitcoins" if you search "bitcoin". Most plural words are equal to their singular forms when you do a search. Ninja and TryNinja are too far apart and something like this would probably result in a lot of false positives (like in this case).

Quote from: suchmoon on September 06, 2020, 07:37:06 PM

Let me know if you'd like some help with the unknown titles. I can give you a dump of post IDs and titles that could significantly reduce the number of posts you'd need to re-scrape.

That would be great.

Quote from: alani123 on September 06, 2020, 08:18:44 PM

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

alani123

legendary

Activity: 2422

Merit: 1451

Leading Crypto Sports Betting & Casino Platform

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Quote from: TryNinja on September 06, 2020, 01:24:59 PM

Goddammit, you are right. I messed up big with my algo. I hate regex. Angry

I will fix it and the results should increase considerably for every other search. It does look inside quotes, but the regex is eating a few parts of the post it shouldn't when there are multiple quotes.

Maybe don't rush with that. Ignoring quotes could be a feature, at least an optional one. Most of the time I would probably want someone the user posted themselves, not when they quoted something.

However searching for partial words would be great. E.g. "bitcoin" should find "bitcoins". Or perhaps it should be an option too, for those cases where you don't want "ninja" to find "tryninja".

Let me know if you'd like some help with the unknown titles. I can give you a dump of post IDs and titles that could significantly reduce the number of posts you'd need to re-scrape.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: ?? on ??

I can't even image how difficult it is.

It's just matter of convenience, but it is possible to bold text on the posts that matches the content/keyword? For example, if i use content/keyword "blockchain", all texts containing "blockchain" would be bolded automatically.

I think it is. I'll look at it later today.

Quote from: Aveatrex on September 06, 2020, 07:45:48 AM

Amazing! I see that you are making use of Reactjs, way to go

. I'm interested in your RESTful API, can you elaborate on how it works?

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go  .  I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
    "Other",
    "Meta"
  ],
  "archive": false
}

Quote from: masulum on September 06, 2020, 09:54:15 AM

For me (maybe different experience with other users), background and the text color its doesn't match. if you are happy with that, its okay nothing to change.

It will be fixed. Thanks!

Quote from: suchmoon on September 06, 2020, 10:43:31 AM

Nitpick: for the front-page charts perhaps it makes sense to exclude today otherwise the charts have a bit of a misleading dip at the end there.

Makes sense. I'll remove today's data.

Quote from: suchmoon on September 06, 2020, 10:43:31 AM

Looks great. Very fast. Not sure how accurate it is though. I can't believe I said "cunt" only 7 times. Does it search only outside of quotes? Only full words?

Goddammit, you are right. I messed up big with my algo. I hate regex. Angry

I will fix it and the results should increase considerably for every other search. It does look inside quotes, but the regex is eating a few parts of the post it shouldn't when there are multiple quotes.

suchmoon

legendary

Activity: 3654

Merit: 8909

https://bpip.org

Nitpick: for the front-page charts perhaps it makes sense to exclude today otherwise the charts have a bit of a misleading dip at the end there.

Looks great. Very fast. Not sure how accurate it is though. I can't believe I said "cunt" only 7 times. Does it search only outside of quotes? Only full words?

masulum

legendary

Activity: 2324

Merit: 1604

hmph..

Everything is good, the search post feature is very helpful, to make this feature are different with show the latest post from Bitcointalk, maybe you can add filter posts there. Not an important thing, just additional features maybe will help in the future.

Also, Link color need to change from blue with more soft color. I try to use this style:

Code:

a {
    color: darkorange;
    text-decoration: none;
    background-color: transparent;
    outline: none;
    cursor: pointer;
    -webkit-transition: color .3s;
    transition: color .3s;
    -webkit-text-decoration-skip: objects;
}

For me (maybe different experience with other users), background and the text color its doesn't match. if you are happy with that, its okay nothing to change.

example result:
Test - Test

Aveatrex

sr. member

Activity: 840

Merit: 375

Amazing! I see that you are making use of Reactjs, way to go

. I'm interested in your RESTful API, can you elaborate on how it works?

Stalker22

legendary

Activity: 1526

Merit: 1359

Quote from: ?? on ??

I can't even image how difficult it is.

It's just matter of convenience, but it is possible to bold text on the posts that matches the content/keyword? For example, if i use content/keyword "blockchain", all texts containing "blockchain" would be bolded automatically.

You mean, something like, how bitcointalk search function does? Yeah, that would be nice. I'm sure TryNinja can make the search term highlighted in the results.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 06, 2020, 05:24:01 AM

Some stats won't be available, since I can't get them (~topics created

Just a thought: you can probably get this if you check which user made the first post in each topic.

Quote

The "Unknown Title" shows up for the posts Loyce provided to me. His data didn't include any title, so I would need to scrape everything from zero to get them. New posts and posts which got notified to a bot user have their title.

Most topics have been deleted, so if you scrape those titles, you can skip a few million topics to speed things up.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: ?? on ??

I really like the feature search post, it's more efficient that using google search and have some unique search option.

Thanks! It was hard to make it usable with 47m+ posts, so I'm glad it's working. I had to learn a lot about how databases work and indexes. It's not perfect, though.

Quote from: SFR10 on September 06, 2020, 04:34:25 AM

Is it possible to get the "general statistics for each user ^{[e.g. like this]}"?
^{- I'm after the last two stats (Most Popular Boards By Posts and Activity).}

Yes! Or kinda of. I'm already working on something like this.

Some stats won't be available, since I can't get them (votes casted, polls created, time spent online, topics created, most popular boards per activity, etc...). But you will be able to see stuff like in which boards you posted the most (and when you did it).

Quote from: SFR10 on September 06, 2020, 04:34:25 AM

I noticed it shows "(Unknown Title)" for threads written in languages other than English ^{[not a big deal though]}.

The "Unknown Title" shows up for the posts Loyce provided to me. His data didn't include any title, so I would need to scrape everything from zero to get them. New posts and posts which got notified to a bot user have their title.

Stalker22

legendary

Activity: 1526

Merit: 1359

This is a great tool and I'm really glad I came across this thread. Well done, TryNinja!
This can really make it easier to search forum posts, especially when discovering alt accounts of ban evaders or merit abusers.

Quote from: protrader786 on September 06, 2020, 04:55:42 AM

Hey SRF, can I know how you were able to manage to view user stats please? Or is it just an old archive ? When I click on my profile or others profiles to see the stats, it tells me that user stats are disabled. Please let me know.

I believe it is an old archive. That's why SFR10 asked if TryNinja could do something similar with his data.

protrader786

member

Activity: 104

Merit: 61

Quote from: SFR10 on September 06, 2020, 04:34:25 AM

<..>

Hey SRF, can I know how you were able to manage to view user stats please? Or is it just an old archive ? When I click on my profile or others profiles to see the stats, it tells me that user stats are disabled. Please let me know.

SFR10

legendary

Activity: 2968

Merit: 3406

Crypto Swap Exchange

Quote from: TryNinja on September 05, 2020, 08:07:12 PM

And you can always suggest/ask for a different kind of data. Just post here/PM me.

Is it possible to get the "general statistics for each user ^{[e.g. like this]}"?
^{- I'm after the last two stats (Most Popular Boards By Posts and Activity).}

I noticed it shows "(Unknown Title)" for threads written in languages other than English ^{[not a big deal though]}.
^{- Apart from that, it looks great. Bookmarked.}

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on September 06, 2020, 04:05:31 AM

For archiving purposes, I don't like editing the font colors away from the original.

You have a point. I will change that.

Quote from: LoyceV on September 06, 2020, 04:05:31 AM

You're welcome

Under "archived", it's safe to change "This may or may not represent the time and date the post was made" to something like "This was long after the post was made".
Also: the word is scraped, not scrapped, right?

Thanks. I still make this mistake more often than I should. Grin

Quote from: LoyceV on September 06, 2020, 04:05:31 AM

Quote

- Unedited/archived posts by address

I'll see what I can pull out of your data dump when I have the time

I plan to always make a new dump every few days so anyone can download it when they want/need it. Make sure to take a look at http://public.ninjastic.space to get the latest one when you decide to use it.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on September 05, 2020, 08:07:12 PM

- New darkmode design.

For archiving purposes, I don't like editing the font colors away from the original.

Quote

- Unedited/archived post: https://ninjastic.space/post/55139442

That's easier than https://loyce.club/archive/posts/5513/55139442.html

Databases FTW Shocked

Quote

(thanks to @LoyceV for his oldposts archive).

You're welcome

Under "archived", it's safe to change "This may or may not represent the time and date the post was made" to something like "This was long after the post was made".
Also: the word is scraped, not scrapped, right?

Quote

- Unedited/archived posts by address

I'll see what I can pull out of your data dump when I have the time

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Tytanowy Janusz on September 06, 2020, 02:04:24 AM

Still the same. Put some more "&"

I'm 100% sure you are looking at a cached version of the page. Grin

Try Ctrl + F5 to clear the cache and try once again.

Topic: Ninjastic.space - BitcoinTalk Post/Address archive + API - page 30. (Read 16648 times)