Ninjastic.space - BitcoinTalk Post/Address archive + API - page 9.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: decodx on August 12, 2022, 10:27:15 AM

the last 1200 or so posts have not been scraped.

Don't worry, I have a backup Tongue

Seriously though: @TryNinja: let me know which posts you missed, and I'll get you a compressed file once your scraper is back. Let's face it: your search system is much better

decodx

hero member

Activity: 1414

Merit: 915

🇺🇦 Glory to Ukraine!

@TryNinja, I think the Ninjastic.space scraper bot may be down.

Last post scrapped: 60738192 posted on 2022-08-12 10:56:09 UTC
The forum is currently at: 60739433, which means that the last 1200 or so posts have not been scraped.

NeuroticFish

legendary

Activity: 3500

Merit: 6205

Looking for campaign manager? Contact icopress!

Quote from: LoyceV on July 27, 2022, 10:54:09 AM

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

I've missed this and it makes perfect sense. Thanks for the explanation.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on July 27, 2022, 11:05:39 AM

AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing.

I just checked for this post, and I found it back in Recent Posts. But only if you log in, which I assume your scraper doesn't do for this.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on July 27, 2022, 10:54:09 AM

Quote from: JeromeTash on July 27, 2022, 10:40:00 AM

I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

Yep.

AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: JeromeTash on July 27, 2022, 10:40:00 AM

I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

JeromeTash

legendary

Activity: 2100

Merit: 1208

Heisenberg

Quote from: NeuroticFish on July 27, 2022, 10:13:21 AM

I have some sort of bug report, I think. It may be nothing, or just a glitch in the matrix, I don't know.

It has 2 parts:
1. If I search for: author GazetaBitcoin, post title paraipan, it gives no returns, hence missing https://bitcointalksearch.org/topic/--5266812
Also if I search for: author GazetaBitcoin, post content paraipan, it doesn't return that topic either, and also for example it's missing https://bitcointalksearch.org/topic/--5135232

It's probably because the post was made in Investigations board. If you remember, Investigations board is off limits to those who have not logged in and newbie ranked users. I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

NeuroticFish

legendary

Activity: 3500

Merit: 6205

Looking for campaign manager? Contact icopress!

I have some sort of bug report, I think. It may be nothing, or just a glitch in the matrix, I don't know.

It has 2 parts:
1. If I search for: author GazetaBitcoin, post title paraipan, it gives no returns, hence missing https://bitcointalksearch.org/topic/--5266812
Also if I search for: author GazetaBitcoin, post content paraipan, it doesn't return that topic either, and also for example it's missing https://bitcointalksearch.org/topic/--5135232

2. For the last 2-3 days not all the mention notifications seem to come and some 6 merits are missed too (meaning 2 or 3 meriting events)

Both are no biggie, you've done great job on both tools, still, I thought that you may want to know about this.

Rikafip

legendary

Activity: 1722

Merit: 5937

Quote from: TryNinja on July 17, 2022, 03:02:37 PM

Like @indah rezqi said above, it's an arbitrary number to keep everything running fine. But since there hasn't been any abuse, I'm uncapping it (at least for now). Let me know if it works. Cheesy

Thanks for uncapping it!

I did some testing on Bitcoin Discussion board (date range set for the whole 2018) and it works as long as I set the cap at 32,000 users. Everything above that (or not setting it at all) gives me an error after some time of processing it but don't know if cause of error is on my side.

Quote from: skarais on July 17, 2022, 03:22:06 PM

I started testing it for the first time, now it's over 10K. From these results, there are 10997 users who actively post on bitcoin discussion boards including child boards during 2020. I think the tool is better now, is there still a limit to the number?

I just did a little bit of comparison regarding active users in the first 7 months (up until July 17th) of 2018 and in the same period during this year and difference is huge: 29815 users active in Bitcoin Discussion board (including child boards) during that period compared to only 3204 in 2022. I did similar comparison for my local board some time ago and results were pretty much similar, decline was around 90% as well.

skarais

legendary

Activity: 2464

Merit: 2094

Quote from: TryNinja on July 17, 2022, 03:44:19 PM

10.000.000, maybe a bit too much...? Cheesy

I tried running it for the entire archive (no filters) and I gave up before the query could finish.

Hahaha, by the way, thanks for the latest upgrade for your awesome tool. I almost asked you for this improvement when updating the board's local activity stats last month, but I forgot about it so quickly. But @Rikafip do it here, everything was fine. Thanks, mate. Cheesy

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: skarais on July 17, 2022, 03:22:06 PM

]I think the tool is better now, is there still a limit to the number?

10.000.000, maybe a bit too much...? Cheesy

I tried running it for the entire archive (no filters) and I gave up before the query could finish.

skarais

legendary

Activity: 2464

Merit: 2094

Quote from: TryNinja on July 17, 2022, 03:02:37 PM

Like @indah rezqi said above, it's an arbitrary number to keep everything running fine. But since there hasn't been any abuse, I'm uncapping it (at least for now). Let me know if it works. Cheesy

I started testing it for the first time, now it's over 10K. From these results, there are 10997 users who actively post on bitcoin discussion boards including child boards during 2020. I think the tool is better now, is there still a limit to the number?

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Rikafip on July 17, 2022, 11:21:41 AM

Is there some way to increase that cap, or its limited due some technical reason?

Like @indah rezqi said above, it's an arbitrary number to keep everything running fine. But since there hasn't been any abuse, I'm uncapping it (at least for now). Let me know if it works. Cheesy

indah rezqi

hero member

Activity: 1330

Merit: 852

Quote from: Rikafip on July 17, 2022, 11:21:41 AM

Hey @TryNinja, I wanted to check the amount of users on some boards years ago and compare it with the current situation but your tool (afaik) counts only up to 10,000 users. Is there some way to increase that cap, or its limited due some technical reason?

I searched in this thread about whether there was a similar request before. I got the answer that this tool was previously only limited to 2K numbers, but has been upgraded to 10K on august 2021. So I think it's still possible to expect it to be 20K to 50K, but it might just depend on the database. Database overload is a problem to be avoided based his post.

Quote from: TryNinja on August 20, 2021, 06:28:55 PM

Quote from: skarais on August 19, 2021, 03:01:22 PM

Will we get bigger limit in the future, maybe 5000-10.000?

Hey, 2000 users was just an arbitrary number that I used to avoid overloading the database, but I can try raising the limit to 10k and you'll tell me if that's enough.

edit: done, the maximum is now 10k

Rikafip

legendary

Activity: 1722

Merit: 5937

Hey @TryNinja, I wanted to check the amount of users on some boards years ago and compare it with the current situation but your tool (afaik) counts only up to 10,000 users. Is there some way to increase that cap, or its limited due some technical reason?

dkbit98

legendary

Activity: 2212

Merit: 7064

Cashback 15%

Quote from: LoyceV on June 21, 2022, 01:13:32 PM

Don't trust all webhosts there, I've been scammed by their "offers". Lowendtalk.com (more or less the same) has good offers too, and I've come to trust some of them (after I got scammed by another one). But I wouldn't expect better uptime than Contabo. High uptime doesn't go well with "low end".

Lol
Lowendtalk is official forum of lowendbox website and you can actually see their link posted on top next to their Facebook and Youtube link, and domain owner is the same, so there is really no difference Cheesy

If I said check that you should check that website it doesn't mean that you should buy the first thing someone posted there.
I think that Hostinger and Namecheap are accepting Bitcoin payments, any maybe few other websites that are less known and less reliable.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: dkbit98 on June 21, 2022, 11:42:13 AM

You should check out and follow lowendbox.com website, they have some great hosting and VPS deals, among other things.

Don't trust all webhosts there, I've been scammed by their "offers". Lowendtalk.com (more or less the same) has good offers too, and I've come to trust some of them (after I got scammed by another one). But I wouldn't expect better uptime than Contabo. High uptime doesn't go well with "low end".

Quote

I am not sure if they can beat Contabo but hetzner.com should be solid and reliable alternative.

They don't accept Bitcoin. I don't want to pay by creditcard.

dkbit98

legendary

Activity: 2212

Merit: 7064

Cashback 15%

Quote from: LoyceV on June 21, 2022, 01:57:34 AM

Contabo's rates are hard to beat, better uptimes can get quite expensive. I'm on the lookout too, my (sponsored) AWS is running out soon.

You should check out and follow lowendbox.com website, they have some great hosting and VPS deals, among other things.
I am not sure if they can beat Contabo but hetzner.com should be solid and reliable alternative.

PS
IT looks like forum is still facing some issues on my side.
When I click on the Quote button to reply I am getting big green LOADING bar on the top, and nothing happens until I refresh page and do it again.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on June 21, 2022, 01:30:56 AM

or I'll move somewhere else. Sad

Contabo's rates are hard to beat, better uptimes can get quite expensive. I'm on the lookout too, my (sponsored) AWS is running out soon.

TryNinja

legendary

Activity: 2758

Merit: 6830

My apologies. Some services didn't start correctly after their maintenance ended.

I already sent an angry email telling them to stop doing that or I'll move somewhere else. Sad

Topic: Ninjastic.space - BitcoinTalk Post/Address archive + API - page 9. (Read 15134 times)