[TELEGRAM] Yet Another BitcoinTalk Notification BOT (merits, mentions, topics,+) - page 49.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on May 21, 2020, 03:45:29 AM

I'm not sure if it's still loading or stuck, just searching for username "LoyceV" or Topic ID "1813624" keeps loading indefinitely.

Not sure why. I just tried searching your username in both “username” and “text search” field and it didn’t take longer than 15 seconds. Can you try again?

edit: Some searches do take some time. I tried doing 3 tabs at the same time: text "theymos", text "LoyceV" and just the last posts from the home page. It took close to 35~50 seconds in the text ones (they are not optimized and search the whole database). I will see if I can make it better.

~~edit 2: it should be slower now while this edit is here since I'm creating some db indexes.~~
edit 3: I think it's over. It should be a lot faster now.

Quote

It's currently 11 GB using bzip2 compression.
I didn't keep the original post dates for this though, but you can always get a raw estimate by scraping one in every 1000 posts.

I would need to ask my host for some additional storage. Cheesy

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on May 21, 2020, 12:06:01 AM

Now, the https://posts.ninjastic.space has all* of Loyce's 2,5 million archive posts saved (*with the rare exception of a very few ones). Most of them have no title, but you can still search by topic id, content, author and date. Expect some graphic bugs since I wasn't planning on having that many data that early. And please be gentle with the database (text content takes some time and it's heavy work).

I'm not sure if it's still loading or stuck, just searching for username "LoyceV" or Topic ID "1813624" keeps loading indefinitely.

To raise the stakes a bit, I have much more posts, although it'll take a few more months to complete scraping. I expect to be done around August, so if you can handle a lot more data to search by then, you can have it all Tongue

It's currently 11 GB using bzip2 compression.
I didn't keep the original post dates for this though, but you can always get a raw estimate by scraping one in every 1000 posts.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: cygan on May 21, 2020, 12:20:16 AM

which other feature i would like to see on this bot is that just like the merit/s, an announcement is also displayed for a trust (positive, neutral or negative) that a user has just received from another bitcointalk user

I honestly don't think this is needed/worth the time. Trust feedback is something that is given to people only a few times in a lifetime. It's more workload for the bot in exchange for very few advantages.

I just got two notifications for your mention, so I freaked out thinking I broke something or the bot wasn't handling a 4,6 GB table. But in the end, it was because I forgot to clean my local db before migrating the data, so it had 2 TryNinjas. Ok, now I can sleep. Tongue

cygan

legendary

Activity: 3304

Merit: 8633

Crypto Swap Exchange

which other feature i would like to see on this bot is that just like the merit/s, an announcement is also displayed for a trust (positive, neutral or negative) that a user has just received from another bitcointalk user

i think this option is also possible or TryNinja?

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Little Mouse on May 20, 2020, 09:14:49 PM

You have way to build watch list, but it would be pretty good if there was a way to stop getting information from certain topic. I am bored of receiving notification from a topic. I do not want to get any notification from tha topic.

It's the next feature. I will work on it tomorrow.

Had a small downtime with the scrapper today since I was doing some heavy migrations in the database. But it was worth it:

Now, the https://posts.ninjastic.space has all* of Loyce's 2,5 million archive posts saved (*with the rare exception of a very few ones). Most of them have no title, but you can still search by topic id, content, author and date. Expect some graphic bugs since I wasn't planning on having that many data that early. And please be gentle with the database (text content takes some time and it's heavy work).

Little Mouse

legendary

Activity: 2156

Merit: 2100

Marketing Campaign Manager |Telegram ID- @LT_Mouse

You have way to build watch list, but it would be pretty good if there was a way to stop getting information from certain topic. I am bored of receiving notification from a topic. I do not want to get any notification from tha topic.

mikhailr

sr. member

Activity: 812

Merit: 443

★777Coin.com★ Fun BTC Casino!

Quote from: cygan on May 20, 2020, 11:59:51 AM

Quote from: cygan on May 19, 2020, 12:41:14 AM

~
Thanks to:
~ list ~

BTCump

You understand that you not only bump post, but also sent a notification in a TelegramBot to everyone who is mentioned in this list Wink

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on May 19, 2020, 01:19:01 PM

Btw, I managed to get your files in my HD (after ~1 hour of extraction and some freezing) and to make a scrapper that loops through them and makes an entry in the database. But I then noticed that they do not contain a title (only some old ones?) or exact date, so they can't just be inserted in my DB the way it is structured now. I'm thinking about making a page and table just for your posts, so you can at least be able to search by content and author (not sure if that can be done easily with 2m+ posts).

For the old posts I hadn't decided on the exact format yet, hence the title.
I have other topic titles too, I use them here. I have just over 100,000 titles and can get you a text file if it helps. But: those are the topic titles as I scraped them once. The title may have changed later, and each post can have a different title too.
The date in my files is my scrape-date, for most posts it's only a few seconds after the post was created.

I sometimes use a "local" search on the server, looping through a few million posts, but that takes a while. It would be nice if a database can allow quick searches through a web interface.

cygan

legendary

Activity: 3304

Merit: 8633

Crypto Swap Exchange

Quote from: cygan on May 19, 2020, 12:41:14 AM

@TryNinja
might be helpful for the other users, if you leave the link in your op behind the post for the respective translations

Quote from: TryNinja on May 16, 2020, 05:43:40 AM

...
Thanks to:
@Piggy for his first bot, which originated this era of telegram bots in the forum.
@giammangiato for his bot, where I took some ideas and message design.
@csmiami for the Spanish translation.
@sheenshane for the Filipino translation.
@mole0815 for the German/Deutsch translation.
@mikhailr for the Russian translation.
@fillippone for the Italian translation.
@cygan for the Polish translation.
@masulum for the Indonesian translation.

like
@cygan for the Polish translation.

what you think?

BTCump

Mitchell

copper member

Activity: 3948

Merit: 2201

Verified awesomeness ✔

This looks like something I've been looking for since Piggy's bot went down. Thank you TryNinja!

dkbit98

legendary

Activity: 2212

Merit: 7064

Using this bot for few days and I love it so far.
It has all the features I need... and I am also waiting for planned ignore option.

One small thing I would like is to have menu be closed by default, and not opened.
I don't use it all the time, it takes space and I always have to close it.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Csmiami on May 20, 2020, 05:30:09 AM

I've just been mentioned in another thread, and the bot has given the following message:

Quote from: SuperNotifier

@Csmiamii think this work, try with me

See how there's a double "i" at the end of the nick. I've checked the post in Loyce Club, and it's not a typo that got fixed.

You are right, there is no typo. This happened because this is the post:

Quote

@Csmiami

i think this work, try with me

Notice the first line ends with Csmiami, and the second starts with "i think". The bot removes all the line breaks, so the "i" goes next to your name, making it all one line.

~~I think I can add an space when there is a line break, instead of just removing it completely.~~ Done!

Btw, since I'm already making this post: I found an issue which I had with BitcoinTalk, where pretty much once per day a request would result in the "getaddrinfo EAI_AGAIN" error, which would cause the bot to get stuck. I have found a workaround for the bot getting stuck with it (which is the important part), but I still didn't find how to avoid this error (or if I can even avoid it). If anyones know more about this, let me know, please. Cheesy

Csmiami

copper member

Activity: 1652

Merit: 1325

I'm sometimes known as "miniadmin"

I've just been mentioned in another thread, and the bot has given the following message:

Quote from: SuperNotifier

@Csmiamii think this work, try with me

See how there's a double "i" at the end of the nick. I've checked the post in Loyce Club, and it's not a typo that got fixed.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: LoyceV on May 19, 2020, 12:22:33 PM

Are you scraping each user's Merit summary instead of just scraping Recent merits? You'll need to scrape the Merited post to see who received the Merit, but it's probably faster (and less scraping) than checking all individual users for Merit transactions.

Yes. Thinking about it now, I could probably do something even better to avoid scraping the posts (again). Since I already scrape every new post shown in the recent posts page (along with its id), I could check for which post id the merit was sent (I can get that from the URL of the post) and see if the posts exists in the database. If it does, I check which user did that post (which I also save). If its an user of the bot, I send him a notification.

But I'm not sure if its worth doing a lot of changes in my database and code for that. Maybe in the future.

Btw, I managed to get your files in my HD (after ~1 hour of extraction and some freezing) and to make a scrapper that loops through them and makes an entry in the database. But I then noticed that they do not contain a title (only some old ones?) or exact date, so they can't just be inserted in my DB the way it is structured now. I'm thinking about making a page and table just for your posts, so you can at least be able to search by content and author (not sure if that can be done easily with 2m+ posts).

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: TryNinja on May 19, 2020, 10:21:03 AM

Well, I have increased the time between each merit scraping and now there is a lot more users than before, so it takes more time (reasoning in the change log). Usually it doesn't take more than 5 minutes for the merit notification to come. Mentions are still quick, though.

Are you scraping each user's Merit summary instead of just scraping Recent merits? You'll need to scrape the Merited post to see who received the Merit, but it's probably faster (and less scraping) than checking all individual users for Merit transactions.

asche

legendary

Activity: 1484

Merit: 1491

I forgot more than you will ever know.

Quote from: Csmiami on May 19, 2020, 10:24:36 AM

Quote from: TryNinja on May 19, 2020, 10:21:03 AM

....

Nah, yours is still working fine

Quote

(UP) MERITS 382 your merit page https://bitcointalk.org/index.php?action=merit;u=903175
From for

The "from for" was the odd message, but not on your bot. But as I recall seeing the message from this morning saying that you 2 could collaborate, I assumed some playing around had been done.

I really hate the forum on the phone....

You'd have to check on giammanto's thread. He is implementing an improvement, however it seems to be a bit buggy for the time being.

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Csmiami on May 19, 2020, 10:24:36 AM

The "from for" was the odd message, but not on your bot.

He is probably testing some stuff.

Quote from: Csmiami on May 19, 2020, 10:24:36 AM

But as I recall seeing the message from this morning saying that you 2 could collaborate, I assumed some playing around had been done.

I think he made his bot with PHP (or at least I assume), while mine is done with Node.js. I also tend to make a lot of critical changes pretty fast so I'm not sure how that would work. Thankfully, I'm testing a lot of stuff to avoid breaking the bot every couple of hours. Cheesy

Quote from: Csmiami on May 19, 2020, 10:24:36 AM

I really hate the forum on the phone....

Me too... the idea of creating an open source mobile app for the forum already crossed my mind (I only wish the forum had an open API..). We can only hope for Epochtalk to come.

Now back to extracting a few GBs of files and running it through an custom parser.

Csmiami

copper member

Activity: 1652

Merit: 1325

I'm sometimes known as "miniadmin"

Quote from: TryNinja on May 19, 2020, 10:21:03 AM

....

Nah, yours is still working fine

Quote

(UP) MERITS 382 your merit page https://bitcointalk.org/index.php?action=merit;u=903175
From for

The "from for" was the odd message, but not on your bot. But as I recall seeing the message from this morning saying that you 2 could collaborate, I assumed some playing around had been done.

I really hate the forum on the phone....

TryNinja

legendary

Activity: 2758

Merit: 6830

Quote from: Csmiami on May 19, 2020, 10:16:10 AM

have you guys started merging both bots?

Well, I have increased the time between each merit scraping and now there is a lot more users than before, so it takes more time (reasoning in the change log). Usually it doesn't take more than 5 minutes for the merit notification to come. Mentions are still quick, though.

Quote from: Csmiami on May 19, 2020, 10:16:10 AM

I've just received the merits notification on giamancciatos first, with some cryptic message before yours

What cryptic message? Tongue

I'll go back to improving the bot but in currently trying to use Loyce's post data on this bot's side project (http://posts.ninjastic.space/). Right now, starting to scrape his 2,5 million .html files to insert in my database. My HD almost froze for a moment. Cheesy

Csmiami

copper member

Activity: 1652

Merit: 1325

I'm sometimes known as "miniadmin"

have you guys started merging both bots?
I've just received the merits notification on giamancciatos first, with some cryptic message before yours

Sorry for typos and stuff, I'm on the phone enjoying the sunlight on the street

Topic: [TELEGRAM] Yet Another BitcoinTalk Notification BOT (merits, mentions, topics,+) - page 49. (Read 19363 times)