Bitcointa.lk what is it??? | Bitcointalksearch.org

cedivad

legendary

Activity: 1176

Merit: 1001

The 7.6 peak was due to the backlog i had on edited content, that i implemented yesterday. If you check the traffic one week ago you should see an average way less than 1 request per second. (and i'm sort of sorry for this peak).

Quote from: theymos on March 26, 2014, 06:17:45 PM

The maximum allowed bot request frequency is 1 request per second. Those IPs are now accessing pages at an average of 2.5 requests per second combined. If you continue exceeding the allowed request limit, we will continue banning your IPs.

This forum has less than 30k posts per day, so those numbers doesn't add up on my side, something must be wrong (especially since that i removed the sync of modified content after this problem). I will write a way to see how many requests i actually do and what % fails/etc. The problem could be peak hours, when there is more than 1 post per second... But i doubt that.
Posted from Bitcointa.lk - #yCkCrM3Xkyk4sagw

theymos

administrator

Activity: 5222

Merit: 13032

Those IPs are not blocked currently. But your other abusive IPs were blocked. Just your quotefast requests (which are only part of what your crawler does) were occurring at an average frequency of 7.6 requests per second in the most recent access logging period. Your requests constituted 3.4% of all forum requests in this period. This is entirely unacceptable and of course resulted in those IPs being banned.

The maximum allowed bot request frequency is 1 request per second. Those IPs are now accessing pages at an average of 2.5 requests per second combined. If you continue exceeding the allowed request limit, we will continue banning your IPs.

cedivad

legendary

Activity: 1176

Merit: 1001

Thank you for passing by, theymos.
Have you changed the behaviour of the DoS automatic banner recently? Because it's the only rationale solution i can think off, if those bans are not manual. The crawler worked just fine for a month until yesterday.

The 5 IPs i'm using right now are 54.217.199.213, 46.137.37.197, 54.195.167.127, 54.195.152.218, 54.73.190.33.

Let me simply know if you want me to kill this project and the crawler.
Posted from Bitcointa.lk - #nicm5KCTPCqeXZOn

roslinpl

legendary

Activity: 2212

Merit: 1199

So now we have two bitcointalks

Great ...

Anyway if you all think this is ok - good.
But some people might be confused while visiting bitcointa.lk.

theymos

administrator

Activity: 5222

Merit: 13032

What IP?

IPs are only banned for over-use if they request pages more frequently than once per second. Faster requests are not allowed.

cedivad

legendary

Activity: 1176

Merit: 1001

It has been running just fine for a month now, and the server handled it perfectly until lately (last 3 days?), when the forum started lagging, but that's not related to a change on my side.

Given that i own a website that has in the order of millions (5-10) hits per day (of non cached and dynamic content, not of gifs), i would be happy to give a suggestion or two on the infrastructure behind bitcointalk.org.
Posted from Bitcointa.lk - #RhbaqkRpcd9z6Mk3

hilariousandco

global moderator

Activity: 4018

Merit: 2728

Join the world-leading crypto sportsbook NOW!

Quote from: SaltySpitoon on March 26, 2014, 11:51:16 AM

Quote from: cedivad on March 26, 2014, 11:19:02 AM

I just wanted to say thank you to theymos for having decided to ban our crawler twice during the last 12 hours.
Posted from Bitcointa.lk - #yUXGFN4tVExKSjIn

I don't think Theymos banned it, its a rather useful site, the moderation staff are using it to catch the people that are bump spamming, and then deleting their posts to get around the 24 hour bump limit.

Haha, sneaky bastards.

tysat

legendary

Activity: 966

Merit: 1004

Keep it real

Your crawler is probably taxing on the server. How often is it running, and how many pages does it load at once?

cedivad

legendary

Activity: 1176

Merit: 1001

I'm surprised to hear that it's used even from the staff.

I'm sorry for having complained, but I'm quite sure it was a manual ban.

I will just use it as an excuse to make the crawler smarter and able to detect when banned... It was in the todo anyway.
Posted from Bitcointa.lk - #xRyIEg5gjSsOYBPf

SaltySpitoon

legendary

Activity: 2590

Merit: 2156

Welcome to the SaltySpitoon, how Tough are ya?

Quote from: cedivad on March 26, 2014, 11:19:02 AM

I just wanted to say thank you to theymos for having decided to ban our crawler twice during the last 12 hours.
Posted from Bitcointa.lk - #yUXGFN4tVExKSjIn

I don't think Theymos banned it, its a rather useful site, the moderation staff are using it to catch the people that are bump spamming, and then deleting their posts to get around the 24 hour bump limit.

cedivad

legendary

Activity: 1176

Merit: 1001

Quote from: Kluge on March 26, 2014, 11:20:49 AM

How frequently does it poll? Possible it was auto-blocked as DoS.

It depends on the number of posts, however a total of < 50k requests/day over 5 IPs.
All together, around one per 2 seconds.

If it was auto blocked as DoS it's one thing, if he just decided to block the competition while this forum is left dying with unloading images, server errors everywhere and a 10 years old board, well, i expected that, but i still have no words.
Posted from Bitcointa.lk - #0zLZOUaxU7Sr2W6A

Kluge

donator

Activity: 1218

Merit: 1015

Quote from: cedivad on March 26, 2014, 11:19:02 AM

I just wanted to say thank you to theymos for having decided to ban our crawler twice during the last 12 hours.
Posted from Bitcointa.lk - #yUXGFN4tVExKSjIn

How frequently does it poll? Possible it was auto-blocked as DoS.

cedivad

legendary

Activity: 1176

Merit: 1001

I just wanted to say thank you to theymos for having decided to ban our crawler twice during the last 12 hours.
Posted from Bitcointa.lk - #yUXGFN4tVExKSjIn

Satosh¡ Slot

full member

Activity: 164

Merit: 100

Indie Developer

Quote from: cedivad on March 25, 2014, 02:29:45 PM

Hi, i coded Bitcointa.lk. Announcement thread: https://bitcointalksearch.org/topic/bitcointalk-the-bitcointalk-community-with-a-2014-forum-software-515932
I always hated how Bitcointalk.org was administrated (or how it was not administrated at all), so i woke up one day and decided to do something.

Short features list:
-Real time synch of every message
-Posts removed on [Suspicious link removed], but you can still read them; you can also see a history of deleted messages
-You can use it to post on [Suspicious link removed]

And naturally, i did it for free and i didn't spent a million dollar in the process.
Posted from Bitcointa.lk - #aROgnmf7qUAOcdNM

Good job. I think the bitcointalk forum could do a much better job at attracting new bitcoin users. Now they come here and see a default simple machines forum with a few tweaks. It could be much more inviting and professional. Bitcointalk also has lots of funds to accomplish this from all of the donations that are now worth a small fortune.

cedivad

legendary

Activity: 1176

Merit: 1001

Deleted content is only visible to registered (linked) users. So there is no "i accidentally posted my email address" problem.
Posted from Bitcointa.lk - #pJeGZFI18BfrDwuA

nahtnam

legendary

Activity: 1092

Merit: 1000

nahtnam.com

Quote from: roslinpl on March 25, 2014, 07:59:07 PM

Quote from: nahtnam on March 25, 2014, 07:48:03 PM

Quote from: roslinpl on March 25, 2014, 07:46:55 PM

Quote from: nahtnam on March 25, 2014, 07:40:52 PM

Its a backup of all posts + its beautiful!

For me it is not very good that you can read deleted posts ... some of them were dangerous - scam, phishing, etc.

Well its really useful for self moderated threads. You can see exactly what people posted.

well from this point of view maybe yes.
... if it would be moderated and he would remove scam posts ... and leave only deleted posts from self moderated threads it can be even more useful. Isn't?

Anyway I really think Bitcointalk.org is no.1 and I never was @ better forum... And just in my opinion is that making a clone that "looks more fancy" and contains deleted post (what is not IMO as good as you know already) is nothing special and it might be even from some points not really nice from a point of view of users that deleted their post, because for example - they did posted their e-mails or something, that they recon they do not want to keep @ forum for some reason.

No one is saying you should use it... I love the current layout of bitcointalk, but some dont. Those who dont now atleast have an option.

roslinpl

legendary

Activity: 2212

Merit: 1199

Quote from: nahtnam on March 25, 2014, 07:48:03 PM

Quote from: roslinpl on March 25, 2014, 07:46:55 PM

Quote from: nahtnam on March 25, 2014, 07:40:52 PM

Its a backup of all posts + its beautiful!

For me it is not very good that you can read deleted posts ... some of them were dangerous - scam, phishing, etc.

Well its really useful for self moderated threads. You can see exactly what people posted.

well from this point of view maybe yes.
... if it would be moderated and he would remove scam posts ... and leave only deleted posts from self moderated threads it can be even more useful. Isn't?

Anyway I really think Bitcointalk.org is no.1 and I never was @ better forum... And just in my opinion is that making a clone that "looks more fancy" and contains deleted post (what is not IMO as good as you know already) is nothing special and it might be even from some points not really nice from a point of view of users that deleted their post, because for example - they did posted their e-mails or something, that they recon they do not want to keep @ forum for some reason.

nahtnam

legendary

Activity: 1092

Merit: 1000

nahtnam.com

Quote from: roslinpl on March 25, 2014, 07:46:55 PM

Quote from: nahtnam on March 25, 2014, 07:40:52 PM

Its a backup of all posts + its beautiful!

For me it is not very good that you can read deleted posts ... some of them were dangerous - scam, phishing, etc.

Well its really useful for self moderated threads. You can see exactly what people posted.

roslinpl

legendary

Activity: 2212

Merit: 1199

Quote from: nahtnam on March 25, 2014, 07:40:52 PM

Its a backup of all posts + its beautiful!

For me it is not very good that you can read deleted posts ... some of them were dangerous - scam, phishing, etc.

nahtnam

legendary

Activity: 1092

Merit: 1000

nahtnam.com

Its a backup of all posts + its beautiful!

Topic: Bitcointa.lk what is it??? (Read 1842 times)