Author

Topic: Scraping bots on this forums (Read 153 times)

legendary
Activity: 1184
Merit: 1013
January 18, 2018, 10:02:58 PM
#4
You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.
Yes, I won't be logging in. Sending requests from my google cloud so it won't even be my IP.
Thanks for the heads-up
legendary
Activity: 1789
Merit: 2535
Goonies never say die.
January 18, 2018, 04:48:44 PM
#3
You want to keep it at least 1 second per request, and you may want to send a friendly PM to theymos to avoid any potential issues/blacklisting. And it's better to request data while not being logged in.
legendary
Activity: 2142
Merit: 1065
✋(▀Ĺ̯ ▀-͠ )
January 18, 2018, 02:48:48 PM
#2
AFAIK bots are blocked by cloudflare which was implemented lately. That's why http://www.bctalkaccountpricer.info for example stopped working... Other members can confirm this.
legendary
Activity: 1184
Merit: 1013
January 18, 2018, 02:45:33 PM
#1
I am collecting some data, a lot of which can be collected from this forums.
How many HTTP requests per minute can i send at max?
I read a post about this last year, it was allowed until then. Confirming status as for Jan 2018.

My kind little bot will do nothing more than scrape this forum for required data.
Jump to: