<…> As long as there are no "are you a human" tests <...>
There aren’t (yet). Cloudflare seems to handle things pretty well, giving you an error if you try to load pages at a rate faster than 1 page per second. Other than that, I have never encountered any issues to date (although I scrape a limited amount during the time span of 12 consecutive hours or so, and every now and then go for a while longer).def scrape_function():
try:
#8 spaces
some_code_to_scrape
except:
pass
some_variable = True
while some_variable == True:
scrape_function()
Depending on the content you're scraping it might be better to contact third parties that are already collecting the data. I think between them they've got most data covered. Some of these might have an API instead of scraping also.
Where can I find these APIs? What data does it cover?Be careful w/ that code btw. If using, you're going to want to put a delay on the loop to ensure you're not unintentionally DOSing the site.
time.sleep(1)
the 1 inside the function signals to stop for one second before moving on. This will mean I access a page slightly less frequently than once per second because my VPS needs to send data to my postgres server and fetch HTML
