The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.
If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk
What I unsuccessfully tried in php was to file_get_contents, then do a javascript setTimeout of 10 seconds, after which I do a window.location.reload.
hmmm, well i guess if they say web crawlers can scan it, there has to be a way. what's the link where you read this?
I went the python route a few weeks back, but got no where. After reading your post, i fired wireshark back up and it looks like there's a hidden form that needs some special key to return in order to let us through ... so I also think we'll need some code to accept their cookie/whatever and respond back similar to how a browser would. still kind of pain to grab some data
update: ok so i did some digging. got this from the cloudflare site:
Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I'm Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist.
My guess is they whitelisted known and allowed search engine crawlers which is why those bots work where as ours dont. Add to it that it looks like Terk opted to put the domain in a permanent I'm Under Attack Mode and we are where we are. So pretty much we need to mimic the expected behavior if we want anything to work ... lovely
I neednt link you now, your text is exactly what I read. I did not reach the conclusion that you did about the crawlers being whitelisted, thanks for helping me think!
As to "kind of a pain", well, it would be worth it so I could get some sleep, I'm plain stupid about having to check every hour and enter data into a spreadsheet. Auto-reading from CM is the only missing link. I'm databasing the exchange rates of 5 different coins once every five minutes, making graphs, all automatically.