Some posts that have been edited, ninjastic shows the old content and the new content of the post, and some do not. It says "No edit history has been found for this post." even though I know for a fact that the post was edited. I can tell by the underline line. What does this depend on?
My bot scrapes every post two times: once when it is posted and
another time after 5 minutes. If the post was edited at that point, after the first 5 minutes, the changes will show up on the edit history. That’s it.
If you edit a post 10 times, or edit a post after 1 day, 7 days, … there is no way for me (and my bot) to know since keeping track of every change is impossible.
Why 5 minutes? Why not 2 hours or 6 hours or 12 hours or even 24 hours to really be inclusive of most of the times that edits might happen.
Most of the time, if I am going to edit, I will edit many of my posts within 5 to 20 minutes, but there are some very substantive edits that I might make several hours later, sometimes even 8-16 hours later. Probably less than 1% of any of my edits would end up happening beyond 16 hours after the initial post... and maybe only right around 50% would actually happen within 5 minutes of the initial post.
Of course, I am kind of guessing about my own behavior, but it seems that a second scraping of a few hours would end up being more inclusive to capture a lot more of the edited posts rather than doing it within 5 minutes.
Don't get me wrong.. I can see that maybe you are correct if you have an assumption that most of the edits happen within 5 minutes of the original post, and the more I sit here and think about it, there are many times that I do edit right away.. but since my posts tend to be long, sometimes it will take longer than 5 minutes just to perform whatever edits that I have.. but sure some of them take less than a minute.. so maybe you have more data or reasons for why you choose 5 minutes for your second scrape. and also choose not to do a third scrape.
