Topic: Unofficial Bitcoin Talk Forums API - page 2. (Read 5289 times)

Roger that.

The solution you propose should be totally fine.

One thing I wonder, though, is if it would be sufficient to have the login request be direct, and subsequent requests with that SMF cookie be proxied.

If the user was banned then I'm assuming the IP address would be reported as the one in the original login request?

If not then I can proxy everything but this would be a nice shortcut/work-around/compromise type thing.

While we're on the topic, can you please un-ban 203.206.164.125? I made a joke account while at work (static ip) but one of the mods didn't find it funny, and now I have to tether to my phone to post

theymos - or maybe you could host the API on bitcointalk.org?

Yes. If people could post things through your API, your API would be acting as a proxy, preventing IP bans from being effective. Maybe some solution to this could be figured out, but it makes more sense for API users to send such requests to the forum directly. Clients don't need to process any forum HTML when posting -- they just need to POST specific data to specific URLs and sometimes store the returned cookies. You could write a library to help clients figure out what data to send and where to send it, or your API could return this data.

I'm more than happy to oblige but I need you to expand on this a little. Are you talking about the IP banning logic?

Thanks! (Well, PM's are important for anybody looking to make a bot, since public topic reply isn't really needed.)

If possible, you may want to have API users POST the forum directly so that we don't have to deal with abuse/bans through your API.

OK manage PMs it is then! Good thing I asked; I assumed managing PMs was the least important.

I'll start work on the auth stuff tonight and update here when I have news.

Login, manage PMs.

(Let's work on a tip bot!)

Nope, I'm not? I'm just saying a PM api would be useful to me really

Also, what about a firehose API?

You're hinting at a tipping bot yeah?

Someone PMed me about that a while ago but it wasn't clear if he was going ahead with it.. I'm surprised nobody is working on it. Do you know something I don't or are you just speculating?

You should make PMs are priority. For example, verification bots "PM CoinLenders-Verify with 28fy28c in the subject".

Woohoo, my first tip!

Login support was low priority but I can move it to the top since I'd rather code things people actually want/need. 

What do you actually want to be able to do? Off the top of my head there are several major functions

- Create/edit topic
- Create/edit post
- Edit profile/forums settings
- Manage PMs

If you let me know which is the most important I can do that first. But I can start coding the basic login steps (auth to server and store session cookie) now.

 

Thanks so much for making this! Might be able to create something nice with this API, now all I need is login support etc (which you'll hopefully add soon)

0.01 BTC sent (https://blockchain.info/tx/9291f180b4d54ffb453ffda8d57ff47bc597ba2634bef38520039789c550c95e). Plus have a kudos from a fellow coder

Bumping this because of recent interest from another member

Yeah, I agree. I can see your posts from the other thread.. You would have seen that theymos said the WAP page wouldn't change, so until the forum upgrade happens then it should be stable. I can't honestly make any guarantee that my API won't break, since I can't control these forums. However I am making my own client so I wanted to publish my own API so that others can use it, since it doesn't feel right keeping it closed source since it's not my content to begin with!

I'll adjust the scraper when the new forums go in if there is no API available.

Thanks for sharing your screen-scraper, but coding to an API is a serious time investment.  That investment is pretty hard to justify if the forum operator does not acknowledge the API or make any effort to avoid breaking it.

Hi,

It was a long weekend last week here in Australia, and I was too broke from buying Bitcoins to go out, so I embarked on a project to create an unofficial Bitcointalk.org REST API.

I will give a short description of the API, then there are some links you can visit which show the API in action.

The API is JSON only, and currently lets you list categories, boards, topics, and posts. Future work may include adding user profiles also. The API only lets you read from Bitcoin Talk; there is currently no way to post, edit posts and so on. The API is hosted on Google App Engine (GAE). NB: Occasionally an HTTP request to GAE apps will take a long time to respond if it has to spin up a new instance.

• Category / board list
• An individual board listing
• An individual topic listing
• An individual page listing
• Convenience parameter - load a topic and known page number in one request, instead of two. The page is loaded into 'requestedPage'
• Convenience parameter - load a topic and the 'latest' page in one request. The latest page is the last page in a multi-page topic or the first page if there is only one page. The page is loaded into 'requestedPage'

If you browse the API you may discover boards and topics which don't appear to have any content - that's due to my 'lazy scraping' pattern detailed below.

A small note about viewing the API from your web browser: I'm using a little trick I learned from some of my old colleagues while developing APIs - the API checks the "Accept" header to detect if you are requesting the content from your browser, or from JavaScript. If you are in your browser then the API returns pretty printed HTML and turns relevant properties into links, so you can follow them. If you request from JavaScript then non pretty printed, non-linked JSON objects are returned. This lets you browse the API easily without a REST client.
 
The API works by using a screen scraper on the WAP version of the forums, as suggested by theymos in this post. The screen scraper is in its own project, separate from the API itself.

The overall design is to avoid scraping old topics, and instead focus on the newest content. The scraper can’t download the entire forum at once, so it uses a ‘lazy download’ or ‘lazy refresh’ approach to scraping content - if a topic or board is requested and it was scraped less than a certain amount of time ago*, then a task will be added to a task queue for the topic to be re-scraped. That means if nobody is using the API then no tasks go in the queue, and if lots of people are using the API then lots of tasks go in the queue. The task queue will then eventually be filled up by many requests to scrape many boards and topics, and I can adjust the rate of the execution of these tasks based on how much theymos yells at me . If an extremely large topic is encountered (like the Wall Observer thread in Speculation) then the first two pages and the last two pages will be downloaded only, to avoid generating too many requests.

* Current ‘freshness’ of boards is 5 minutes and freshness of topics is 1 minute. If you’re making a client then I recommend simply hard-coding the list of categories and topics since it’s not likely to change that much.

Limitations:

• Fixed number of posts in a page (20, same as these forums)
• Currently JSON only
• There isn't any time/date information on posts - that's because I'm using the WAP version of the forums and the date isn't displayed.
• Similar to the point above, the formatting for the BBCode (quote) feature is lost and it just comes out as plain text like 'Quote: ...
  
  

Code is open source at GitHub (Java): BitcoinTalkScraper and BitcoinTalkAPI

Stay tuned this week for news on my unofficial Bitcointalk mobile/tablet forum client.

I am doing this totally for the love of Bitcoin and these forums, and everything is free, but if you are feeling really generous then please tip me at 17SbWcyRoZd7u1tZeJtjzm834a3gAHdf2A. Thanks!