if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
But this wouldn't be like rebuilding the full forum on a database?
I mean, there are 2 ways to do this:
1.- You take all the forum data, and put it together on a database and then your search engine makes calls to that database. But for this, you will have to live update that database or at least have a cron job to add the new data each x time.
2.- Search for the data directly on the site, but for that, you would have to do some kind of hack to the current search engine.
If you have other way in mind i would love to know how it work.
It would be better if there was a way to improve SMF's search function or query the relational database directly, but i don't know if Theymos would give anybody direct access to the database or allow anybody to completely rework smf's soucecode .
I don't think he would, and for good reason offcourse... It would require absolute trust in the person building the search engine.
But you're right, it would be completely rebuilding bitcointalk's database, like several other members are doing aswell (more or less)..
If anyone has access to implement a change like this in the forum, that one is our verified hacker PowerGlove (https://bitcointalksearch.org/user/powerglove-3486361), but the fact that the search function doesn't work fine at all must be for a reason. Maybe the forum used to have some kind of attacks from that vector.
This project would be easy if the RSS was still active on the forum, but sadly it has been removed:
https://bitcointalk.org/index.php?type=rss;action=.xml
Quote
action=.xml is disabled due to slowness. If you use this, write a post in Meta explaining your usage.
The problem is the moment to index the new threads, without RSS we have to "Ping" each board to know if there is new content.
