fetch bitcointalk data | Bitcointalksearch.org

lahm-44

sr. member

Activity: 336

Merit: 250

Quote from: minifrij on December 14, 2015, 08:59:05 AM

Quote from: lahm-44 on December 14, 2015, 08:42:19 AM

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework; I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.

Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned.

thinks for the link finally i got a point from where i can start researching about it and i hope html dom are not only the single thing which is not taught to me on the free learning sites anyway thanks for the help.i am keeping this thread open so if anyone like to give any other helpful reference then u are welcome i will study it

minifrij

legendary

Activity: 2352

Merit: 1267

In Memory of Zepher

Quote from: lahm-44 on December 14, 2015, 08:42:19 AM

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework; I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.

Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned.

lahm-44

sr. member

Activity: 336

Merit: 250

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

mocacinno

legendary

Activity: 3514

Merit: 5123

https://merel.mobi => buy facemasks with BTC/LTC

Quote from: OmegaStarScream on December 14, 2015, 08:25:02 AM

I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer

I think they do this by parsing the profile page, and then parsing all post history (like i said in my first post)

I didn't browse the full sourcecode, but i immediately noticed following string on their github page:

Code:

Document profileSummary = Jsoup.connect("https://bitcointalk.org/index.php?action=profile;u=" + userId + ";sa=summary").get();

OmegaStarScream

staff

Activity: 3500

Merit: 6152

I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer

mocacinno

legendary

Activity: 3514

Merit: 5123

https://merel.mobi => buy facemasks with BTC/LTC

you can write a parser/scraper to just load the user's profile page, and parse their recent posts up untill the very first post..

A lot of work, but still doable Wink

lahm-44

sr. member

Activity: 336

Merit: 250

do i can fetch bitcointalk data like post count and activity like bit-x campaingn. i know they dont have api feature but i see some site/campaingns are fetching it

Topic: fetch bitcointalk data (Read 482 times)