Excuse me, I've locked the thread
LOL
Topic: Is there an API on bitcointalk? (Read 323 times)
As usual, you guys really like to go off topic
Excuse me, I've locked the thread
LOL
Excuse me, I've locked the thread
LOL
Since another member advised me not to get involved in reputation boards
No one should take away your freedom. There is no restriction in the forum, you can contribute anywhere you wish.I am surprised that the reputation drama had been in existence even when you were a newbie. I thought it is a new generation drama.
Meanwhile, I had to check the location of "Switzerland" in the map, to see if it's isolated. So, I didn't catch the joke in the nickname "Switzerland"
I have a small furniture business, and I market via Instagram. So I thought if I could monitor the ikea site or any major site in my country that sells furniture it might be easy for me to decide what to sell. But that's for the future, I'm still a small business. And I just got this method after seeing the scraping site model. Maybe it can be more effective than SEO and advertising
You can even do that, and get interesting data to analyze. But it will be difficult for you to understand the real shopping trends that are being made, since this data is not public.
Remember that scraping systems only work for public data. If the information is not public, it will not be collected.
Now, of course, it makes no sense to collect data if you don't have any specific objective, if you're not just going to waste time.
Yes you're right, And it seems I have too much time to waste
I have a small furniture business, and I market via Instagram. So I thought if I could monitor the ikea site or any major site in my country that sells furniture it might be easy for me to decide what to sell. But that's for the future, I'm still a small business. And I just got this method after seeing the scraping site model. Maybe it can be more effective than SEO and advertising
Since another member advised me not to get involved in reputation boards
No one should take away your freedom. There is no restriction in the forum, you can contribute anywhere you wish.Yes you're right. Maybe at first I thought this site was the same as other discussion sites, and I liked the discussion so I tried to fight the arguments of other members, but I chose the wrong opponent. And some members recommended me to step down from the reputation board, and now I realize that is the best choice
Since another member advised me not to get involved in reputation boards
No one should take away your freedom. There is no restriction in the forum, you can contribute anywhere you wish.No, no. I don't have a specific goal. I just want to know how this process works. I know I won't become an expert by just reading on the internet, especially since this mechanism is too difficult for someone who really doesn't understand coding.
Sometimes you don't even need to know much about coding.
Using the program I mentioned, you can get almost any information you want that is public on the forum.
Now, of course, it makes no sense to collect data if you don't have any specific objective, if you're not just going to waste time.
There's no Bitcointalk API. All of the data you see floating around comes from the official Simple Machines Forum endpoints (which also powers the forum frontend) and all information can be gained from the path index.php?action=blablabla;more=parameters;follow=here. It's just that you will get a ton of HTML along with this request, which needs to be filtered and parsed. But as you probably figured, there is unofficial rate-limiting on the whole website; you can't make more than one request per second, or theymos blocks your IP address.
Those above-mentioned users are all programmers or developers; they develop their own tools used for specific things.
Those tools are being used to scrape data from the forum, but it requires some kind of approval from the forum administrator (some sort of IP whitelisting) in order to grant their tool access to the data they seek.
Those are how I understood it. I am not a programmer either.
I was going to say that they don't need special permission from theymos before they could scrap the forum until LoyceV said so. It is dedication, if it is something you want to do, you can do it. But there are many people in the data scraping of a thing and they are doing great.Those tools are being used to scrape data from the forum, but it requires some kind of approval from the forum administrator (some sort of IP whitelisting) in order to grant their tool access to the data they seek.
Those are how I understood it. I am not a programmer either.
Since another member advised me not to get involved in reputation boards, I was confused about what to do in the forum, so I studied your tool a bit, Loycev.club. also tried using ninjasctic, tableu DdmrDdmr. From there my curiosity emerged.
I have to ask: what are you trying to accomplish?
No, no. I don't have a specific goal. I just want to know how this process works. I know I won't become an expert by just reading on the internet, especially since this mechanism is too difficult for someone who really doesn't understand coding.
Since another member advised me not to get involved in reputation boards, I was confused about what to do in the forum, so I studied your tool a bit, Loycev.club. also tried using ninjasctic, tableu DdmrDdmr. From there my curiosity emerged.
Is this method also effective for business? like spying on web competitors?
I do not scrape the site. But, some time ago, I wanted to collect some information, and I used the Octoparse software, which worked perfectly for what I wanted.
Therefore, anyone can scrape the forum without major problems. You just have to know what you want and use the right tools for it.
Therefore, anyone can scrape the forum without major problems. You just have to know what you want and use the right tools for it.
How did the scrapping work last week when Cloudflare was acting up
Scraping didn't work.My bot was down for almost 2 full days (until theymos apparently whitelisted our IPs so we could bypass it).
I figured I'd ask theymos. His previous whitelist attempt (last December) didn't work, and now he fixed it 
How did the scrapping work last week when Cloudflare was acting up and Bitcointalk was running slowly or not at all for certain actions? Some users reported they couldn't post, others submitted multiple posts in row, I had problems editing and previewing posts, etc. It wasn't DDOS-ing but still a Cloudflare issue. Did it affect any of your regular scrapping work?
Scraping is pretty much impossible when Cloudflare is cranked up. Requests are blocked (403 error) and there are captchas everywhere. 
My bot was down for almost 2 full days (until theymos apparently whitelisted our IPs so we could bypass it).
That's not true. Anyone can scrape the forum, as long as they keep it under 1 request per second. The IP whitelisting is only needed when Cloudflare becomes very active against DDOS.
How did the scrapping work last week when Cloudflare was acting up and Bitcointalk was running slowly or not at all for certain actions? Some users reported they couldn't post, others submitted multiple posts in row, I had problems editing and previewing posts, etc. It wasn't DDOS-ing but still a Cloudflare issue. Did it affect any of your regular scrapping work?
I also unofficially scrape the forum. I mostly use Javascript’s fetch to make requests and cheerio to parse most of the data.
My code is open source so no secrets there: https://github.com/ninjastic/bitcointalk-supernotifier-v2
My code is open source so no secrets there: https://github.com/ninjastic/bitcointalk-supernotifier-v2
The only API that I remember seeing is the one developed by TryNinja[1] for his Ninjastic.space[2] project. Do note that this isn't an official API for the forum but since TryNinja already scraps the forum, he setup this API for users that could have any need to directly interact with some of data that is collected. Depending on your needs you can talk with TryNinja to see if he's able to help you out:
[1]https://docs.ninjastic.space
[2]https://bitcointalksearch.org/topic/ninjasticspace-bitcointalk-postaddress-archive-api-5273824
Quote
API:
If you have a cool project or project idea that requires any posts/addresses data, I can help you with my REST API.
Here is the documentation: https://docs.ninjastic.space
If you have a cool project or project idea that requires any posts/addresses data, I can help you with my REST API.
Here is the documentation: https://docs.ninjastic.space
[1]https://docs.ninjastic.space
[2]https://bitcointalksearch.org/topic/ninjasticspace-bitcointalk-postaddress-archive-api-5273824
you are like other members and don't have special access to the forum?
Correct.Quote
Can you tell me what mechanism you use? I'm honestly curious how it could work
I just use some scripting.Quote
Can you reference the website to study this?
There's Wget - GNU Project, but that won't help you much. As a Linux user, you can do many things once you learn how to use the command line. But anything else that works for you will do, downloading from the internet is no rocket science.I have to ask: what are you trying to accomplish?
Depending on what data I need, I use Patrol, Recent, data dumps or some of the "normal" pages on the forum (such as the Merit page, user profile or just pages in a topic).
So what are you all doing from outside the forum? I mean, you are like other members and don't have special access to the forum?
I'm not a programmer, but I've tried to learn programming languages by myself, but failed because of my busy life. Can you tell me what mechanism you use? I'm honestly curious how it could work
but wget is the easiest.
Can you reference the website to study this?
LoyceV's answer was that he uses wget to get data from website.
That's the answer to "how", not to "where". There are many more command line tools for downloading, but wget is the easiest. 
I once asked similar question in Bpip.org ANN thread and LoyceV's answer was that he uses wget to get data from website.
Where does the data owned by DdmrDdmr, LoyceV, and ninjastic.space come from?
Depending on what data I need, I use Patrol, Recent, data dumps or some of the "normal" pages on the forum (such as the Merit page, user profile or just pages in a topic).it requires some kind of approval from the forum administrator (some sort of IP whitelisting) in order to grant their tool access to the data they seek.
That's not true. Anyone can scrape the forum, as long as they keep it under 1 request per second. The IP whitelisting is only needed when Cloudflare becomes very active against DDOS.
Where does the data owned by DdmrDdmr, LoyceV, and ninjastic.space come from? Is there an API from bitcointalk?
I don't know how LoyceV, and ninjaastic.space collect data but you can fetch topics/posts from pages like unread posts since last visit ---> https://bitcointalk.org/index.php?action=unread
Then all you will do is analyze, collect and organize the data, whether using public.tableau.com or any personal tool.
You will only need to have the IP whitelisting done by admin due to high requests which may be blocked by Cloudflare
This code[1] is old, but it can explain the idea to you.
Some data like trust are weekly updated but it's a good opportunity to tell us how they do that or help who wants to learn how to make data analysis like that.
[1] https://github.com/mprep-btc/Unofficial-Bitcointalk-API
In simple words those 3 scrape data from bitcointalk (post/reply, users, merits, trust), save it on their servers and offers public api to access those data, well at least ninjastic.space have it[1], i dont know the other two.
[1] https://bitcointalksearch.org/topic/ninjasticspace-bitcointalk-postaddress-archive-api-5273824
[1] https://bitcointalksearch.org/topic/ninjasticspace-bitcointalk-postaddress-archive-api-5273824
Those above-mentioned users are all programmers or developers; they develop their own tools used for specific things.
Those tools are being used to scrape data from the forum, but it requires some kind of approval from the forum administrator (some sort of IP whitelisting) in order to grant their tool access to the data they seek.
Those are how I understood it. I am not a programmer either.
Those tools are being used to scrape data from the forum, but it requires some kind of approval from the forum administrator (some sort of IP whitelisting) in order to grant their tool access to the data they seek.
Those are how I understood it. I am not a programmer either.
I'm curious mate,
Where does the data owned by DdmrDdmr, LoyceV, and ninjastic.space come from? Is there an API from bitcointalk?
I am not an expert in the field of websites. I'm just a blogger
Where does the data owned by DdmrDdmr, LoyceV, and ninjastic.space come from? Is there an API from bitcointalk?
I am not an expert in the field of websites. I'm just a blogger
Jump to: