Author

Topic: Querying bitcointalk from script (Read 201 times)

legendary
Activity: 3304
Merit: 8633
icarus-cards.eu
December 11, 2022, 04:17:55 AM
#11
Nothing complex just small query for this a small quiz by cygan https://bitcointalksearch.org/topic/m.61422530

Isn't that like cheating? Just saying...  Grin


you are welcome to join us today for the next round and then prove your skills and quickness again Grin

Nothing complex just small query for this a small quiz by cygan https://bitcointalksearch.org/topic/m.61422530

Isn't that like cheating? Just saying...  Grin


Well not really... because when I saw 3 answers to that question only then I thought of checking how many legendary users there are fitting this criteria  _ _ _ t _ _ _ _ and found out there are quite few.


this is how it looks Smiley
copper member
Activity: 1526
Merit: 2890
December 11, 2022, 03:04:24 AM
#10
Nothing complex just small query for this a small quiz by cygan https://bitcointalksearch.org/topic/m.61422530

Isn't that like cheating? Just saying...  Grin


Well not really... because when I saw 3 answers to that question only then I thought of checking how many legendary users there are fitting this criteria  _ _ _ t _ _ _ _ and found out there are quite few.
legendary
Activity: 1820
Merit: 2700
Crypto Swap Exchange
December 10, 2022, 02:30:30 PM
#9
Nothing complex just small query for this a small quiz by cygan https://bitcointalksearch.org/topic/m.61422530

Isn't that like cheating? Just saying...  Grin
legendary
Activity: 2800
Merit: 2736
Farewell LEO: o_e_l_e_o
December 10, 2022, 01:05:47 PM
#8
Shit Bot was supper fast 🤣. [...]
You are clearly confusing God with Bots.
I am not, Bot knows about it 😉

You generally will not get permission to make more frequent queries unless you are paying for some kind of API access.
Is it something to do with the IP where the forum is hosted or the system admin can create any rule for their own interest? I don't see theymos have any intention to make money from selling data.
copper member
Activity: 1666
Merit: 1901
Amazon Prime Member #7
December 10, 2022, 10:22:09 AM
#7
[...]
Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

Edit: Now TryNinja too LOL

In general, unless you have permission otherwise, a good rule of thumb is to limit your queries to a website to 1 per second. You generally will not get permission to make more frequent queries unless you are paying for some kind of API access.
copper member
Activity: 1526
Merit: 2890
December 10, 2022, 09:31:24 AM
#6
Depending on what data you're looking for, someone may have it already. If it's something I have, a data dump can save you months of scraping.

Yes of course LoyceV your dumps are the first ones I will always refer to and in fact I'm using one right now from here related to usernames https://bitcointalksearch.org/topic/loycevs-complete-list-of-all-bitcointalk-usernames-5202231

Nothing complex just small query for this a small quiz by cygan https://bitcointalksearch.org/topic/m.61422530



I also have some data if that helps: https://docs.ninjastic.space

And can also give you a data dump on request (Grin).

Yes I have seen this too and thank you for that.

Actually you, LoyceV and DdmrDdmr too and I know you guys have very useful data dumps will always be appreciated.



For me I don't do much but for my own searches mostly, I also have data of some the users e.g Satoshi, Hal Finney, theymos and of course some of my favorite posters on WO (i don't want to name..) and I use it to query and refer to old posts for a quick search.

For example this or when recently LoyceV said that he did not use word vibrant so I thought of confirming myself and at first I found he did use it but... later on manual inspection I found that it is a quoted word but quoted in a wrong way, wrong in a sense that bots will not know, and here too, that's where the AI fails Smiley



Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.


You are clearly confusing God with Bots.


hero member
Activity: 1554
Merit: 880
Notify wallet transaction @txnNotifierBot
December 10, 2022, 08:34:49 AM
#5
I bet @Tryninja, @mprep and @LoyceV and other forum tech geeks can answer you about this, mentioning them might be a big help.

After some research, although i don't know exactly about the script you're trying to make, but here's an answer from @Tryninja for the telegram bot he made for probably scrapping data from bitcointalk, i don't know actually.
-What's the frequency of the scrapping? giammangiato's is 5 minutes for merits and 1 for quotes; and it sometimes takes a lot longer to do so.
Mentions/quotes every 5 seconds.
Merits every 25 seconds with a delay per each user. edit: increased to 30 seconds + 5 seconds  per user.

I tested and it's quite faster than the one from giammangiato (noticed 1 min VS 30 min merit notification with few users, besides the complete merit information). Not that it matters a lot (no competition here), but it's still nice.

Edit: Wew, lol. Seems like it take a lot for me to type this reply, while all the mentioned guys already made a reply.
legendary
Activity: 2800
Merit: 2736
Farewell LEO: o_e_l_e_o
December 10, 2022, 08:32:50 AM
#4
[...]
Shit Bot was supper fast 🤣. I was you in mind while I was writing the response to OP suggesting to keep it longer than 1 second. Long ago I read somewhere that a safe zone is to have 1 second delay between queries.

Edit: Now TryNinja too LOL
legendary
Activity: 2758
Merit: 6830
December 10, 2022, 08:30:31 AM
#3
From theymos himself:

Yes, but:

 - All of the posting rules still apply to bots.
 - You are not allowed to send more than one HTTP request to the forum per second on average.

I also have some data if that helps: https://docs.ninjastic.space

And can also give you a data dump on request (Grin).
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
December 10, 2022, 08:29:17 AM
#2
can I decrease it to 1 sec
Yes.

Quote
or less?
No. You'll get rate limited if you do that (although a short burst period is allowed).



Depending on what data you're looking for, someone may have it already. If it's something I have, a data dump can save you months of scraping.
copper member
Activity: 1526
Merit: 2890
December 10, 2022, 08:26:12 AM
#1
I tried to google but could not find an answer.

If I'm querying bitcointalk from a script what's the minimum time I should leave between each call? To avoid any IP ban or tagged as DDoS?

I have been doing this for quite some time and I always keep e.g. 2~2.5 second delay between each request. But this seems very slow now can I decrease it to 1 sec or less?


Jump to: