I created the Bitcointalk Stats Live Counter

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: NotATether on August 24, 2021, 03:51:01 AM

As another example, if the entire forum started using cloudflare to load posts and stuff, then my post scraping script would probably break as well.

Yes, but they usually do this for “heavy” pages. A page of a thread isn't going to request much, but if you choose to get all the replies of the thread (by adding ;all next to it in the url) a cloudflare waiting screen will show up.

I really wonder, though, is the statistics page that heavy and why did they change their minds this year? Did it suddenly become heavier or they realized it was already heavy?

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: NotATether on August 24, 2021, 03:51:01 AM

It most likely thinks your script is a bot (which it is TBH) and then presents it a captcha that it doesn't have a chance at solving, at first load.

Nope, the Statistics Center shows Cloudflare's "Checking your browser"-page on my normal browser too (when I click it for the first time). It's the same thing that happens when you click "All" to see more than one page in a topic, that's what made me think it has to do with resource usage.

NotATether

legendary

Activity: 1568

Merit: 6660

bitcoincleanup.com / bitmixlist.org

Quote from: BlackHatCoiner on August 23, 2021, 01:32:51 PM

Quote from: LoyceV on August 23, 2021, 01:24:31 PM

Maybe it was too resource-intensive?

No, it doesn't have to do with that. Possibly Cloudflare protects bitcointalk from getting embedded.

It most likely thinks your script is a bot (which it is TBH) and then presents it a captcha that it doesn't have a chance at solving, at first load. People mainly use it to verify all the visitors are human.

As another example, if the entire forum started using cloudflare to load posts and stuff, then my post scraping script would probably break as well.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: LoyceV on August 23, 2021, 01:24:31 PM

Maybe it was too resource-intensive?

No, it doesn't have to do with that. Possibly Cloudflare protects bitcointalk from getting embedded.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: BlackHatCoiner on August 23, 2021, 11:42:17 AM

This site is now abandoned due to the usage of CloudFlare in the statistics page.

That's too bad Sad

Maybe it was too resource-intensive?
I've been downloading empty pages for the past 2 months too.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

This site is now abandoned due to the usage of CloudFlare in the statistics page.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: hosseinimr93 on January 21, 2021, 07:02:03 PM

The information displayed for dates January 16 and January 17 is wrong.
It seems to be a bug.

Yes. That's true. I went into the database directly to see why that and saw this:

A brief explanation.

Generally, I don't save the new posts, topics and members everyday. I just save the total posts, topics and members of every single day and once you want to get the info of one day I show you:

the_day_after - your_day

Example for 2021-01-21's members:
members = 3052229 (which are the total members of 01/22) - 3046556 (which are the total members of 01/21) = 5673

The day after is not written on the list until 01/23/2021 - 00:00 UTC.

Now, I don't understand why it wrote zero for everything in 2021-01-17. It must be a web hosting error? Or mine? I don't know. Unfortunately, since 01/16/2021 is affected by 01/17/2021, I'll have to remove those two dates. Nonetheless, I have seen this once again, but it was correct:

Quote

2020-12-24 : New posts: 9259, new topics: 180, new members: 2377
2020-12-25 : New posts: 8673, new topics: -10456, new members: 2692
2020-12-26 : New posts: 9160, new topics: 109, new members: 2535

The "new topics" seems bugged but it's not. An admin/mod just deleted over 10,000 topics that day.

hosseinimr93

legendary

Activity: 2380

Merit: 5213

The information displayed for dates January 16 and January 17 is wrong.
It seems to be a bug.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Spam alert! These are the stats of the last 3 days. Lots of farming accounts may be on their way.

Quote from: SFR10 on December 18, 2020, 02:31:01 AM

A cool feature indeed but you should probably consider moving it to a separate/unique page instead ^{[like all of the links at the bottom of the page]} since it's not connected to other stats from this forum

It's losing its point if I move it on a different directory. I moved it on the bottom, I think it's the best option I had.

SFR10

legendary

Activity: 2968

Merit: 3406

Crypto Swap Exchange

Quote from: BlackHatCoiner on December 17, 2020, 04:34:37 PM

I added this cool feature that sums all the UTXOs it receives. Then, it refreshes the number every 3 seconds to replace it with the new result. Try it!

A cool feature indeed but you should probably consider moving it to a separate/unique page instead ^{[like all of the links at the bottom of the page]} since it's not connected to other stats from this forum

Vod

legendary

Activity: 3668

Merit: 3010

Licking my boob since 1970

Quote from: BlackHatCoiner on December 17, 2020, 04:34:37 PM

I recently found out that blockchain.info provides an API that contains web sockets completely free.

If you want to download larger amounts of data, you'll find it slow or... not free.

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

I added this cool feature that sums all the UTXOs it receives. Then, it refreshes the number every 3 seconds to replace it with the new result. Try it!
These are lots of bitcoins and I got them in less than 1 minute. Mempool, the wild west...

I recently found out that blockchain.info provides an API that contains web sockets completely free. I was going to make a thread about that, but it seems that web sockets are available to the public for free.

SFR10

legendary

Activity: 2968

Merit: 3406

Crypto Swap Exchange

Quote from: BlackHatCoiner on November 25, 2020, 03:24:29 AM

I think that in order to fix this problem and make it 100% accurate I'll have to import the activity unix times on my database and then start calculating each rank based on the list. I can't figure out of a smarter way.

You'll need one more thing apart from that if you're after 100% accuracy:

Somehow go over the data for each profile ^[scrape] and exclude periods in which someone has been inactive, then do the automated calculation.
- Perhaps having a limit/interval of some sort in between profile searches, wouldn't put too much stress on the server.
  _{- I'm not a back-end coder but in theory, this should work.}

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

@Husna QA, mine's calculator is surely not accurate. Once I became Sr. Member it was 2 days late. I think that in order to fix this problem and make it 100% accurate I'll have to import the activity unix times on my database and then start calculating each rank based on the list. I can't figure out of a smarter way.

Husna QA

legendary

Activity: 2296

Merit: 2892

#SWGT CERTIK Audited

Quote from: ?? on ??

-snip- I don't know, maybe something like that has already been offered here, I haven't read the whole topic, but it was possible to add 2 lines in this calculator: -snip-

I tried to compare the BlackHatCoiner version of the Rank Calculator with Iasenko's version. I use my registration date data in this forum; there is a difference in the two versions' date. I don't know exactly which of the two versions is more accurate because I don't log the history as I move up the rankings at each level.

https://btctalk.live/rankcalculator/get.php?date=February%2012,%202018

https://docs.google.com/spreadsheets/d/1khSA28Ab6miwyknp3JvStcL0Yu05SIbGBDZDKcOLO6s/edit#gid=0

@BlackHatCoiner, by the way, I got this

https://btctalk.live/free-bitcoin/

dzungmobile

sr. member

Activity: 854

Merit: 424

I stand with Ukraine!

Quote from: LoyceV on November 22, 2020, 05:11:11 AM

Quote from: BlackHatCoiner on November 22, 2020, 05:05:44 AM

the rank calculator isn't working properly and I don't understand why. I've set to count 14 activity for every 2 weeks.

Those 14 Activity are adjusted on a fixed time (somewhere on Tuesday), did you account for that?

Somewhere from 5 AM to not more than 9 AM forum time. I guess so.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: BlackHatCoiner on November 22, 2020, 05:05:44 AM

the rank calculator isn't working properly and I don't understand why. I've set to count 14 activity for every 2 weeks.

Those 14 Activity are adjusted on a fixed time (somewhere on Tuesday), did you account for that?
That means a Newbie can reach 28 Activity within a day if he creates his account at the right moment (and creates 28 posts).

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

Quote from: ?? on ??

I don't know, maybe something like that has already been offered here, I haven't read the whole topic, but it was possible to add 2 lines in this calculator:

Code:

 Minimum Legendary Data
Maximum Legendary Data

I would do that it ain't anything special, but as you mentioned I've also noticed that the rank calculator isn't working properly and I don't understand why. I've set to count 14 activity for every 2 weeks.

LoyceV

legendary

Activity: 3290

Merit: 16489

Thick-Skinned Gang Leader and Golden Feather 2021

Quote from: BlackHatCoiner on November 15, 2020, 04:02:05 AM

Speaking of issues, I would find it nice to create a livecount for recent merit transactions. Recent merit stats can be found here, but the problem is that guests cannot see them, in which btctalk.live is based on. So I can't get those stats from there.

You'll need to scrape the data as a user. Or try Ninjastic.space - BitcoinTalk Post/Address archive + API, it currently doesn't have what you're looking for, but TryNinja might be able to add it.

Quote

On the other hand we have the LoyceV's database. The issue here is that these are enormous files that cannot be refreshed every 5 seconds.

It's also updated only once a week Wink

BlackHatCoiner

legendary

Activity: 1512

Merit: 7340

Farewell, Leo

I updated the OP. Also, I changed it a little bit. I always wanted to put a nice header like that.

Today and Alldates are back again after a break (some issues).
Stats from 09/09/2020 until 11/14/2020 are not available.

Speaking of issues, I would find it nice to create a livecount for recent merit transactions. Recent merit stats can be found here, but the problem is that guests cannot see them, in which btctalk.live is based on. So I can't get those stats from there. On the other hand we have the LoyceV's database. The issue here is that these are enormous files that cannot be refreshed every 5 seconds. So I'm ran out of ideas.

Topic: I created the Bitcointalk Stats Live Counter (Read 2819 times)