Author

Topic: [UPDATED] bitcoin/bitcointalk.org Stats Animation - all in one thread (Read 1376 times)

legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Here’s the data, up to last Friday.

Thanks DdmrDdmr. The data worked perfectly.

 I have updated the  OP with lots of new local animations, as requested by some people. Smiley
legendary
Activity: 2380
Merit: 17063
Fully fledged Merit Cycler - Golden Feather 22-23
<…>
I’m still using the forum’s search function to detect mentions, but somehow skipped this one, and wasn’t really aware of the petition until @fillippone posted today …


You know, @ddmrddmr, resistance is futile.
Time has come:
#meritislife: how to be notified on a smartband of merits and mentions

Join the army of the hairy wrist band so you will never skip a mention ever!
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<…>
I’m still using the forum’s search function to detect mentions, but somehow skipped this one, and wasn’t really aware of the petition until @fillippone posted today …

Here’s the data, up to last Friday. I’ve exported it using a different method, but the format should be the same as on prior occasions:
https://drive.google.com/file/d/1wdUBx9p0KTtbxlucBm_S6CIR9UqOgRxs/view?usp=sharing

Let me know if anything is off in the file.

<…>
I’m must be old fashioned (which doesn’t mean I won’t cave-in eventually): I want to go and get my notifications at my convenience and pace …
legendary
Activity: 2380
Merit: 17063
Fully fledged Merit Cycler - Golden Feather 22-23
Hey,
I know this thread hasn’t been updated for a long time.
Is there the possibility to have a merit race for the Italian board?
Or any new idea you came up with in those few months?
Thanks!

Of course!
For local boards I need some extra data from @DdmrDdmr. Can you please send me an updated file of including Section/subsection column? no need to have too many details, I will consider any local child board as local only.
Thank you DdmrDdmr

Any update on this?
@ddmddmr can you do that for us?
I have just updated my meta post, and I would like to include that video in the post!


legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Hey,
I know this thread hasn’t been updated for a long time.
Is there the possibility to have a merit race for the Italian board?
Or any new idea you came up with in those few months?
Thanks!

Of course!
For local boards I need some extra data from @DdmrDdmr. Can you please send me an updated file of including Section/subsection column? no need to have too many details, I will consider any local child board as local only.
Thank you DdmrDdmr
legendary
Activity: 2380
Merit: 17063
Fully fledged Merit Cycler - Golden Feather 22-23
Hey,
I know this thread hasn’t been updated for a long time.
Is there the possibility to have a merit race for the Italian board?
Or any new idea you came up with in those few months?
Thanks!
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<...>
I’ve uploaded a file with the same structure as last time to the following link:
https://drive.google.com/file/d/1dANGltunbyBo1JwNaVLUhdGfQVnTdR4O/view?usp=sharing

The file is a compressed file, that contains the flat file "TXs_Bitmover_export_20210228.txt", with data as of last Friday.

Bear in mind that I haven’t had my morning coffee yet, so please check that the content is as expected…
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
You can use the following file: https://drive.google.com/file/d/1rOvFPbd9T6q5qrtLtAD1gy4fBMcUrg3Z/view?usp=sharing
It contains all the historical TXs, including the board description (section/subsection) It occupies some 75 MBs. You’ll have to load them to a DB or whatever repository you are going to use, and filter by subsection = Turkish (last column).

The data is as of last Friday 19/06/2020. How you create a the race itself is up to you (python, Tableau, etc.).


Hello @DdmrDdmr,

Can you update this file again, please?
I have some ideas about new charts, a network chart for some local boards.

Thank you.
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
<…>
Since tomorrow we receive the new Merit.txt file, it’s probably better to wait for that update, in order to have the most recent version. It should therefore be ready tomorrow (max. Saturday). I’ll let you know when it’s done, and I’ll trim the information on child boards from it (leaving the granularity at subsection level again).

Edit: File-> https://drive.google.com/file/d/1meWPgG_bIyChkUIcxHhTQnB4WPwp1oAV/view?usp=sharing


Thank you DdmrDdmr.

The page was just updated with the latest data
 link
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<…>
Since tomorrow we receive the new Merit.txt file, it’s probably better to wait for that update, in order to have the most recent version. It should therefore be ready tomorrow (max. Saturday). I’ll let you know when it’s done, and I’ll trim the information on child boards from it (leaving the granularity at subsection level again).

Edit: File-> https://drive.google.com/file/d/1meWPgG_bIyChkUIcxHhTQnB4WPwp1oAV/view?usp=sharing
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
<...>
I've uploaded all the TXs to the following file, including the TXs from today’s merit.txt file:
https://drive.google.com/file/d/1MnRa6e2O0RH7K2gw7AUoT7FwK8rHbX1-/view?usp=sharing


Hello DdmrDdmr.
Do you have an updated file for merit transactions? So i can update the merit table.
It doesn't need to be so big like the old one. I decided not to include those child boards, and focus more in other projects. Please share the link with me when you can.

Thank you Smiley
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Wow, well done, glad to see this Wink

I have a suggestion for you. Maybe you can disable report from zero value column. it only causes additional scrolling.

Thanks examplens Smiley

I liked your suggestion, however I am not able to do it for now.
As I explained here, I tried to generate the table myself using bootstrap, which would allow a lot of customization options. But the loading time was terrible. I think I need to learn more about SQL or AJAX or soemthing like that to make a server side processing.

I am not able to do it.

I tried to use bootstrap + datatable.net to create the table, so I would have more customization options. However, I do not know how to efficiently parse the data, and my HTML file had 30mb and consumed a lot of processing power.

Flourish is very efficient in parsing the data, so I will keep the table in this format and move on to other projects. At least for now. Maybe in the future as I learn more about JS and SQL...



Quote
Also when we talking about the scroll, if it's possible to change horizontal to vertical scroll, it is easier to use on desktop pc. That will be fine.
Tried on a mobile device, and there is okay. Table fields are placed one below the other.

This I can do. This is a dataframe transpose. And it is really a nice suggestion. I will make some tests Smiley
legendary
Activity: 3472
Merit: 3507
Crypto Swap Exchange
The process was so painful that I intentionally edited the file in notepad to change it, intentionally breaking unwanted characters. Smiley

I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html

I will try to work in the new categories (path l4 first) on Monday.


Wow, well done, glad to see this Wink

I have a suggestion for you. Maybe you can disable report from zero value column. it only causes additional scrolling. Also when we talking about the scroll, if it's possible to change horizontal to vertical scroll, it is easier to use on desktop pc. That will be fine.
Tried on a mobile device, and there is okay. Table fields are placed one below the other.
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<…> I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html <...>
Looks good. We differ a bit on the last decimal of the percentages, but minimal, likely due to using different rounding precision.

<…> I just changed encoding to UTF-16 and it worked without too much a hashle (just did a few adjustments). My line 20 is intact now and I can see Aльтepнaтивныe кpиптoвaлюты correctly <…>
Great! slayed the Unicode dragon pretty swiftly there.
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Ideally, without needing to edit the file in between, since the editor may change the format and be part of the cause. This is normally a small painful process of getting the environment to interpret this kind of file properly.

The process was so painful that I intentionally edited the file in notepad to change it, intentionally breaking unwanted characters. Smiley

I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html

I will try to work in the new categories (path l4 first) on Monday.



Edit:

The resulting file opens with Excel seamlessly, as does notepad (notepad indicates that the format is actually UTF 16).

I just changed encoding to UTF-16 and it worked without too much a hashle (just did a few adjustments). My line 20 is intact now and I can see Aльтepнaтивныe кpиптoвaлюты correctly Wink
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<...>
Actually, the data I scrape and store goes to a table where the fields are defined as nVarchar (as opposed to varchar). nVarchar supports Unicode characters, which is what I need to store the scraped information whilst preserving the content’s format.

Due to the above, the export I created requires me to mark Unicode in the export parametrization (forget the UTF8  -> Unicode overrides it even if I select it).

The resulting file opens with Excel seamlessly, as does notepad (notepad indicates that the format is actually UTF 16). You probable need to see how to treat Unicode files in your environment without all those prior conversions. Ideally, without needing to edit the file in between, since the editor may change the format and be part of the cause. This is normally a small painful process of getting the environment to interpret this kind of file properly.

legendary
Activity: 2352
Merit: 6089
bitcoindata.science
The "path" field (last field) does indeed look weird (*) now on your quoted example, but the rest seem ok.

Check if row number 20 shows as this:
Quote
"20","2018-01-24 22:33:27","1516833207","1189487","Prometheu$","Full Member","1028592","nikola22","Full Member","1","https://bitcointalksearch.org/topic/m.28837085","Local","Russian","Aльтepнaтивныe кpиптoвaлюты ","","","Bitcoin Forum > Local > Pyccкий (Russian) > Aльтepнaтивныe кpиптoвaлюты (Moderators: xandry, Xal0lex) > [ANN][ICO] TOKENGO BUSINESS TOKENIZATION PLATFORM - Bonus up to 50%"
If the fields are displayed as above (and not as a bunch of question marks), you’re good to go.



This is my line 20

Code:

"20","2018-01-24 22:33:27","1516833207","1189487","Prometheu$","Full Member","1028592","nikola22","Full Member","1","https://bitcointalksearch.org/topic/m.28837085","Local","Russian","Альтернативные криптовалюты ","","","Bitcoin Forum > Local > Русский (Russian) > Альтернативные криптовалюты (Moderators: xandry, Xal0lex) > [ANN][ICO] í ¼í¼€TOKENGOí ¼í¼€ BUSINESS TOKENIZATION PLATFORM - Bonus up to 50%"


I can clean this with regex. I will just lose the pathl4 column in some rows (specially locals), which is the one with this crazy "    ÐÐ»ÑŒÑ‚ернативные криптоваÐ...    "...
But, overall I will can use pathl4 column, unless it has some crazy encoding.

How did you manage to work with this kind of characters?     льтepнaтивныe кpип and     . Are you using UTF-8?
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<...>
<…>
Yep, I sometimes encounter issues with encodings I receive on files. Takes some time fighting to overcome it. If you’ve now got a method to load the file, that is not too much of a hassle, we can leave the file as is (or go over it if you want me to change the format).

The "path" field (last field) does indeed look weird (*) now on your quoted example, but the rest seem ok.

Check if row number 20 shows as this:
The problem is being caused by UTF-8 bytes being interpreted as Windows-1252 (or ISO 8859-1) bytes.
https://www.i18nqa.com/debug/utf8-debug.html
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Let's see if you can load this file properly. I has to make it Unicode because of the non Latin characters - i.e. russian charactrs on path_l4, which is the first childboard level. RowId number 20 is the first case with such characters.


Thanks for that.

This file was not easily parsed... I had to change a lot of configurations in python notebook, even data rate limit... I learned a lot parsing your data.

I also had to open it in Notepad and I changed it encoding manually to UTF-8, breaking all those special characters (as I won't use them anyway)

My final csv was like this:

Code:
"8","2018-01-24 22:29:38","1516832978","1344962","dspencer","Member","1101839","Scout Georgenson","Jr. Member","1","https://bitcointalk.org/index.php?topic=2384335.msg28854772#msg28854772","Alternate cryptocurrencies","Announcements (Altcoins)","Tokens (Altcoins) (Moderators: mprep, Welsh)","","","Bitcoin Forum > Alternate cryptocurrencies > Announcements (Altcoins) > Tokens (Altcoins) (Moderators: mprep, Welsh) > [ANN] [ICO] Xwin – CryptoBet betting platform based on blockchain"

then I was finally able to open your dataset Smiley

Thank you again. I will work in that data to update my Super Merit table soon.
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<...>
I've uploaded all the TXs to the following file, including the TXs from today’s merit.txt file:
https://drive.google.com/file/d/1MnRa6e2O0RH7K2gw7AUoT7FwK8rHbX1-/view?usp=sharing

Observations:
-   I’ve only uploaded the .rar file, which is 16 Mb. If you’d rather have I upload the csv file without compression, tell me (264 Mb).
-   The files is csv, with quote qualifiers.
-   Code page is 1252 (ANSI – Latin 1) and Unicode.
-   The file contains the additional fields described here: https://bitcointalksearch.org/topic/m.54895308

Let's see if you can load this file properly. I has to make it Unicode because of the non Latin characters - i.e. russian charactrs on path_l4, which is the first childboard level. RowId number 20 is the first case with such characters.
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Just in case you prefer using a more recent dataset, I’ve uploaded the file with all TXs as of last Friday. I’ve not overwritten my prior file just in case you prefer to use that dataset.

The newly uploaded files are located here:

CSV version (77Mb):
https://drive.google.com/file/d/1kOuW7gO-Om2X8zvwJhSrEet5eNjlvSRu/view?usp=sharing

Rar version (9Mb):
https://drive.google.com/file/d/1UY7YON7y-DzUGYIWh1V3jOt-P7l7kmS8/view?usp=sharing

Hello DdmrDdmr.

Can you share with me your last updated data from merit dashboard again?
I am using it in my merit table (https://www.bitcoindata.science/supertable.html) and in some other projects as well. Smiley
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Just added Super Merit table and Turkish merit race.

To do:

-More local boards animations per request. Portuguese is the obvious next one Wink
-More tables and different graphic visualizations with the new DdmrDdmr data. As now I can get data per board/topic I have wider possibilities
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Made some updates:

- Added DdmrDdmr data. Now i can make animations and graphics per board. I am open to requests
- Made Wall Observer board animation
- Added Avatars to all animations.

How can I do this for the local Turkish section?

Hello Tom Bombadil. Yes, I can do that animation now. I will do as soon as I can.
TUrkish and any other request board (I can do any local)
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
Just in case you prefer using a more recent dataset, I’ve uploaded the file with all TXs as of last Friday. I’ve not overwritten my prior file just in case you prefer to use that dataset.

The newly uploaded files are located here:

CSV version (77Mb):
https://drive.google.com/file/d/1kOuW7gO-Om2X8zvwJhSrEet5eNjlvSRu/view?usp=sharing

Rar version (9Mb):
https://drive.google.com/file/d/1UY7YON7y-DzUGYIWh1V3jOt-P7l7kmS8/view?usp=sharing
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
<…> There is data in the @DdmrDdmr application. But I can't copy the data from there. I wish it was possible to copy the tables @DdmrDdmr  Roll Eyes
You can use the following file: https://drive.google.com/file/d/1rOvFPbd9T6q5qrtLtAD1gy4fBMcUrg3Z/view?usp=sharing
It contains all the historical TXs, including the board description (section/subsection) It occupies some 75 MBs. You’ll have to load them to a DB or whatever repository you are going to use, and filter by subsection = Turkish (last column).

The data is as of last Friday 19/06/2020. How you create a the race itself is up to you (python, Tableau, etc.).


It took me a while, but I downloaded it today. Very good and organized data... I will do something with it in a few days . I will start with wall observer and then I will make other graphics for local boards.
Thank you.
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
Small updated:

I added Exchanges Volume Race. You can see the direct link to the animation here[/td][td]|[/td][/tr][/table].

This is a very poor data and hard to find. But, anyway, I made the animation. I was able to get data since 2010. Sadly, missing binance and  a few others.
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
How can I do this for the local Turkish section?

I can create a race for selected users in Turkish section. just give me a list of all the users you want, and I can make their race. However, merits earned outside the Turkish board would still be counted. But you can select user who post most of their time in the Turkish board.
legendary
Activity: 2338
Merit: 10802
There are lies, damned lies and statistics. MTwain
<…> There is data in the @DdmrDdmr application. But I can't copy the data from there. I wish it was possible to copy the tables @DdmrDdmr  Roll Eyes
You can use the following file: https://drive.google.com/file/d/1rOvFPbd9T6q5qrtLtAD1gy4fBMcUrg3Z/view?usp=sharing
It contains all the historical TXs, including the board description (section/subsection) It occupies some 75 MBs. You’ll have to load them to a DB or whatever repository you are going to use, and filter by subsection = Turkish (last column).

The data is as of last Friday 19/06/2020. How you create a the race itself is up to you (python, Tableau, etc.).
newbie
Activity: 20
Merit: 5
Are the lists above based on participation times or data?some accounts are inactive so its hard to count them

Do you need to update it periodically, or does it update automatically?
There are lists of people who have become legendary since the merit system was established. Can you prepare an illustration that explains the competition between them and how quickly they reach these ranks?
sr. member
Activity: 364
Merit: 304
How can I do this for the local Turkish section?
Get all topicIDs on the Turkish board, then get all Merit transactions for those topics.
It means scraping about 1000 pages, so you should wait at least 1 second between requests.

Wow! Shocked It looks crazy. It seems to last for days. Thousands of topics have been opened since 2018.
Thank you anyway, @LoyceV
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
How can I do this for the local Turkish section?
Get all topicIDs on the Turkish board, then get all Merit transactions for those topics.
It means scraping about 1000 pages, so you should wait at least 1 second between requests.
sr. member
Activity: 364
Merit: 304
How can I do this for the local Turkish section?

There is data in the @DdmrDdmr application. But I can't copy the data from there. I wish it was possible to copy the tables @DdmrDdmr  Roll Eyes
legendary
Activity: 2758
Merit: 6830
Not sure how you would calculate activity on a topic or post.  :/
Maybe he meant activity as the number of posts in a thread rather than the actual "activity points". It would be interesting to see the Wall Observer thread making his way to the top 1. But someone would need to scrape those posts and make a list available with the time of each post.
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
I would like to see animated comparison of bitcointalk topics with most activity and their progress in time.

Not sure how you would calculate activity on a topic or post.  :/
legendary
Activity: 2086
Merit: 1282
Logo Designer ⛨ BSFL Division1
Interesting stats animations.
I would like to see animated comparison of bitcointalk topics with most activity and their progress in time.
Not sure it is possible or easy to do this.
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
Are you saying there's some "backscratching" going on? Returning the favor when someone gets Merited, thus leading to more "backscratching" for Merit sources, because they Merit more users?

I don't think it's intentional abuse.  If someone receives merit they were not expecting, it may be easier for them to find your post merit worthy.  It's human nature to like and help those that helped you.  In my experience, if you recognize talent with a merit, they will support you, even subconsciously, and it makes them feel good (dopamine hit) to return the favor.  Smiley

legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
Once I stopped being a source, the quality of my posts seemed to drop ten fold.  :/
Are you saying there's some "backscratching" going on? Returning the favor when someone gets Merited, thus leading to more "backscratching" for Merit sources, because they Merit more users? I think it happens a bit, I've seen Merit transactions shortly after sending Merit. But it could also be first Merit transaction brought the Merit sender to the receiver's attention, and if he reads more of his posts, the posts might deserve it.
legendary
Activity: 2254
Merit: 2406
Playgram - The Telegram Casino
Any graph that involves merit sources should have those lines "glow" so we can separate them from the non-sources easily.
This would not be possible as not all sources have publicly admitted that they are.
Another idea would be a graph of merit against activity, to show how quickly one earned merits based on their activity score it could also be used to check how long one stayed at a particular rank. I would imagine satoshi would have a straight line (merit growth)
I don't know if this could be made into a group data or per-user.
Vod
legendary
Activity: 3668
Merit: 3010
Licking my boob since 1970
I think it is better to see (also) the race in merit per member. I mean there might be memberX spending 1-2 merits each time to 1000 members that is behind memberY who spent 5-10 merits to 100 members for example.
I like the idea, an animation for the number times a user received a Merit transaction would make me #1 Tongue

I like the idea as well.  Any graph that involves merit sources should have those lines "glow" so we can separate them from the non-sources easily.   Once I stopped being a source, the quality of my posts seemed to drop ten fold.  :/
legendary
Activity: 3290
Merit: 16489
Thick-Skinned Gang Leader and Golden Feather 2021
I think it is better to see (also) the race in merit per member. I mean there might be memberX spending 1-2 merits each time to 1000 members that is behind memberY who spent 5-10 merits to 100 members for example.
I like the idea, an animation for the number times a user received a Merit transaction would make me #1 Tongue
legendary
Activity: 3178
Merit: 1140
#SWGT CERTIK Audited
I think it is better to see (also) the race in merit per member. I mean there might be memberX spending 1-2 merits each time to 1000 members that is behind memberY who spent 5-10 merits to 100 members for example. I find the first guys more interesting to see Smiley
BTW nice OP
full member
Activity: 868
Merit: 185
Roobet supporter and player!
Nice innovation OP.
I saw those three animation race from sender of merits, received merit and DT1 strength. And those are cool. Do you have plan to add some more stats? I suggest to make also some race stats about trust (positive/negative), posts which are the most numerous posts and activity of the forum and most numerous thread makers.

I am currently studying about python programming and it's really hard to learn about codes specially if there is no resources.
hero member
Activity: 1722
Merit: 801
I think you can post one photo for all types of race to make your topic visually. For the other races, you can list them in a table.

The link to DT1 strength race in OP is not correct. It is 2881001, not 2870679.


|
ANN thread link
legendary
Activity: 2352
Merit: 6089
bitcoindata.science
[WARNING] Some Data is not up to date.

This post is a compilation to my other Race animations regarding bitcointalk and bitcoin stats.
I will keep this thread updated as my work progresses.




|
ANN thread link
Jump to:
© 2020, Bitcointalksearch.org