Topic: Blockdata Dashboard - Daily and Weekly data! 📊 (Read 553 times)

Learning a database is just as easy once you understand the concept of value pair.  Your hesitation may be caused by SQL - and the fact you can change large amounts of data at once.  (Like the SQL injection hack on BPIP)   If you have no need to edit your data once downloaded, you won't need to deal with that.  But a database is designed to index - both physically and logically.  No matter how fast you think your queries are - a database will make it faster. 

Learning how to use a database has been on my if-only-I-had-the-time list for a while too, but processing data with a script is quite easy.

If it helps: I can quite easily make daily/weekly/monthly-versions of only certain columns. If you don't need all data that should make it a lot smaller.
If you can use it, I can gzip the file too: blockdata.lastweek.txt.gz (this sample file is scheduled to be deleted in 7 days).

Here you go

You can find the description of each item here, in blockchair documentation:
https://blockchair.com/api/docs



I have skills and server limitations. I am not able to handle a huge amount of data. I believe 1 week data is already a lot (700kb file). One month data will be 2.8MB, which is huge for a webpage.

I will try to learn how to handle databases so I could process many gigabytes of data in the server, and just deliver the results to the client. But I don't have the skills (yet) neither the server for it .

For now, I will have to work with small amounts of data.

Thank you for the information (from your data extrapolation) and the given data pages. As you are familiar with it, would mind give me headers of columns that are separated by commas, please.

I don't know which column is for which variable/ parameter -- this info is necessary to run analyses, then I'll look at and play with it when I have time.

I don't need json file as I am not a coder.  

How about a little modification to show results within certain blocks or dates?

Stop flirting with my brain

Indeed, it is interesting and it makes perfect ^[ish] sense

I think we both know the moderator that clearly enjoys moderating that section in a strange way
^{- He/she is the reason why I haven't posted or even lurked on those boards in the past few years.}

Thanks. I had a lot of time to study JavaScript and python in this quarantine. I like to work on this. I think i will take a break now, before my next project (no plans yet)


i had a little chat with a friend who is statistician and he said barcharts are the way to go (just like the article I shared earlier)

Vertical / Horizontal depends on the amount of data you have.

Personally, I will just keep pie charts if I have less than 5 values to compare. Otherwise, bar charts.

I think horizontal ones are more beautiful. But I like vertical ones as well.


Edit:

I noticied some miners are really resistant to segwit

Take a look for example EMCD in daily data.
It mined 3 blocks. No empty blocks. How come they only have 37.2% segwit adoption while all others have 50%?
They are intentionally removing segwit transactions, you can see that when you compare their total reward: it is lower than the others, only 7.08BTC per block, while the average is 7.36.

I am thinking about sharing this kind of data in the mining board, but I am afraid of those moderators who just delete everything lol

I think it is important that people become aware that some mining pools are getting lower rewards just to favor legacy transactions.

Edit2:

This can also be seen here:


There are no miners in the bottom right side of the chart: Low Reward and High Adoption.
But you can see many miners in the bottom left side: Low Reward and low adoption.

All high reward miners are in the top right side: High Reward and High adoption.

if you want to work with the raw data:
https://loyce.club/bitmover/blockdata.lastweek.txt
https://loyce.club/blockdata/blockdata.lastday.txt

I have all that data in json format in the domain, which I prefer. I can help you access it if you need.

You are a very hard-working guy, bitmover.  

That's helpfulness and beauty of bar charts.

I made a big update!

Finally I was able to handle week data, thanks to LoyceV who created a new directory for me.

Now you can just click on the button "last week" or "last day" to get the data you want (you will need to wait a few moments, 700kb file needs to be loaded)




Following tranthidung suggestion, I made an horizontal bar charts for miners:



and my favorite one, the scatter plot of block reward x segwit adoption:


Here we can see that miners who put more segwit transacftions inside a block get more fee rewards. That's interesting.

I will try to find a way to get more data into it!

Some updates.

I guess the dashboard is mostly done now.

added 3 main sections.

First one is the block session, with rewards and  block size data (focus on empty blocks)



Second is about transactions. Segwit Adoption and fees.


And mining section, with total mined blocks per pool and segwit adoption per pool. I noticed some pools are resisting to adopt segwit.



Soon I will add weekly data.

---

I also added funded address section, with LoyceV database of funded addresses:

Thank you for your kind words


Thanks for nice words examples.
I am still learning and being as minimalist as possible.

Wow, @bitmover, this is very nice work from you.
I was just checking the server load from your account, you are very frugal 




maybe there’s not even a need for another type of font. Heading may be marked with biggest/smallest or bolded font if they need. I think differnet font type draws attention, more than the size.

You are talented!

Generally, you did a great work to improve that page, especially visual chart. It is good to know you are an open-minded person and this characteristic will help you growing more. Some people are too conservative and that characteristic restricts their learning, and working process.  

Would you mind explaining what is crowed? I don't understand it.
I can imagine the horizontal bar chart can contain too much space on your page and it can be a trouble for that page, generally. Is it what you meant?

---

I still think the font size for mining pool names should be a little bit smaller. Better visual and also better to catch attention of viewers on main and non-main points on your chart. Of course, it is personal perspective on which is main/ non-main points.

My heads up.

• Is it possible to include % (on top of bar) for top mining pools? Currently, the stats are in total blocks found by each mining pool.
• It will be messy if you include % for all pools and I only suggest for top mining pools every day. For non-top mining pools, skip the %.

I cannot add Stable blocks because they are not in the current blockchain (not the longest chain).
I added the information regarding empty blocks. It is now more visible and you can see a tooltip:





I also sorted the miners pie chart and create a new bar chart (vertical, as horizontal was too crowed).
I liked the visual. By mistake I made the background green and I liked. I will redesign the dashboard soon with new colors.

Tree charts (the chart from coin360 you are referring to) are not exactly loved by statisticians
Take a look at this comparison:it is worse than bars in every single case.

https://www.theinformationlab.co.uk/2014/12/16/treemap-vs-bar-chart-end-treemap/

Treempas Are basically a fancy pie.

I think bar charts, as suggested by tranthidung are the way to go. If I had only 5-10 pools pie would be better. But bars are really more effective  for nearly 20 pools.

I will try to chart are chose better colors for the pie.. if I cant I will move to bars .

You are free to make design suggestions,  but this is just an alpha version of the dashboard.

With regards to how Mining Pool data should be shown, I do suggest you can try to improvise how Coin360.com displays the crypto markets price using boxes which size depends on how huge a single element are compared to the others. I would be pleased to try and design/redesign the website if you would let me to


Actually, you can. There is a framework called Django (most famous for web dev), which lets you create a backend of a website. With regards to syntax, Python is much simpler and easier to understand. There are tons of javascript syntax that is complicated in ways that a Python can provide in a single line of code. Have you also tried node.js? It is a better version of JavaScript

Python is amazing, but it cannot be used for web development.  I created this website with javascript

Python has similar syntax, so it kind of helped me (I learned python first)

It is only example to explain my suggestion to customize your chart's color bands. I don't know Python (I will begin my Python learning in 2021) but a quick search points me to these guides

• https://stackoverflow.com/questions/59320564/how-to-access-and-change-color-channels-using-pil
• https://pillow.readthedocs.io/en/stable/handbook/tutorial.html

It is similar to what I code for my chart color when I want to customize it. Default colors are always terrible, you know.  
Link

SFR10 and tranthidung, you both are right and made nice suggestions

I can add guessed miner to the line chart tooltip.  I can also sort the piechart by size( it is now sorted by who mined first in the time period).

The colors from the website you suggested are not match the dark theme colors. I need to take a look more carefully at that.

I will do all suggestions when I get home next week , I am traveling now away from my dev environment.  This wasa rush job, but I really wanted to release it . Maybe it was a bad decision. But the community really give nice inputs.

I need to make a better table for miners as well. I will add many stats there for each miner, like total reward earned in the period in usd and btc, hashpower %  , and so on.

Empty blocks also need more attention, as well as zero segwit blocks.

@SFR10,

• Your post reminds me that the current pie chart is arbitrarily displayed. I don't know how mining pools are sorted out in that chart as I can not find any orders by alphabetical or percentage orders.
• If @bitmover wants to stick with the piechart, I think it's better to use a simple color theme (in the order of transparency (light to dark or cold to hot and vice versa).
• The color theme from jochen-hoenicke.de can be considered. It is better than mempool.space's color theme (that somewhat is similar to the current color in the pie chart, mix arbitraritly).
• I explained my idea there

Another interesting way of showing those stats ^{[you never fail to amaze me]}, keep them coming

• Suggestions/questions:
  
  • Would it be possible to include also the pool names that mined each block when hovering over the "Block Size (bytes)" chart?
  • tranthidung made a valid point but perhaps you could stick with the pie chart and just remove the names ^{[it'll be cleaner]} and only show them when someone hovers over them ^{[e.g. like your Segwit adoption pie chart]}.
    
    • You might want to also reconsider the color combination for "some" of its parts and the white-colored font ^{[it'sa bit hard on the eyes]}.