Topic: Reliability of sites such as mempool.space for estimating fees. (Read 376 times)

I agree with you but let me explain why I prefer Jochen-hoenicke.de site again.

Even the red bands in Jochen-hoenicke.de don't always mean overpaying but also can means expensive, sometimes they are fee rates you have to pay to get a confirmation quickly (next few blocks), in my view, it is still a red flag (warning) for people. If you are not in emergency to move coins, it's good to sit back, think more about red-banded fee rates and you can wait a few days, weeks to move your coins. With this view, that site and its color design are very helpful.

My guess only but likely the person behind that site, actually has clear idea about what colors to use and for what purposes. It's different than mempool.space and how they use colors.

Thank you, I don't have skills to do that but your post is damn helpful, I like it.  

More specifically, you can read how the suggestions are calculated here: https://github.com/mempool/mempool/blob/827b0f6ad1c419e79d1a5cfa55b74b9c993a4748/frontend/src/app/docs/api-docs/api-docs.component.html#L199

Snipped and reformatted for easier reading:


As I said earlier in the thread, I think those estimates are bit on the high side. At the moment, the "No Priority" fee of 16 sats/vbyte still puts the transaction less than 1.5 MvB from the tip, which is not what I would call "No Priority".

According to the image you posted, mempool.space is saying that if no new transaction is made, the next block will include transactions using the fee rate of at least 14 sat/vbyte and the fifth block will include tranactions paying 11-12 sat/vbyte.
Mempool.space isn't saying that if you use the fee rate of 12 sat/vbyte, your transaction will be included in the fifth block.

If you want to use mempool.space for fee estimation, you should use the fee rates displayed under "High Priority", "Medium Priortiy" and "Low Priortiy"

Mempool.space suggests using the high priority fee rate for placing your transaction in the first mempool block, the medium priority fee rate for placing your transaction between first and second mempool blocks and the low priority fee rate for placing your transaction between second and third mempool blocks.


See the image I just took from mempool.space.



According to the image, Mempool.space is saying that if no new transaction is made, the next block will include transactions paying the fee rate of at least 15 sat/vbyte and the second block will include transactions paying 14-15 sat/vbyte.
Mempool.space is suggesting the fee rate of 18 sat/vbyte for placing your transaction in the next block and the fee rate of 17 sat/vbyte for placing your transaction in the next two blocks.

I would extrapolate the transactions made in the past few hours, and use that as an estimate of transactions in the coming hours. It's not always accurate and there can be exceptions, but most of the time it's pretty accurate.
That's why I like Johoe's site: it quickly shows fees are currently slowly going up, which means if you pay the minimum and it doesn't get confirmed quickly, it's likely to take a lot longer. If it would have been the opposite and fees are slowly dropping, paying the minimum is likely to get your transaction confirmed quickly.
I can't get this information from mempool.space's estimate.

To be fair, there isn't a way to figure out how many new transactions will be produced in advance.

Maybe a machine learning model could be made, that looks as previous mempool activity over the past 2 or 3 years and then tries to estimate how many new blocks will be created at any given time, and then estimate the median fee for all of these transactions, and then "push" the stated transaction fee right now closer towards this median. But that would be resource intensive.

I never really looked at it before until now:

It looks like their estimate in confirmation time is based on the assumption that no new transactions are being made. So, if you pay 12 sat/vbyte now, your transaction will be in the fifth block. That sounds acceptable to me, except for the fact that there's about 1 block worth of new transactions every 10 minutes, so it can take days to clear just those 5 blocks.

You can reverse the colors used on the chart because it's implemented as a gradient, and then run your own frontend - since this stuff is defined inside Angular source which is then compiled down into not-so-easy-to-edit JS.

First go to this file: https://github.com/mempool/mempool/blob/master/frontend/src/app/components/svg-images/svg-images.component.html

Clone the repository, and change all the instances of this:


to this:


Then follow the instructions to self-host Mempool.space on your own hardware.

That's not what the colors on Jochen's site mean. The fee bands are simply different colors. There is no color that means "overpaying". Go back to May this year or go back to the end of 2017 and there were plenty of times when you had to pay fees in the red bands to have a reasonable chance of being confirmed within the next few hours. This was not an overpayment, but rather completely appropriate.

Nope.

I did neither mean about accuracy of mempool.space fee recommendations nor the color in your screenshot. I meant that band colors in their chart and about red flag, overpaying, it is exactly what I meant.

Like on Jochen-hoenicke.de, when fee rate band color hits hot bands like red, it means overpaying. For lower fee rates, even you are overpaying, it is still fine because not too much expensive.

Compares it to mempool.space, I hope you see difference in their color design. Let me emphasize, I only dislike colors used in their chart.

Mempool.space is open source and you can run it yourself as self-hosted, so you don't have to depend on any third party.
If you don't want to do that than you can check mempool state from several sources and get median mid value.
You can check my BTC Mempool Fees Extension/add-on for Firefox/Chrome browser that includes mempool.space with two more source (blockchain.com and bitcoinber.live).
Default settings in mempool.space but you can do manual change quickly.

PS
Note that nothing is 100% precise since all of this is just estimated fees.

If the change is smaller than the dust limit, that's the only option. The difference could even be more than the dust limit, as adding another output would increase the fee further.

I've made mistakes with fees, especially when I try to rush things instead of taking my time to review the transaction. It happens.

I don't think I emphasised it well before. Not just the fee. I know that in every block there are people who pay high fees well above what would be reasonable to include the transaction in the next block. I mean the fee in relation to the amount you are sending. If someone sends 2 bitcoins I am not surprised that he overpays the fees. But in the transaction I have posted, there is one input and two outputs of about 0.001 bitcoin and he is paying a third in fees. I don't know how someone who is sending 25 euros can be in such a hurry to pay 8 to get into the next block. That's not what bitcoin is about as far as I know. It's not done for privacy, or to avoid an unspendable change or anything else. That's why I say I think he's someone who I think has no idea what he's doing.


I did not say that.


It has happened to me sometimes that it is the wallet itself that rounds up the fees.

Sometimes mempool.space was recommending me to send transaction with 6 sat/vByte and within minutes fee was increased up to 26 sat/vByte, that happens and mempool.space has nothing to do with this.
By the way, you can manually check some recent blocks, upcoming ones too and decide yourself what fee will be enough to include your transaction in next few blocks. You have to keep in mind that sometimes miners manually add transactions in their blocks and you shouldn't include added transactions when you calculate fees.

With the privacy implication of linking whatever payment you were making to your favorite VPN or account at some website. Sometimes it's simply not worth it for the sake of 1000 sats.

Alternatively, send it to the public donation address for something like Tor or EFF.

I take it as a challenge to waste as few sats as possible. This small output for instance could have added a few days to my favourite VPN, or topped up the balance at some website.

Look at any block and you will see dozens of such transactions.

Here's the most recent block, for example: https://mempool.space/block/000000000000000000001d63a0516748160bb6b01741ab14b41277416fd5f54d

mempool.space lists transactions in a block in the order the miner includes them. Since miners preferentially include higher fee paying transactions, then usually all the highest fee transactions are listed first. I count 19 transactions which pay a fee of 100 sats/vbyte or more, when actually 8 sats/vbyte was all which was needed to get in to that block. Click on any other block and you will see the same thing.

You can't say that every single one of these transactions is simply because someone doesn't know what they are doing. Some will be, sure, but there will also be people who absolutely must get their transaction in to the next block for whatever reason, centralized exchanges with their poorly optimized algorithms, people who simply don't care about the extra few bucks they are spending in fees, some poorly configured wallets or software, and so on. I for one have created transactions with excess fees when paying a regular fee would leave me with a change output of a few hundred or a few thousand sats. Rather than be left with an output which would cost 50% of itself in fees just to spend, I'll just add it to the fee I'm already paying and donate it to the miners.

Right. But it's not really difficult to use the website.

I just took a screenshot of jochen-hoenicke website.



Acording to the image:
There are around 143 vMB of transactions paying at least 1 sat/vbyte.
There are around 125 vMB transactions paying at least 2 sat/vbyte.
..
..
..
There are around 22 vMB of transaction paying at least 6 sat/vbyte.
There are only 0.37 VMB of transaction paying at least 7 sat/vbyte.


Considering that each block can contain up to 1 vMB of transactions, if I want my transaction to be confirmed fast, I would set the fee rate to at least 7 sat/vbyte.
If I set the fee rate to less than 6 sat/vbyte, my transaction would be at least 22 vMB from the tip and considering that more and more transactions will enter the mempool, it will take a very long time until my transaction is confirmed.

The image clearly shows that how increasing the fee rate from 6 sat/vbyte to 7 sat/vbyte can increase the chance of getting confirmation.

I was just looking and I see that in the last block, transactions with a little under 7 sats/vB have been mined, but one transaction has caught my attention, because I understand that there are people who pay very high fees if they send a large amount of money, but don't you think the fees paid by this transaction for the amount they send is very unreasonable? I understand that this is someone who does not know very well what he is doing.

The fees required for transactions are always changing, if you use a slightly lower fee and your transactions still get  confirmed, It is possible that within a very short time while you were not monitoring the fees, that the fee dropped to that amount of fee you used which made your transaction to be confirmed before it went back up. I personally don't like to wait so long for transactions to be completed, so I always use a medium or high priority fees so I have the assurance of a quick transaction.

For newbies: https://mempool.space/
For advanced users: https://jochen-hoenicke.de/queue/#BTC,24h,weight

I always prefer to recommend mempool.space for news. Advanced users that understand very well how to use jochen-hoenicke.de will prefer jochen-hoenicke.de.

It is still one of the best mempool site that newbies can use, although jochen-hoenicke.de can be more accurate. I have used mempool.space times without number with good fee rate estimation.

There are currently over 440,000 transactions with the least priority (which pay 1-6 sat/vb), so don't enter a fee rate of that range, as they're probably there for days. Click on the most left block from mempool.space to see yourself what fits your need best.

For example, paying something like 7.1 sat/vb can get you confirmed in a couple of hours in the worst case.

What? First time hearing a bad experience with mempool.space. They're like the most favorite block explorer. Tells you exactly what you need to know, and the styling rocks. What hot and cold colors? Why should high fees be red flag if there aren't overpaying?

The issue at the moment, which has been the case for a few weeks, is that there is a huge backlog of transactions paying 6 sats/vbyte waiting to be confirmed, with the volume above that spread very thinly. So it is entirely possible that 8 sats/vbyte would put you within 0.1 MvB of the tip of the mempool, 7 sats/vbyte would put you 1-2 MvB from the tip, while 6 sats/vbyte puts you anywhere from 2-30 MvB from the tip. So you pick low priority at 6 sats/vbyte, and then a few blocks are found in quick succession and your transaction is mined. That's not a failure of mempool.space, but rather a quirk of state of the current mempool.

Bear in mind of course that transactions don't pay either 6 or 7 sats/vbyte, and there is a continuous range between these two numbers which most fee estimators will not pay attention to. 6.0 sats/vbyte and 6.9 sats/vbyte will be a difference of over 25 MvB from the tip at the moment.

Having said that, mempool.space is currently stating the "No Priority" fee is 14 sats/vbyte, which is only 2.5 MvB from the tip. That's not "No Priority" in my book. If I had a "No Priority" transaction, I'd set it to around 6.8 sats/vbyte and expect it to be confirmed sometime in the next 24 hours.

So yeah, as others have said, take a look at Jochen Hoenicke's site instead. Not only can you place your transaction exactly as far from the tip as you want, but more importantly you can view the current mempool trends and see if it is filling up or emptying out.

Yes, I knew that. The point is that I had previously been guided by mempool.space and found it to be approximately reliable. It has been the last few times that I have had the impression that it was giving estimates that were too high.


I guess with the time I've been using bitcoin, it's about time I used something more sophisticated, like this website. It's not the first time I've visited it, I guess I've seen it mentioned on the forum, bud you have to do the calculations yourself instead of being guided by the simplest thing, which is that a website gives you the estimated fees for a quick confirmation.

I understand that I can take this as a general rule? 1vMB from the top for a confirmation of a few hours and a few more sats for faster confirmations.


Yes, that's for sure.

I prefer to use Jochen-hoenicke.de website when I need to check mempools too.

Two reasons why I dislike mempool.space website.

• It is very bad designed, bad organized in fee rate bands' colors. Their color bands are mixing from hot to cold colors that is annoying already. Moreover, they use hot colors at bottom and cold at top. We know, most expensive fee rates are considered as 'dead' ones. They should use hot colors like red flags. I don't get ideas why mempool.space team designed like that or perhaps they see low fee rates as red flags.
• In the past, their chart is static, not dynamic. Static means when I hover a mouse on their chart, fee rates won't be updated. It is not dynamic like Jochen-hoenicke.de website. Recent months (around 2021 or 2022), they updated their chart to be a dynamic one but I still dislike it because of color bands.
  
  • Static chart: https://web.archive.org/web/20210811200553/https://mempool.space/graphs#24h
  • Dynamic chart: https://web.archive.org/web/20220823122230/https://mempool.space/graphs/mempool

Well, "estimates" are just estimation based from statistics and shouldn't be relied on with 100% accuracy.
The longer your projection is (like 48hours), the harder it is to make a close estimate since nodes' mempools states in the future cannot be predicted.

As for mempool.space, it seems like they have their own fee rate estimation using their own API  (e.g.: mempool.space/api/v1/fees/recommended)
Based from the result of the above, it's not similar to Bitcoin Core's estimatesmartfee or estimaterawfee.
For accuracy, there's no way to tell or rather, fee estimation algorithms aren't really meant to be accurate, those are just trying to suggest the "best" fee rate for your target block.

Here's the source code of their fees API, BTW: https://github.com/mempool/mempool/blob/master/backend/src/api/fee-api.ts

I prefer to use Johoe's Bitcoin Mempool Site Statistics, which quickly shows how fees developed in the past 24 hours. It shows there was a bump in fees yesterday, and if you would have made a 6.x sat/vbyte transaction at that point, it would have taken 10 hours to confirm. You couldn't predict that bump in fees before it happened.

Correct. I like to put it this way: the recommended fee is an estimate based on past transactions and blocks, while your new transaction depends on other transactions made and blocks found in the future. It's an estimate, that's it.

For having a good estimation, the best thing you can do is to check and analyze the mempool by yourself. To do so, you can use jochen-hoenicke website.
At the time I am writing this post, mempool.space is suggesting the fee rate of 11 sat/vbyte for high priority and 9 sat/vbyte for low priority. If you check jochen-hoenicke website, you will see that you can put your transaction at 1 vMB from the tip with only 7 sat/vbyte.

If I want to make a transaction now and I want it to be confirmed very fast, I would set the fee rate to around 10 sat/vbyte.
If I can wait for a few hours, 7 sat/vbyte is more than enough.

Mempool congestion is not constant. It can drop and it can increase. If the mempool congestion drops, the fee rate and the fee required to get high priority confirmation will be reduced. If the mempool is becoming more congested, the fee rate and the fee required to get high priority confirmation will be increasing.

It depends on how mempool is. But right now, the purging is at 5.96 sat/vbute (not constant either). Any transaction with fee rate that is less than 5.96 sat/vbyte will not be successfully broadcasted right now.

I had usually relied on mempool.space to guide me on what fee to use to send my transactions but lately it seems to give me too high a fee. I remember a couple of times recently when I wasn't in a hurry to get a transaction confirmed and if the site gave high priority 8 sat/vB, medium priority 7 sat/vB and low priority 6 sat/vB, I sent it with 6 sat/vB and it was confirmed in a few blocks. I didn't check exactly but I would say it was confirmed in three or four blocks at most, in less than an hour for sure. Then, the last time I sent a transaction I still sent it with a lower fee, if it gave me low priority 10 sat/vB and 9 sat/vB as no priority, I sent it with 9 and it also confirmed relatively quickly.

I don't know if you have had this impression recently with mempool.space or the sites you use.

The thing is that I could send it with an extremely low fee and then raise it with RBF but if I am not in a hurry to get a transaction confirmed but I would like it to be confirmed within 48 hours maximum, if mempool.space marks 9 sat/vB as no priority it seems silly to send the transaction at 2, for example, because I will have to end up bumping the fee for sure and I don't think it's worth the hassle to save a few satoshis.