I think the fee should be a function of the number of unconfirmed tx at the time of sending. Would this work? Clients need to be more intelligent.
I don't think this is necessary. The number of transactions per hour that the Bitcoin network will process is not going to vary widely on a hour to hour basis, it will also not change very much on a day to day basis, and probably not even on a week to week basis. The reason why the Bitcoin network will process
roughly the same number of transactions per hour throughout the day is because of it's global nature - a peak spending period in one part of the world is going to be a period when most people are asleep (and otherwise spending very few transactions) in another part of the world.
Economies of scale are not going to change
very much on a short term basis. People are not going to wake up one day en masse and decide to start using bitcoin for everything they do. People will start to learn about the benefits of using Bitcoin, will use it for a very small number of purchases, tell their friend about it, and over time use it for a greater percentage of their overall spending. The time from when someone does not own any bitcoin to when they spend it whenever they can will be, on average several months, if not longer.
Sure the Bitcoin network will have spikes in transactions for a variety of reasons, although such spikes are, IMO, unlikely to cause great delays in having transactions confirmed, especially delays of more then 6 or 7 blocks.
I would argue that a better solution would be to have a client's default fee policy to be changed (if necessary) each time a new version is released.
But, Quickseller, your argument is clearly incorrect given the recent stress test. It was very clear that the number of backlogged transactions increased drammatically during a short period of time and if clients took the backlog into account when suggesting a fee, how could that be anything but an improvement? All clients should allow their users to set their own fee at the end of the day, but suggesting a fee based on more information (rather than less) when that information is freely available to any bitcoin node is clearly a good idea.
Sure it is possible for the mempool (aka the backlog of transactions) to grow dramatically in a short period of time. However the latest instance of this happening was not the result of anyone making a rational economic decision. The backlog of transactions was artificial and over the long run it is unlikely that these kind of blimps will show up, nor will be sustained over more then several blocks.
You can call it "artificial" or "irrational" but its occurrance was an empirical fact. To suggest that something which just, in fact, happened, doesn't happen is like covering your eyes and saying 'see no evil'.
The devs of the various clients have limited resources and I don't think this is an overall good think to invest limited resources into. If you want to invest your own money/resources into creating such a client, then you are free to do so.
Of course I am free, and I thank you for reminding me of it. I guess. Anyway, what we're talking about here is essentially a one-liner in code. And yes, I may indeed fork the wallets I use in order to implement it. I can't see any rational reason not to.
Even if you were to ignore my conclusion that the number of transactions per hour does not vary widely over short periods of time, it would really not be a viable feature to have your client estimate the required fee to get a transaction confirmed quickly for a number of reasons.
1 - The various pools, and miners (when solo mining) ultimately create their own policy as to which transactions they confirm when they find a block. This policy could be to include no transactions, only transactions that have been propagated and meet certain criteria, transactions originated by themselves that have not been previously propagated, transactions provided to them by entities that have pre-existing arrangements with them that may or may not have been previously propagated, among a wide range of other potential criteria.
I would not at all be surprised if a number of entities that push a large number of transactions to the network have agreements with various pools that guarantee their transactions be confirmed in blocks found by those pools if they are not already confirmed.
The policy of each pool has the potential to be, and likely is different from other pools.
Each policy can be different but they all follow the rule of preferring transactions with greater fees. You seem to be forgetting the fact that you don't need to offer any certainty here. It's quite simple and clear that when the mempool is backed up and you offer a greater fee then you are gaining likelihood of a faster confirmation. No one is saying you have to estimate to the minute when it would be confirmed.
2 - Blocks are found, on average, once every 10 minutes, but are often found less frequently. It would only take seconds for the mempool (and thus the likely required fee to have your transaction quickly confirmed) to have grown in size dramatically (someone could potentially push several thousand valid transactions to the network all at once). This means that someone could push a transaction with a fee that their client claims will cause it to be quickly confirmed at time n, then at time n+1, someone pushes 10,000 transactions to the network, many with "better" fees, and before the next block is confirmed after time n. This would create a false sense of a guarantee to the end user.
No one should be offering a guarantee, as that would indeed be misleading. Your argument here seems to fall into the fallacy of the perfect being the enemy of the good. If I send a transaction at one moment there's clearly nothing I can do about any number of transactions which are sent after me with greater fees. But if I decide to ignore facts about transactions which have been sent before mine, then that's information which is available to me which I am choosing to ignore. And your arguments that it's somehow better to ignore this information simply aren't strong ones.
3 - Over the long run, the majority of end users are not going to run clients that are full nodes to spend their bitcoin. This means that they will need to place a certain level of trust in other full nodes when deciding what level of fees to include. This is trust that probably shouldn't be given. Today, if full nodes give incomplete information about unconfirmed transactions or about the blockchain, then any node that it connected it to will get that information from someone else. If that full node gives information about invalid transactions, then such invalid transactions will be ignored by everyone else.
Here you say end users are going to trust nodes but they probably shouldn't be doing that? I just don't follow or I'm missing how this is relevant. I apologize. No node is going to have complete information. Especially if we're talking about the future. But anyone with access to a network connection can use information available at the moment in which they send their transaction. The utility of this was demonstrated to me quite empirically when I sent a transaction last monday and waited 11 hours for confirmation. If I had merely recevied some warning about the mempool status I would have sent with a higher fee or waited to send the next day. Not having that information presented to me was clearly not as nice as if I had been presented that information. Knowledge is power and wallets that present information about the current state of affairs to their users are clearly superior to ones that don't.
As it is today, the mempool of every full node is potentially different, and often is going to be different from many other full nodes. The node that your client gets the size of it's mempool from may not be an accurate reflection of the network. It is also possible that a pool could have a large number of nodes that artificially increase the reported size of their mempool in order to encourage people to include larger tx fees then is necessary in an effort to increase the overall tx fees they receive. Requiring a node to prove the size of their mempool would open up a whole new can of worms that would simply not be worth it
Again, no one is suggesting that you need to prove the unprovable. You seem to be going off a deep end here. The idea, as I understood it, was merely to present better information to the user at the moment when a transaction is being sent. Despite all your typing, I still can't see that this would be a bad thing.