Topic: sendtoaddress sendmany api change proposal. (Read 1555 times)

Well, do something different than all the previous "enterprise" programmers. There was a common thread in all the "enterprise" patches submitted to the core team: they interfered with or completely destroyed the blockchain reorganization behavior. I'm on the record disagreeing with various technical issues regarding Bitcoin Core client, but I fully support the Bitcoin Core developers in their insistence on preserving the fundamental properties of the Bitcoin protocol: the global consensus is achieved through a somewhat random convergence that could involve orphaning some blocks.

The list of things you seem to be missing is actually longer:

1) The coin-selection is a non-trivial problem that is NP-complete in a general case (I've already mentioned that the current solution is a stochastic approximation, which is non-monotonic and therefore breaks the code like yours)

2) Bitcoin wallet isn't static while your code snippets are running, new blocks can appear randomly and change the coin selection both by adding and removing coins from it. (This is the most common mistake in the various "enterprise" patches that I've seen: mishandling of orphan blocks)

3) After you correctly solve the first two problems there is an issue of repeated greedy local optimization leaving wallet full of near-dust coins, so the optimization needs to be upgraded to some non-local, near-global or at least allow for optimizing coin selection for a set of transactions.

4) After you correctly solve problem (3) you need to provide a solution for the common operational requirement of keeping 2 wallets: hot and cold, and manage the coin transfers between the two of them that fulfill the non-scalar optimization goal of maximizing security and minimizing fees.

I hope that the above list helps you understand the true "enterprise" requirements and why your proposal is nothing more than a short-sighted temporary hack.

From myself I'll add that any true "enterprise" solution would obey http://en.wikipedia.org/wiki/Two-phase_commit_protocol and will easily integrate with any of the available http://en.wikipedia.org/wiki/Transaction_processing_system . From your insistence on using unstructured RPC calls I surmise that you have neither "enterprise" coding experience nor even plain multi-threaded development.
The technical reason is called "inversion of control" (i.e. server calling client to make a decision). Again this was discussed on this forum many times, at least since the wx->Qt UI transition. Use the search function.

If it is really such a "sharp pain point" then you are better off basing your work on a different code that Bitcoin Core and submit raw transactions to be reference implementation.

oh and also, if you look at this pseudo code again:



Notice that there is a new fee parameter to sendtoaddress(), which in this case is the value returned by calcsendfee().

That fee would act as a limit for sendtoaddress.    So if the fee that sendtoaddress generates pseudo-randomly is higher than the limit, then it would not send and would return an "insufficient fee" code.   If it is less or equal, then it would send.

So I believe that handles your objection.

You seem to be missing the point that this is a sharp pain point for developers and businesses.  And just plain bizarre that one can't know a fee until after the fact.   too bad if the fee turns out to be 20%, 50%, 100% or even more of your send amount!

Also, if you look at my original proposal in this thread, you will see I am well aware that the fee could change between API calls.  That's why I proposed an alternative with a fee_limit param for sendtoaddress.  That would work fine with the existing pseudo-random sendtoaddress implementation.

Even still, I can wish for something better.  As I said, "perfection to me would be...".

Bitcoin-qt allows the user to view the fee and either send or not.  So what technical reason prevents that capability from being exposed at the RPC API level?   Do we need to add some locking/state around calcsendfee/sendtoaddress?  fine, let's do that.

Or there could be a deterministic alternative to sendtoaddress that RPC users could use when they want a fee calculated.

I believe this is not a technical problem so much as a "la la la" we don't want to fix it cultural problem.

My concern is that if I (or anyone) go to the work to fix it and submit a patch, it won't get accepted, due to core team resistance.

Am I wrong?

My hope with this thread is that we can get some amount of concensus on an approach, so that when it gets implemented, it will be likely to be accepted.

You seem to be missing the crucial fact: the coin selector in the "send" is a stochastic knapsack solver. For any non-trivial wallet it pseudo-randomly selects different coins resulting in a different size of the transaction and thus requiring a different minimal fee. This has been discussed many times on this forum, please use search.

I'm not entirely sure what good worrying about the average of fees in total is when I have zero control over it using send* apis.  In fact I am here making a proposal because I do worry about it.

The case of small amounts may actually be where this is most useful because the fees can presently eat up 50% or even 100+% of the send amount.  But its hard to KNOW that until you actually try to send.  And then you get the surprise fee amount like an unwanted toy out of a cracker jack box.   So it is often preferable to wait until those inputs have aged sufficiently and/or enough other larger inputs have come in that the overall fee percentage goes down to something like 1%. Or maybe just wait till btc price rises and those dust are worth something.  whatever.... it should be my application (or its user's) choice, not some algo within bitcoind.

In the case you mention where many dust inputs are received and it jams things up, it is still useful for the app to be able to make informed decisions.  Maybe the decision is to call lock_unspent for some utxo, or log a warning or email the site administrator.  Or maybe it is just to send again with a higher fee limit or no fee limit at all.

Right now the app developer's options are to either blindly call sendto* and take the fee it arbitrarily comes up with or else write your own coin selection and fee calculation routines and generate a raw transaction.  I've been down that route....  it is painful and error-prone, and only a few people will get it right.


I'm not saying the proposed API change is perfect... actually perfection to me would be if I could write app code something like:


There would be a corresponding calcsendmanyfee also.  sendtoaddress might (rarely) return insufficient_fee error code if conditions have changed since calcsendfee() returned.  In that case, the application should retry, possibly multiple times.

I'm guessing this approach is a bit more involved to implement than my first proposal, but maybe not too bad since it is already calc'ing the fee internally anyway.

thoughts?

I don't know the full answer, but I can give you a handful of pointers.

• The mailing list is a good place to discuss Bitcoin Core implementation specifics, especially non-trivial changes. (It's not a good place for arbitrary questions / thoughts about Bitcoin Core....)
• The coding guide is an essential read.
• There's also a lot of discussion that goes on in the GitHub repo, especially in pull requests -- if you're thinking of implementing something, I'd check here to see if someone might already be working on it, or if the idea has already been tried but had some issues.

Ah, ok.

Removed my comment.

Thanks for explaining.

I've been considering getting involved and making contributions in the development of Bitcoin Core.  I'm still figuring out that process (I've never been involved in programming or designing for any open source projects before).

Seems like some sort of mentoring system to make the transition easier would be useful for talented programmers that want to get involved in the effort but aren't sure where or how to get started.  Are there any good tutorials or primers that I should take a look at?

Do I just install git on my MacBook, clone the repository, throw a dart at a BIP list, and start from scratch on my own implementation of whatever proposal my dart lands on?

We don't generally use BIPs for random UI things.

As far as the idea goes— perhaps. Though I'd recommend adjusting your business plans such that the average of the fees in total is what you worry about; rather than the fees on any particular transaction. Otherwise you're just going to get jammed up and unable to transact when someone has sent you a number of dust inputs and the only way to eliminate them is to pay a somewhat high fee... all your low value transactions will end up failing until you up that limit.

Having worked on a few sites using bitcoind, a recurring pain point for me (and apparently many others) has been the inability to know the fee prior to sending.  This is particulary troublesome because the fee can vary wildly as a percentage of the send total, and business rules may preclude sending when a fee is higher than a given percentage.

It's sort of like if you walk into Western Union or Money Gram, and ask how much it will cost to send money.  They say, don't worry about it.... it's kinda complicated to figure out so just leave us your credit card and you can look at your statement later to find out how much we charged you.  They don't do that because people would not accept it.  We like to know up front, or at least be able to set limits.

Often when sending bitcoin it may be desirable to wait until the fee is less -- either because the coins have aged, or we have restructured our inputs, or we are sending a larger total resulting in a smaller percentage fee.

In other words, the fee itself can be an important factor in decision making whether to send a transaction.

I understand it may be difficult or problematic to implement a getsendfee() type of API because the fee could change, etc.

However I know that for my own use cases, it would be very helpful to be able to set a fee limit.  So that sendtoaddress or sendmany would succeed if the fee is below the limit and fail if over.

So the APIs could be:


where max-fee can be either:
a) a fixed amount, eg: 0.001
b) a percentage amount, eg: 1%

If the transaction would exceed max-fee then an error is thrown.  Applications can catch that error and optionally try again with a different combination of amount and max-fee.  At least the app knows the fee was insufficient and can act accordingly.

Use cases for this functionality

1. A merchant or other bitcoin receiving website maintains a hot wallet and regularly sweeps to cold-storage.  

   Owner wishes to wait until either:
        a) enough funds have accumulated and coins aged sufficiently to send with 0 fee.
        b) funds total passes a given threshold, and will then accept up to a 1% fee.
        c) funds total passes a higher threshold and will then accept up to a 3% fee.
        etc.

2. A mining pool or other agent needs to send to many recipients.  

   Agent wishes to wait until enough funds have accumulated and coins aged sufficiently to send with 0 fee.
   Agent will send to individuals sooner if requested, but will deduct max_fee amount from their payment to cover the send fee.


But what about floating fees and txconfirmtarget?

At first I was hoping that floating fees would remove this pain point, but it doesn't really seem like it does.  According to Gavin:


Now that sounds pretty fuzzy to me.  I don't see how I can use that capability to enforce a business rule that we don't send unless fees are below X percentage.  If it is possible, please enlighten me.



So... what do you think?    good idea or bad?    and how easy or hard to implement correctly?


I'd be glad to a crack at this and submit a patch, but only if there is some amount of consensus that it is desirable functionality and reasonably straight-forward to implement.