Topic: JSON-RPC method idea: list transactions newer than a given txid (Read 6778 times)

Salute to you

What is the new state of affairs regarding this issue, after 3 years ?

Is there a way to poll or get notified when things happen on the network ?

thanks

Yes, listtransactions "*" is possible.  The other account routines could return a new "invalid account name" error if given "*".

I've got two issues with it, though:

1.  listtransactions "*" will have to iterate over every transaction in the wallet (transactions are not indexed by time), which will be slow for large wallets and will get slower over time.  And indexing transactions just so that listtransactions * is fast violates the "an optional feature shouldn't cost anything if it is not used" principle.

2. What is the use case for "list the last N transactions across all accounts" ?   The only one I can come up with is developing an alternate GUI that communicates with bitcoind via the JSON-RPC, but to support that at least a couple of other features would have to be added at the same time (e.g. listtransactions would needs to add account and bitcoin address information to the objects it returns....)

That would be great!


What about the empty string? It's easy to use in a programmatic context (e.g. in Python), and also in a shell context:

Moreover it would be consistent with the current API, since the empty string already means "any account", for example when used with "sendfrom" (and probably in other situations). Let's not use several special cases that are special in subtle, different ways. The simpler, the better.

As long as the interface is designed for things like showing the user the last N transactions history, it's fine, now that we have the Accounts feature making it easier to do payment detection the right way.

Gavin, could listtransactions have an option to list transactions for all accounts?

I'm not sure what the interface could be, maybe:
listtransactions [count]

It would be hard to do that from the command line though.

I can't think of a good solution for the interface, that's the problem.  Maybe "*" special case like "" is.  Everyone would have to make sure no user can create account name "*".

Quote from: jgarzik on December 09, 2010, 12:13:50 PM

Sure, and that's easy enough to track with transactions.

I don't get how that's "easy" to track with transactions.

Sure, and that's easy enough to track with transactions.

That's not always the case. Sometimes the customer will have a balance at the website. The operator of the website will certainly want to know if a past transactions becomes invalid and disappears.

I agree with you and satoshi about "txs after ".  My listtransactions (now xlisttransactions) patch pointedly does not have that feature, and never has.

I'll and add another reason not to have a "list transactions that happened after " :

move "transactions" don't have a transaction id, but they do affect account balances (and are listed in listtransactions).

Your code is going to get really messy if you expect to call listtransactions and then squirrel away the txid of the last item returned.  If it was "category":"move",  there WILL be no txid...

RE: eliminating polling:  at some point fairly soon, I plan on cleaning up my "monitorreceived" patch, to POST to a URL when transactions come in or blocks are accepted... but I need to do some Deep Thinking to redesign based on lessons learned from 'accounts'.  It might turn into a very minimal API, where the notification is "Hey, txid <123ae4221...> just got to N confirmations, you might want to call gettransaction and getbalance to get up-to-date."

I'm not talking about the normal risk for a given minconf level, I'm talking about additional pitfalls from listtransactions when used this way.

The OP's example of listtransactions [count=10] [txid] seems to imply and it would be very easy for programmers to assume that if they pass in the last txid of the previous call to listtransactions, they will never see the same transaction more than once, which is not the case.  It would be very easy to double-count payments if you don't maintain your own persistent map or dictionary to track which txid's you've already accepted.

It doesn't seem right to have a function that seems tailor made to be used a certain obvious way, and that way is a non-obvious trap.

Quote from: jgarzik on December 08, 2010, 07:07:22 PM

Quote from: satoshi on December 08, 2010, 06:36:45 PM

3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.

Suppose both spends are to the same address.  getreceivedbyaddress would always count only one or the other spend at any given time, never both.

Using listtransactions, it would be very easy to count both.  You see the first spend, you count it.  You see the second spend, you count it.  Total is double counted.

So it would be a design problem in the developper's app, not a problem inherent to listtransactions. What you're saying is: if listtransactions is used without care, it can be dangerous. OK, but many programmatic tools have traps too. For example, programming with threads is full of design traps. That doesn't mean that that tool shouldn't be made available to developpers, but that the documentation must warn strongly about those traps.

My point: let's make listtransactions available, but document it precisely, like any API, and explain the design mistakes that shouldn't be made when using it.

I think the important point is that these things are unavoidable unsafe after a certain point, even for listreceivedbyaddress.  So I think it's helpful to compare current website behavior under mainline bitcoin, with listtransactions.   Let us call this the confirmation point.  Addressing your queries...


bitcoinmarket and mtgox and other sites seem to consider 6 confirmations their "confirmation point", the moment at which a transaction may be considered "safe." If a past transaction becomes invalid and disappears, the website cannot avoid potential loss, because the user has already received their PayPal-USD or LR-USD or Pecunix GAU and disappeared.

Same for a web store or brick-and-mortar store.  There is a confirmation point at which the customer receives goods.  If a TX becomes invalid after that, the store takes an unavoidable loss, because the customer is already gone with the purchased goods.


Let us assume that confirmation count for txid 0x1234 is bouncing wildly between 0 to 10 to 0 and back and forth, due to a large number of block reorgs.

Whether it is listreceivedbyaddress or listtransactions, you still have a binary confirmation point, a moment in time, at which the transaction crosses the "approved by store" level of confidence.  At that confirmation point, the customer leaves with purchased goods, and store takes a loss regardless of further block chain or TX behavior.

I do agree a programmer may make a mistake, and assume that number of confirmations will always increase.  But that human mistake, too, will cause danger when used with listreceivedbyaddress.


On this point, I agree that listtransactions presents some additional danger here, due to changing txid, if and only if the new double-spend matches destination bitcoin address being sought by the JSON-RPC user.

Nevertheless, in this case too, the user's security rests entirely on their level of confidence:  if the TX is replaced before 6 confirmations, software will likely not notice anything.  If the TX is replaced after 6 confirmations, the customer has already left with purchased goods, and the bitcoin user takes a loss.


This is simply inherent to bitcoin itself.  A block chain reorg might happen after 50 blocks.  But websites do not want to make their users wait 50 blocks before receiving goods.  It is the well-known snack machine problem.  listtransactions does not add anything to this problem, beyond that which is already vulnerable through listreceivedbyaddress.

Transactions can and will be replaced after the binary "confirmation point."  All users of bitcoin must figure this into their business plans, just like they account for credit card chargeback risk or shoplifting risk.

Then how do you cope with the issues I listed in the message you quoted?

At some point, a website or person accepting a transaction must take this risk.  It is unavoidable, whether you use listreceivedbyaddress or listtransactions.  This is why listtransactions reluctance seems so unusual.

Almost every exchange or website accepting bitcoins achieves a binary decisionpoint, where the transaction is accepted, and goods are shipped or money is exchanged.  After that binary decisionpoint, even if the block chain is reorg'd or transactions disappear, there is nothing the website can do but take a loss (or pursue a refund outside of bitcoin).

From the website's point of view, there is zero effective difference between 'listtransactions 6' and 'listreceivedbyaddress 6', because the end result to the website operator is the same:  the goods have been shipped / order accepted / money exchanged.

I can see the motivation for davux's request. In developing the e-commerce plugins I've released, I've wondered about the viability long term of using a "dump" method against the wallet, since over time the data returned is just going to grow and grow.

What I'd prefer is being able to attach a JSON-RPC callback URL to an account or an address, maybe to be invoked each time a confirmation or any other status change to that account or address comes in, until the callback is cleared. That would eliminate the need for polling the wallet entirely.

I threw a "feature request" post in about that someplace.

The wallet is a key/value db4 database (and a key/value map in RAM).

Neither data structure is ordered by time.

It's not safe to use listtransactions this way.

I know I've been criticized for being reluctant about listtransactions.  Let me explain my reluctance.

Transactions are dynamic.  Past transactions can become unconfirmed, go away and come back, become invalid and disappear, or be replaced by a different double-spend.  Their date can change, their order can change.

Programmers are naturally inclined to want to use listtransactions like this: feed me the new transactions since I last asked, and I'll keep my own tally or static record of them.  This will seem to work in all regular use, but if you use the amounts for anything, it is highly exploitable:
1) How do you know if a past transaction becomes invalid and disappears?
2) When there's a block-chain reorg, it would be easy to double-count transactions when they get confirmed again.
3) A transaction can be replaced by a double-spend with a different txid.  You would count both spends.

The model where you assume you only need to see new transactions because you've already seen previous transactions is not true.  Old transactions can change at any time.

Any time you take an action based on payment amounts received, you always need to go back to bitcoin and ask for a current balance total (or use move or sendfrom), and be ready for the possibility that it can go down.

Now that we have the Accounts feature making it easier to do it the right way, we're better prepared to have listtransactions.

Surely since the WalletTx are added sequentially to the wallet, we can assume that older txid's appear earlier in the wallet?

Then it's simply trivial to do a linear lookup and then continue on spitting out the txid's after that one.

Can anyone confirm this?

If we're adding code to watch for new transactions, then we should be including gavin's monitor patches:
https://github.com/gavinandresen/bitcoin-git/tree/monitorreceived

+1

I haven't got my hands dirty there yet, and would rather spend what time I find to do import/export of keys, but that is something that would help some of my scripts a lot.

It would be great to have a JSON-RPC method for listing new transactions that are newer than a particular transaction id. This would enable developpers to watch new transactions easily, by just keeping track of the latest known txid and polling that method at the rate of their choice.

A possible way to do it would be to extend listttransactions so that it accepts an optional txid argument:

If txid is given, don't show it, nor any transaction older than it.

(As a sidenote, maybe listtransactions, with or without txid, should accept filtering by transaction category: generate, send, receive, move.)

I'm sorry I can't propose any patch here, as I can't do any decent C++.