What is the best SPV for handling large number of users?

tl121

sr. member

Activity: 278

Merit: 254

Quote from: gmaxwell on October 28, 2015, 12:39:58 AM

Services providing services to other users should almost certainly not be using SPV.

It would be very helpful if you would explain your reasoning behind this. From a functional point of view, SPV either works or it doesn't. If it doesn't work properly on behalf of multiple users, it's not clear why it would work properly for a single user. If there is a performance problem with SPV supporting a service with multiple users, it's not clear why there would be a performance problem with those same nodes supporting multiple SPV clients.

This seems to be related to the basic design of bitcoin core, which has (since the beginning) combined three separate functions that would be clearly separated with well defined interfaces in any proper software engineered implementation: 1) maintaining the distributed ledger and associated local database, 2) providing database and notifications on behalf of client applications, and 3) providing an easy to use graphic user interface suitable for bitcoin newbies.

BTCDDev

sr. member

Activity: 255

Merit: 251

A little update, I've got it working really well, associating user addresses (unique for each deposit) with each user in my rdbms, implementing the cold wallet storage now.

It was tricky to efficiently monitor user deposits in real time, as currently you need to use walletnotify, which as I stated before launches a new process each time an incoming tx is seen, and the zeromq implementation only allows for sending every seen tx on the network, which would force me to use rpc call for EVERY tx!

I opened a pull request here https://github.com/bitcoin/bitcoin/pull/6925

but it was closed, so for now others who want to monitor deposits will have to do so in a loop tx-->bitcoind-->my application-->bitcoind rpc-->my application

Envrin

sr. member

Activity: 318

Merit: 251

Whoops, thanks for pointing the small typo out. All fixed.

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: Envrin on October 29, 2015, 02:14:35 AM

http://synala.com/

Looks interesting although from the "How secure are incoming funds?" FAQ I found this:

Quote

...While Synala does allow the convenience of online singing...

Am guessing you meant "signing". Cheesy

Envrin

sr. member

Activity: 318

Merit: 251

http://synala.com/

Might do the trick for you. Will easily support a large number of users, can handle both product purchases and invoices, keeps all tx data you need, generates a new address for each deposit, has a hook system so you can easily run additional PHP code when various actions occur, etc. Free, and open sourced on Github. It does require a full node on your server though, although doesn't use the wallet.dat file.

If you need a software system that's a little more commercial / enterprise level, we do have that available, and just drop me a PM with requirements.

BTCDDev

sr. member

Activity: 255

Merit: 251

Quote from: CIYAM on October 28, 2015, 11:25:15 PM

Quote from: BTCDDev on October 28, 2015, 11:07:33 PM

I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

So basically for incoming funds you can use "listunspent" for the wallet determining account balances according to specific addresses (and adjusting your RDBMS account balances accordingly) and then move the funds to cold storage periodically.

So basically you need an index for Address and User (which presumably is only unique with the combination as each user should probably be encouraged to use different addresses for each deposit).

To handle re-orgs you might want to keep two balance figures one for amounts already shifted into cold storage and one for the total current set of UTXOs for each user and you'd probably only want to move into cold storage those UTXOs that are older than any re-org you think might occur (say 100 blocks to be safe even if a huge fork occurs).

You might also want to keep a third balance of "unconfirmed funds" just so a User can known that funds they have just sent are "on the way" (without them having to use blockchain.info or something else to check).

All good advice, thanks a lot.

Yes, I will mark an address as used or not, whenever a user wants to know his deposit address, if it's received funds already, it will generate him a new one.

I've been researching the bitcoind api, I am already familiar with zeromq (we use nanomsg in SuperNET) so it appeals to me as a much superior method of notifying my application of incoming funds than walletnotify (which spawns a new process(!) on each notification)

However, it looks like the zeromq flags don't allow for pushing the tx hashes of only those transactions which contain outputs to my wallet. I forked btc and added one in myself, if it works well then I may submit a pull request at some point. Seems like a big oversight.

Also, yeah the unconfirmed balance as a separate field is a good idea.

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: BTCDDev on October 28, 2015, 11:07:33 PM

I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

So basically for incoming funds you can use "listunspent" for the wallet determining account balances according to specific addresses (and adjusting your RDBMS account balances accordingly) and then move the funds to cold storage periodically.

So basically you need an index for Address and User (which presumably is only unique with the combination as each user should probably be encouraged to use different addresses for each deposit).

To handle re-orgs you might want to keep two balance figures one for amounts already shifted into cold storage and one for the total current set of UTXOs for each user and you'd probably only want to move into cold storage those UTXOs that are older than any re-org you think might occur (say 100 blocks to be safe even if a huge fork occurs).

You might also want to keep a third balance of "unconfirmed funds" just so a User can known that funds they have just sent are "on the way" (without them having to use blockchain.info or something else to check).

BTCDDev

sr. member

Activity: 255

Merit: 251

Quote from: CIYAM on October 28, 2015, 10:49:46 PM

I think perhaps look into running an Electrum server https://roll.urown.net/server/bitcoin/electrum-server.html (and https://github.com/spesmilo/electrum-server/) as it will index the UTXOs by address (the key thing needed really).

Note that it also requires "bitcoind" to be running (so you'll need a lot of disk space and presumably some decent hardware as well although if you are planning to service thousands of users am guessing you would already be planning for that).

I've got the hardware. I think I'll just use bitcoind and keep user addresses and balance in a rdbms, requiring a high level of confirms to reduce the chance of reorgs to near 0. If total wallet unspents become large, I'll move excess funds to multisig offline wallet.

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

I think perhaps look into running an Electrum server https://roll.urown.net/server/bitcoin/electrum-server.html (and https://github.com/spesmilo/electrum-server/) as it will index the UTXOs by address (the key thing needed really).

Note that it also requires "bitcoind" to be running (so you'll need a lot of disk space and presumably some decent hardware as well although if you are planning to service thousands of users am guessing you would already be planning for that).

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: BTCDDev on October 28, 2015, 10:14:08 AM

This is ok for hot wallets, but I will want to keep large funds stored offline in a cold wallet.

I could keep cold wallet addresses in the hotwallet as watch-only, but how would I relate an unspent to a cold address to calculate the balance of a specific user?

The "listunspent" output includes a "spendable" attribute (which I assume will be "false" for watch-only addresses). So you can determine the total balance and the "immediately spendable" balance also (if that is the concern).

Bear in mind if you do use a "bitcoind wallet" then you are going to have big problems once it has hundreds of thousands of addresses (from what I've read from others it doesn't scale well).

Although I've not looked into the code my guess is that even if you provide "listunspent" with a set of addresses it will iterate through every UTXO that the wallet contains (so will get slower and slower as the UTXOs increase).

BTCDDev

sr. member

Activity: 255

Merit: 251

Quote from: CIYAM on October 28, 2015, 09:41:14 AM

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).

EDIT: Hmm... it seems you still can't do "listunspent" for an address that is not in your wallet (am rather surprised that hasn't been added as it makes sense for "bitcoind" to have that feature so people don't resort to using things like blockchain.info). The indexing flag is only for txid lookup (which can be problematic due to malleability as mentioned).

This is ok for hot wallets, but I will want to keep large funds stored offline in a cold wallet.

I could keep cold wallet addresses in the hotwallet as watch-only, but how would I relate an unspent to a cold address to calculate the balance of a specific user?

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

Quote from: _mr_e on October 28, 2015, 10:05:28 AM

This sounds like something that should have already been done 1000x by now, is there really not a better way or some kind of library or wrapper that handles this stuff?

Although I think it is a pity that "listunspent" (assuming some sort of indexing flag has been put into the config) can't be used to find UTXO's for any address the main problem that you have in trying to link "bitcoind" to an external RDMBS is the whole "re-org" thing.

In the RDBMS world once you "commit" a transaction you can no longer "roll it back" but in the blockchain world this is simply not the case - thus marrying the two technologies is always going to result in some issues (as are often reported about for blockchain.info).

For the Bitcoin wallet I developed I have a "Check Balance" button to issue a "listunspent" in order to make sure it keeps in sync (of course that check could be automated but it would still need to be done prior to trying to create a new tx just in case the UTXO set has changed).

_mr_e

legendary

Activity: 817

Merit: 1000

Quote from: CIYAM on October 28, 2015, 09:41:14 AM

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).

This sounds like something that should have already been done 1000x by now, is there really not a better way or some kind of library or wrapper that handles this stuff?

CIYAM

legendary

Activity: 1890

Merit: 1086

Ian Knowles - CIYAM Lead Developer

You most likely do not want to use the "bitcoind" idea of "accounts" (it is not the same idea that anyone who has ever done bookkeeping could make any sense of) and the wallet has scaling issues.

I think you are probably best to use the RPC to extract UTXO information from "bitcoind" into accounts that you keep in an RDBMS (from memory that will require a special flag for indexing but admittedly I haven't actually used that so you'll need to read up on exactly what that does).

The key command to retrieve UTXOs is "listunspent" although normally that will only output UTXOs that belong to your "bitcoind wallet".

Another very important thing to understand is you need to keep your RDBMS accounts "in sync" with the blockchain (small re-orgs of one or two blocks do occur and you should be very careful about unconfirmed txs).

EDIT: Hmm... it seems you still can't do "listunspent" for an address that is not in your wallet (am rather surprised that hasn't been added as it makes sense for "bitcoind" to have that feature so people don't resort to using things like blockchain.info). The indexing flag is only for txid lookup (which can be problematic due to malleability as mentioned).

BTCDDev

sr. member

Activity: 255

Merit: 251

Fair enough, running a full node was my original plan anyway.

However, I've only heard negative things about using bitcoind's accounting system, that it should not be used in production environments.

Alternative method?

jim618

legendary

Activity: 1708

Merit: 1066

Yes got to agree with Greg on this.

We did some consultancy for an exchange and recommended they ran Core nodes for their network layer.
SPV is perfectly ok for single users but you want full nodes to support large numbers of users.

gmaxwell

staff

Activity: 4284

Merit: 8808

Services providing services to other users should almost certainly not be using SPV.

BTCDDev

sr. member

Activity: 255

Merit: 251

Just like the title asked. I need an SPV for handling a large number of users. Needs to have support for accounting.

Running Ubuntu.

Currently looking at picocoin and libbitcoin.

Topic: What is the best SPV for handling large number of users? (Read 3159 times)