Topic: Is bitcoind reliable when depending on it for web services? (Read 2144 times)
Polling = fail
I cant work out a decent way to handle a high number of users with bitcoind - the problem im having is seeing what new transactions have come in.
listtransactions appears to be severely flawed - you have to walk backwards though the list (how do I avoid missing something if it can be re-organised at any time?).
List unspent is wallet wide, and it would only work if the entire transaction is cleared from the wallet after it comes in (i'm assuming change addresses would show up here after a transaction)
having one account for the entire site, with 24 hour throwaway deposit addresses could work (they'd only be polled waiting for incoming transactions for 24 hours) - but its not ideal for the user.
walletnotify might work. but its not supported by litecoind (yet?)
how does everyone else do it?
listtransactions appears to be severely flawed - you have to walk backwards though the list (how do I avoid missing something if it can be re-organised at any time?).
List unspent is wallet wide, and it would only work if the entire transaction is cleared from the wallet after it comes in (i'm assuming change addresses would show up here after a transaction)
having one account for the entire site, with 24 hour throwaway deposit addresses could work (they'd only be polled waiting for incoming transactions for 24 hours) - but its not ideal for the user.
walletnotify might work. but its not supported by litecoind (yet?)
how does everyone else do it?
Busy doing development on a Bitcoin service at the moment. Although we haven't launched yet (soon!), we've tested the system and everything seems to working quite well so far. (Not sure on scalability of course).
You have to use the listsinceblock command. This makes getting new transactions much clearer. The ones that come in, you're sure will be recent transactions you have not seen before. Then you sort the list via the 'timereceived' parameter. Depending on how you handle transactions, but once a new transaction is confirmed, you save the blockhash. Now when you poll for new transactions, you have a place to start from.
You'll probably have to create a replica of the wallet transactions in another DB for persistency (if the server goes down etc).
I hope this helps a bit more, it's hard to give exact advice if I'm not exactly sure what you need. However, I would definitely recommend looking into the listsinceblock command to get your transactions, rather than listtransactions.
I cant work out a decent way to handle a high number of users with bitcoind - the problem im having is seeing what new transactions have come in.
listtransactions appears to be severely flawed - you have to walk backwards though the list (how do I avoid missing something if it can be re-organised at any time?).
List unspent is wallet wide, and it would only work if the entire transaction is cleared from the wallet after it comes in (i'm assuming change addresses would show up here after a transaction)
having one account for the entire site, with 24 hour throwaway deposit addresses could work (they'd only be polled waiting for incoming transactions for 24 hours) - but its not ideal for the user.
walletnotify might work. but its not supported by litecoind (yet?)
how does everyone else do it?
listtransactions appears to be severely flawed - you have to walk backwards though the list (how do I avoid missing something if it can be re-organised at any time?).
List unspent is wallet wide, and it would only work if the entire transaction is cleared from the wallet after it comes in (i'm assuming change addresses would show up here after a transaction)
having one account for the entire site, with 24 hour throwaway deposit addresses could work (they'd only be polled waiting for incoming transactions for 24 hours) - but its not ideal for the user.
walletnotify might work. but its not supported by litecoind (yet?)
how does everyone else do it?
Please proceed to getting a clue.
The accounts feature does not scale in bitcoind, once you pass a few 10k accounts it starts slowing everything down.
These problems instantly go away as soon as you start doing the accounting externally.
The accounts feature does not scale in bitcoind, once you pass a few 10k accounts it starts slowing everything down.
These problems instantly go away as soon as you start doing the accounting externally.
This is something I've been wondering recently. So if you expect to manage dozens thousands addresses, then it's better not to use bitcoind wallet at all, manage everything on your own (on the level of unspent outputs which your software choses to build rawtransaction) and operate on rawtransactions? Use bitcoind only as block reader and then for pushing tx?
Accounts does not scale if implemented naively.
A combination of local cache and subscription can however scale as i described before. BTW this is the logic of the wallet too, just tightly coupled in the satoshi implementation
A combination of local cache and subscription can however scale as i described before. BTW this is the logic of the wallet too, just tightly coupled in the satoshi implementation
Please proceed to getting a clue.
The accounts feature does not scale in bitcoind, once you pass a few 10k accounts it starts slowing everything down.
These problems instantly go away as soon as you start doing the accounting externally.
Accounts does not scale if implemented naively.
A combination of local cache and subscription can however scale as i described before. BTW this is the logic of the wallet too, just tightly coupled in the satoshi implementation
A combination of local cache and subscription can however scale as i described before. BTW this is the logic of the wallet too, just tightly coupled in the satoshi implementation
Don't use the "accounts" feat. it doesn't scale.
Manage accounts yourself in whatever data store you're using.
Manage accounts yourself in whatever data store you're using.
I suggest to have some cache at your side that is initialized with a scan and than subscribes for updating ttansactions.
The stateless calls can consult that cache. I would persist the cache and rescan from the server at startup from the time point it was last persisted. This avoids having inconsistent state with the servrer and gives best performance.
The stateless calls can consult that cache. I would persist the cache and rescan from the server at startup from the time point it was last persisted. This avoids having inconsistent state with the servrer and gives best performance.
The matchRequest or scanRequest is to scan the block chain. This would be useful if you import a private key or rescan history for a wallet.
You can submit a long list of patterns (addresses) with a single matchRequest, that is more effective, than one by one. The scanRequest is the same but with abloom filter, this would be useful for to scan for an ever larger set of patterns.
The filterRequest is an ongoing subscription to transactions coming in and matching the filter.
Beware that the bloom filter (for scanRequest or filterRequest) is probabilistic so you can get some transactions you do not care, but it reduces the regular transaction feed by magnitudes. The filter also gets updated automatically, so it will e.g. match for transactions spending a transaction that previously matched the filter. You might want to periodically reset the filter on a long standing connection.
The filter is a bitmap in style of a bloom filter at the moment I can only refer to https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/BloomFilter.java
how it is compiled from byte patterns (keys, addresses, outpoints) you want to filter for.
The Java class https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/DefaultAccountManager.java
shows how filter subscription and scanning is used to implement a client side wallet.
You can submit a long list of patterns (addresses) with a single matchRequest, that is more effective, than one by one. The scanRequest is the same but with abloom filter, this would be useful for to scan for an ever larger set of patterns.
The filterRequest is an ongoing subscription to transactions coming in and matching the filter.
Beware that the bloom filter (for scanRequest or filterRequest) is probabilistic so you can get some transactions you do not care, but it reduces the regular transaction feed by magnitudes. The filter also gets updated automatically, so it will e.g. match for transactions spending a transaction that previously matched the filter. You might want to periodically reset the filter on a long standing connection.
The filter is a bitmap in style of a bloom filter at the moment I can only refer to https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/BloomFilter.java
how it is compiled from byte patterns (keys, addresses, outpoints) you want to filter for.
The Java class https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/DefaultAccountManager.java
shows how filter subscription and scanning is used to implement a client side wallet.
Ok, so I think I've got this down, basically the idea should be rather than directly accessing the 'supernode' from a stateless PHP execution I should probably be creating something like a worker service that will subscribe to batches of addresses and then start storing the results of such like when an address has reached a target amount received?
Cheers for all the help, good luck with the full release
The matchRequest or scanRequest is to scan the block chain. This would be useful if you import a private key or rescan history for a wallet.
You can submit a long list of patterns (addresses) with a single matchRequest, that is more effective, than one by one. The scanRequest is the same but with abloom filter, this would be useful for to scan for an ever larger set of patterns.
The filterRequest is an ongoing subscription to transactions coming in and matching the filter.
Beware that the bloom filter (for scanRequest or filterRequest) is probabilistic so you can get some transactions you do not care, but it reduces the regular transaction feed by magnitudes. The filter also gets updated automatically, so it will e.g. match for transactions spending a transaction that previously matched the filter. You might want to periodically reset the filter on a long standing connection.
The filter is a bitmap in style of a bloom filter at the moment I can only refer to https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/BloomFilter.java
how it is compiled from byte patterns (keys, addresses, outpoints) you want to filter for.
The Java class https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/DefaultAccountManager.java
shows how filter subscription and scanning is used to implement a client side wallet.
You can submit a long list of patterns (addresses) with a single matchRequest, that is more effective, than one by one. The scanRequest is the same but with abloom filter, this would be useful for to scan for an ever larger set of patterns.
The filterRequest is an ongoing subscription to transactions coming in and matching the filter.
Beware that the bloom filter (for scanRequest or filterRequest) is probabilistic so you can get some transactions you do not care, but it reduces the regular transaction feed by magnitudes. The filter also gets updated automatically, so it will e.g. match for transactions spending a transaction that previously matched the filter. You might want to periodically reset the filter on a long standing connection.
The filter is a bitmap in style of a bloom filter at the moment I can only refer to https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/BloomFilter.java
how it is compiled from byte patterns (keys, addresses, outpoints) you want to filter for.
The Java class https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/api/DefaultAccountManager.java
shows how filter subscription and scanning is used to implement a client side wallet.
Is there by any chance someone where that documents all the possible messages that can be set and expected results etc? Even as a really rough document?
/queue/filterRequest - send a FilterRequest message to get a ongoing feed of Transaction messages on the reply queue that match the Bloom filter, this will keep going until you disconnect
/queue/matchRequest - send an ExactMatchRequest to scan the blockchain for transactions matching the filter, where filter is a list of byte vectors. The byte vectors can be addresses, keys or any data in a transaction script. This scans once starting from an optional time point "after" in the message. Resulting transactions will appear on the reply queue.
Thanks so ideally I'm guessing to monitor any of the addresses I've got to either:
1) do a MatchRequest for each address with my address as the receiving address?
Or
2) do I just create a filterRequest with a filter for my receiving address and I'll essentially get my transactions that way?
if so do you have a rough guess on how long these requests are likely to take?
Is there by any chance someone where that documents all the possible messages that can be set and expected results etc? Even as a really rough document?
I try quick and dirty here:
All messages are STOMP ByteArray where the payload is Protobuf as defined in https://github.com/bitsofproof/supernode/blob/master/api/src/main/protobuf/BCSAPIMessage.proto
Messages go to queues and topics. Queues are read by a single server instance (this does the load balancing) while topics are broadcast to all server and clients listening.
The following broadcast is allowed from clients to servers
/topic/newTransaction - send a new Transaction
/topic/newBlock - send a newly mined block
Synchronous requests from client to server:
Client sends a message to a request queue and sets the reply-to queue of the message to a temporary queue /queue/temp.XXXXX
The server that picks up the request will send reply to that temp queue.
The following synchronous requests are allowed from clients to a server:
/queue/blockRequest - send a Hash message to get a Block message on the reply queue
/queue/headerRequest - send a Hash message to get a Block message on the reply queue, but transactions field is empty
/queue/transactionRequest - send a Hash message to get a Transaction message on the reply queue
/queue/filterRequest - send a FilterRequest message to get a ongoing feed of Transaction messages on the reply queue that match the Bloom filter, this will keep going until you disconnect
/queue/matchRequest - send an ExactMatchRequest to scan the blockchain for transactions matching the filter, where filter is a list of byte vectors. The byte vectors can be addresses, keys or any data in a transaction script. This scans once starting from an optional time point "after" in the message. Resulting transactions will appear on the reply queue.
/queue/scanRequest - works like match request with bloom filter
/queue/ping send a ping msg to server you get back the same. This is to measure latency or to check availability.
Is there by any chance someone where that documents all the possible messages that can be set and expected results etc? Even as a really rough document?
Just a few days of patience please. We are in conference mode. Probably I get it written up while in the airport lounge
That is an excellent idea, VPN powered Virtual Server, multiple connections from strategic locations, excellent. Multi Coin management, This will definitly make it possible to go beyond the 500kb/block limit.
so how far have you been able to implement it?
It is implemented.
Product release is tomorrow.
You can run the server yourself, and we have also hosted offers, as of:
Bits of Proof launches world's first enterprise-ready Bitcoin server
The primary API is the Bus, that speaks STOMP and is transporting Protobuf format messages, so you can connect e.g. with python, ruby or PHP, I am sure you find examples using both STOMP and Protobuf quickly with google for your choice of language.
Bits of Proof launches world's first enterprise-ready Bitcoin server
The primary API is the Bus, that speaks STOMP and is transporting Protobuf format messages, so you can connect e.g. with python, ruby or PHP, I am sure you find examples using both STOMP and Protobuf quickly with google for your choice of language.
Is there by any chance someone where that documents all the possible messages that can be set and expected results etc? Even as a really rough document?
If not is there an alternative to be used that can keep up with an expanding web service?"
An alternative that can really take the heat of a web service is the BOP Enterprise Bitcoin Server.
You can run several instances of it and send requests to them through a load balancing message bus.
See the example repository https://github.com/bitsofproof/bop-explorer that shows a simple REST API forwarding to the bus.
The server code is in https://github.com/bitsofproof/supernode
Bits of Proof will launch its offer in two days in San Jose, dramatically reducing the pain of integrating with the Bitcoin protocol.
That is an excellent idea, VPN powered Virtual Server, multiple connections from strategic locations, excellent. Multi Coin management, This will definitly make it possible to go beyond the 500kb/block limit.
so how far have you been able to implement it?
In my experience bitcoind has been stable.
It's also fairly slow, and has been unable to handle minimal (~3k users) "getBalance/listtransactions/move" load on btct.co even with a caching layer on top of it for frequent queries.
I tackled the missing replication with frequent backups + external transaction logging. (such that I should be able to restore the wallet, then replay the transactions.) Rolling out a drbd implementation is next on my list.
Unfortunately I've also stumbled into a fairly major bug for a webservice. It's really strange but listtransactions does not return transactions in chronological order for me anymore.
https://bitcointalksearch.org/topic/listtransactions-output-is-not-in-order-205647
Not sure what's going on there.
Hope that helps somewhat.
It's also fairly slow, and has been unable to handle minimal (~3k users) "getBalance/listtransactions/move" load on btct.co even with a caching layer on top of it for frequent queries.
I tackled the missing replication with frequent backups + external transaction logging. (such that I should be able to restore the wallet, then replay the transactions.) Rolling out a drbd implementation is next on my list.
Unfortunately I've also stumbled into a fairly major bug for a webservice. It's really strange but listtransactions does not return transactions in chronological order for me anymore.
https://bitcointalksearch.org/topic/listtransactions-output-is-not-in-order-205647
Not sure what's going on there.
Hope that helps somewhat.
Sounds great, genuinely excited, have you got particular locations for running hosted solutions, like will you be running some one cloud providers like Amazon or will you be purely in house?
I'd love to put some time in to make a PHP client library if I can get the time for it, obviously might be quite difficult until you guys have full documentation down.
I'd love to put some time in to make a PHP client library if I can get the time for it, obviously might be quite difficult until you guys have full documentation down.
Bits of Proof already operates several servers at world class data centers in Germany.
Incremental server locations will be aligned with our customer's need and preference.
Sounds great, genuinely excited, have you got particular locations for running hosted solutions, like will you be running some one cloud providers like Amazon or will you be purely in house?
I'd love to put some time in to make a PHP client library if I can get the time for it, obviously might be quite difficult until you guys have full documentation down.
I'd love to put some time in to make a PHP client library if I can get the time for it, obviously might be quite difficult until you guys have full documentation down.
You can run the server yourself, and we have also hosted offers, as of:
Bits of Proof launches world's first enterprise-ready Bitcoin server
we even have a complementary offer during Bitcoin 2013.
Pricing is not by request, but we offer maintenance and support contracts of several levels and hosted servers, the later will be priced by vCPU allocated.
The hosted solution saves you the effort to set up a server for development and I hope you will find it care free for your production.
We will have shared and dedicated servers and also load balanced server farms depending on your need.
The primary API is the Bus, that speaks STOMP and is transporting Protobuf format messages, so you can connect e.g. with python, ruby or PHP, I am sure you find examples using both STOMP and Protobuf quickly with google for your choice of language.
The rest API in the bop-explorer is just a demo how easy is to create one. The BCCAPI that powers the Android app is also technically a REST adapter over the Bus.
The clients side Java code using the Bus API adds a layer of abstraction, that can surely be achieved with other languages. I hope that a community around the product will share those that are popular. You are not exposed to java directly if you do not want it. In the hosted case you do not even have to care of compiling and running the server using java.
The weakest point in our offer is the documentation, but this is just temporary as all resources went into development to hit the target for Bitcoin 2013, and we did. Next focus is documentation. I believe the example code you find in our GitHub repos give a head start.
Bits of Proof launches world's first enterprise-ready Bitcoin server
we even have a complementary offer during Bitcoin 2013.
Pricing is not by request, but we offer maintenance and support contracts of several levels and hosted servers, the later will be priced by vCPU allocated.
The hosted solution saves you the effort to set up a server for development and I hope you will find it care free for your production.
We will have shared and dedicated servers and also load balanced server farms depending on your need.
The primary API is the Bus, that speaks STOMP and is transporting Protobuf format messages, so you can connect e.g. with python, ruby or PHP, I am sure you find examples using both STOMP and Protobuf quickly with google for your choice of language.
The rest API in the bop-explorer is just a demo how easy is to create one. The BCCAPI that powers the Android app is also technically a REST adapter over the Bus.
The clients side Java code using the Bus API adds a layer of abstraction, that can surely be achieved with other languages. I hope that a community around the product will share those that are popular. You are not exposed to java directly if you do not want it. In the hosted case you do not even have to care of compiling and running the server using java.
The weakest point in our offer is the documentation, but this is just temporary as all resources went into development to hit the target for Bitcoin 2013, and we did. Next focus is documentation. I believe the example code you find in our GitHub repos give a head start.
Jump to: