It would.. but I suggest you take a look at https://vibanko.com/ before you start doing anything like this it's open source.
This is awesome. Thanks!
Topic: Scalability of BitCoinD for a wallet service (Read 2860 times)
This is awesome. Thanks!
On the topic of scalability, would setting -connect=xxxx.IP help in offloading some of the processing? If it does then we could setup a dedicated node to connect to
It would.. but I suggest you take a look at https://vibanko.com/ before you start doing anything like this it's open source.
On the topic of scalability, would setting -connect=xxxx.IP help in offloading some of the processing? If it does then we could setup a dedicated node to connect to
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.
Load balancing is definitely one method and it has crossed my mind, but that would defeat the "move accounts" feature that provides instantaneous transfer for users within the system (same wallet.dat). Having multiple bitcoind instnaces and moving counts from one wallet to other would also create additional network transactions in addition to delays required for confirmations.
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.
I bet a load balancing method is trivial indeed; Just thinking about securing an API system to query eachother would be frustrating, but that's only because I would go MySqlDB route is why I say this.
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
If you want to scale really big with this approach you need several servers each running bitcoind, and each managing their share of accounts. A load balancer for this setup is fairly trivial to set up.
You should talk to Jan at instawallet about scaling. I know he has struggled off and on with scaling and would probably be able to offer some good advice on what to do and what to avoid.
Looking at the various arguements brought forward, I am going ahead with the "let bitcoind manage" the transactions approach using the "Accounts" feature as a means of storing balances. At the moment I'd rather deal with scalability issues vs having unscynchronized balances between a separate DB vs wallet.dat.
There does not seem to any performance metrics as to what would constitute an upper limit for storing transactions in the wallet.dat. I have tried running some simulations by dumping more than 200k "move" transactions into the wallet and so far it appears to be running fine. I am assuming that the hardware in use would also play a critical role as to how far this would scale. However, the real tests would be concurrent use.
Thank you for all your suggestions and if may add this in as a feature requests (somewhere down the long list), it would be nice to be able to store the transactions into a separate DB i.e odbc, mysql, etc
Cheers
-MT
There does not seem to any performance metrics as to what would constitute an upper limit for storing transactions in the wallet.dat. I have tried running some simulations by dumping more than 200k "move" transactions into the wallet and so far it appears to be running fine. I am assuming that the hardware in use would also play a critical role as to how far this would scale. However, the real tests would be concurrent use.
Thank you for all your suggestions and if may add this in as a feature requests (somewhere down the long list), it would be nice to be able to store the transactions into a separate DB i.e odbc, mysql, etc
Cheers
-MT
Optimizing the accounts code to add a berkeley db index table that indexed wallet transactions by account, and that cached account balances (and invalidated or updated the cache on receive/send) shouldn't be terribly hard for somebody who already knows c++ and berkeley db.
It is not on my short-term TODO list because there are too many other higher priority things on my TODO list, but a nice clean well-tested upward-compatible patch would be most welcome.
PS: for ClearCoin, I used the "bitcoind keeps track of the bitcoins" architecture, and I never regretted it-- no problems with synchronization, less possibility for MtGox-like hacks that create mythical bitcoin balances out of thin air by adding an entry to a database.
It is not on my short-term TODO list because there are too many other higher priority things on my TODO list, but a nice clean well-tested upward-compatible patch would be most welcome.
PS: for ClearCoin, I used the "bitcoind keeps track of the bitcoins" architecture, and I never regretted it-- no problems with synchronization, less possibility for MtGox-like hacks that create mythical bitcoin balances out of thin air by adding an entry to a database.
So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
My professional opinion is as follows: (1) for the short-term do a recompilation and any small modifications to the Satoshi client to let it inter-operate correctly with the rest of your infrastructure; (2) for the long-term keep your fingers crossed that the libbitcoin team (bitcoinconsultancy?) delivers on their promises.BerkeleyDB isn't that difficult to master and the documentation is top-notch superb. I'm actually quite positively surprised with what had happened to it under Oracle's wing. The new release even has the SQL built in. But you'll have to at least recompile and possibly make number of small modifications.
I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.
So in your opinion you believe that it is better to handle with berkelyDB given that you know what your doing?
I guess I just prefer MySql to manage it instead
And herein lies the problem. Switching from BerkeleyDB to any relational DBMS is a quite nontrivial tasks. Many people undertook it and nobody delivered a working code yet. Thankfully Gavin isn't accepting patches that modify the basic behavior of the Bitcoin with respect to block chain reorganizations.As for those who modify the underlying database without understanding the code: this was probably the story of MyBitcoin if one takes Ted Williams' posts on the face value. Some programmers completely cannot grasp the concept of chain reorganization and write code that is subject to an abuse that the unmodified Satoshi client is completely invulnerable.
Okay yeah so in that case yeah you can manage it, I guess I just prefer MySql to manage it instead
So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?
Not "bitcoin address" but "bitcoin account". Please read carefully the output from "bitcoind help" and the original post. Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things:1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.
So your saying I can move bitcoins from one bitcoin address to another bitcoin address with out waiting for confirmations from the block chain?
can anyone else verify this?
Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation.
You seem to be missing two things:1) RPC move between the accounts instantly
2) wallet.dat can be modified live with correctly compiled BerkeleyDB program that uses the same DBENV. wallet.db is a database and BerkDB is a quite powerful DBMS, its just that most programmers (including the 'core development group') are unfamiliar with it and don't understand its features.
Honestly, I think its more work to go based off the bit coin daemon, if you want to move funds from a receiving bit coin address to a specific address in the same wallet it will have to go through the block chain which will add more wait time in waiting for even just 1 confirmation. Its just easier to use a database and periodically scan your transactions and then update the database with Bitcoin addresses with corresponding information like confirmations, balance, etc,etc. This method can help save block time and its more efficient to look up account information as well, since if you relied on the daemon for information you'd have to rescan every request which is inefficient. With a database you can cache all data in the wallet in one swoop then have the program query the database.
I'd take a look at this thread:
https://bitcointalksearch.org/topic/bitcoind-for-php-25094
This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight. I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).
https://bitcointalksearch.org/topic/bitcoind-for-php-25094
This fork makes it easier to poll/import transactions into your own DB and then do whatever you need with them rather than relying on bitcoind to keep accounts straight. I'm using it for my own purposes (though I'm not accessing it from PHP but Python; I still need to keep track of transactions in a RDBMS).
Thank you for the tip. However, all things being equal I would prefer to stick with the mainstream releases
Can I ask why you don't want to use your own code to manage balances and transactions? It seems to me that you need to do those things anyway, at least in most scenarios that I'm familiar with.
By having the bitcoind manage the balance and transactions I could avoid any synchronization and integration issues. This would also simplify the development process
Can I ask why you don't want to use your own code to manage balances and transactions?
I'm not mtbitcoin, but I can give you an answer to this: the current code is horribly complex and doesn't obey any rules demanded by any legitimate accounting methods. For example it routinely re-dates the transactions and can disappear them upon block-chain reorganization.The whole idea of stochastically-solve-the-inverted-one-dimensional-knapsack-problem before you can compute the transaction fee is hair raising to any legitimate accountant. On the other hand it is music to the ears of people planning to pull some sort of a fraud.
Jump to: