Topic: bitcoin core RPC compatible lite wallet proxy? (Read 4547 times)

Hello Paul,

I dont use proxy at all in newest core version. I have problem to reach the point when its loading blockchain.
They cannot help me at work. I asked even engineer buddy to help =(

It already does exist.  Again, check out http://synala.com/

I'm not saying use Synala, but it is open sourced on Github, so you should at least check out the code and database schema to see how it's done properly and securely, as Synala supports exactly what you're looking for.

Anyway, this thread has been enough of a time suck.  I'm out.

My plan is to have the daemon process continually sync by polling listsinceblock every N seconds. Then some kind of other accounting system to track which addresses belong to which 'account' (this is not the bitcoin accounting system), so that you can build raw transactions by gathering utxos.

Like I said in the OP, I didn't want to do any of this, since it seems like it ought to already exist, but there we are.

Ok, so what's your plan of attack here?  You're going to run "listsinceblock" via crontab every 5 minutes, or?

Also, your OP mentioned you need to separate multiple wallets with one install of Bitcoin Core.  How are you going to tackle that with listunspent and listsinceblock?  You need some way to organize them, as to which goes to what wallet.

listsinceblock is more robust because you can just store the last processed block and get all transactions from that point. This negates the need to have a special case for a retry queue.

So instead of creating a retry queue, you're going to rely on "listunspent" instead?  How are you going to separate that with multiple wallets / systems?  How will you know which UTXO goes where?  Using the "accounts" within bitcoin core, or?

Not bitcoind, your own daemon process which is being notified. If that suffers some outage, you're left out of sync.

No.  If bitcoind goes down for a few hours for whatever reason, it'll pick up where it left off, and fire all necessary blocknotify commands from the time it went down as it downloads the rest of the blockchain.

I'm not using either. I'm using importaddress, but its not a BIP32 generated key.


And if your monitoring daemon goes down? I don't use blocknotify for that reason. Any chain scanner must be able to carry on where it left off after a power cut or any other outage. Relying on blocknotify for that would leave you out of sync with the network.

Depends.  How are you generating these addresses?  Are you using the wallet.dat file and getnewaddress RPC command?  Or are you using a BIP32 key(s)?  If the latter, then at the very least you need to store a database of addresses with their associated key indexes, otherwise you won't know how to generate the proper private keys to send the funds.  Even without that, you're still going to need to separate the UTXOs between your multiple wallets somehow, and bitcoin core isn't going to do that for you.

The blocknotify is used for confirmations.  One block = 1 confirmation, right?  Say your software is set to confirm a tx after 4 confirms.  So every time blocknotify fires, you add 1 confirmation to each tx.  Once a tx in your system hits 4 confirmations, you look it up in the blockchain using gettransaction, and ensure it still shows 4 confirmations there.  If a block reorg happened, the gettransaction command will report back with -1 confirmations, so you know to abandon that tx.

It would help if you give performance requirements.

- number of addresses
- number of concurrent requests
- response time

Otherwise, nearly anything can be a UTXO database. Including the blockchain.

That's basically equivalent to using listunspent, or listsinceblock in combination with the watch only features. I personally wouldn't build a system around blocknotify as its fragile and cannot deal with historical data.

No.  Add "walletnotify" / "blocknotify" options to your bitcoin.conf file.  Then use the "importaddress" RPC function to import any addresses your software generates, and bitcoind will fire the "walletnotify" command any time funds hit any address you've imported.  Then use "blocknotify" to track confirmations within your system.

Assuming you're using a HD BIP32 wallet though, then the "listunspent" isn't going to be enough, especially if you want to separate things between wallets / systems.  You'll also need to store, a) which wallet the funds are assigned to, and b) which key index was used to generate the address, so you know how to generate the child private key from the BIP32 master key.

Again, maybe take a look at the code in Synala if you know PHP.  It provides good examples of how this is done.

Doesn't that mean you need to manually parse the blockchain for UTXOs to your scripts/addresses? That means calling getblock and then getrawtransaction for each transaction in every block?

I don't buy your claim that your solutions "work beautifully". You are basically serving the market of "everyone knows the root password of the server". You can't properly provide elementary security like, e.g.

1) the web design subcontractor can't have the passwords to the accounting server
2) the are two independent web store fronts: the production one that is open 24*7 and the staging one for beta testing that is open intermittently and possibly managed by a different webmaster

It is also very telling that in your last paragraph you foresee closing down web front-end while having the accounting open. But the most real businesses operate in an opposite way: the web front end is open 24*7 but accounting is regularly being closed (weekly, monthly, quarterly) for periodic reporting and auditing.

I understand that there is a big market for "fly-by-nights" that essentially operate until the first audit by taxation authorities or first investigation by police/prosecution/regulators. I can't blame you for serving it, but I will stay away and always advise everyone else to stay away too.

There's a big disconnect between the two markets you mentioned: one-password shop with up to five people and multibillion corporation. The best example is what Microsoft targets with Server Essentials (a.k.a.) Small Business Server. The organizations of up to about 50 people who tend to subcontract various non-essential activities to the specialist firms. The people who know how to produce reports using Microsoft Access and why every commercial SQL database engine has some sort of per-column security.

Anyway, returning back to the ideas described in the opening post of this thread: it would be nice to have a Bitcoin engine that works somewhat like any other database engine. In the sense that you can start the engine itself to track the Bitcoin P2P network and then attach and detach wallets like one would attach and detach databases to the SQL engine.

But this isn't going to happen under current Bitcoin Core Development Team. In 2012 I mentioned something similar in the "What's the problem with getting Bitcoin compliant with GAAP???" and Gavin Andresen replied:
I think the current governance situation in the Core team could lead to the change in the general direction of the development. Time will show.

Ok, we're talking about two completely different things here.  You're talking about large organizations with multiple corporations and bank accounts spread across the globe, who deal with 10s or 100s of millions in funds, and report to the likes of FinCEN, correct?  In that case, I doubt an out of the box solution would even be possible, because every system would have to be customized for that specific business (model).  I'm assuming folks like Bitstamp go with custom solutions, because they have millions in their budget to do so.

And yes, my solutions are meant for the small business owners, the 1 - 5 man shops or similar.  People that make say $100k - $500k/year.  For my target audience, my solutions work beautifully.

If you're looking towards folks like BitStamp, et al, then sounds as though the market is wide open for you, if you can put together even a base system that can be easily customized.  You're basically talking about writing your own node, correct?  That's a fairly adventurous task, and many have failed at trying, including myself.  Could be lucrative if you managed to pull it off though.  Probably wouldn't get too many clients, but when you're signing $1.5mm contracts, you don't really need many.

EDIT:  You mentioned the bitcoind & "accounting" servers must both be online, but that's not true.  If bitcoind goes down, when it comes back online, it'll begin firing all necessary "blocknotify" and "walletnotify" commands as it downloads the blockchain from where it left off.  Then just write a quick JSON API to communicate between the servers, queue the txs / blocks, send JSON request, if a proper response is received remove tx / block from queue.  If proper response not received, leave in queue and retry at regular intervals until proper response is received.

What you describe is acceptable only to one-person shops or similar setups.

Most organizations will have clear interdepartmental security delineations and will have a preexisting accounting system already based on the (accounting, securities, etc.) prevailing standards.

"walletnotify" and "blocknotify" are really hacks that cross the security domain in wrong directions: from the Internet front end to the accounting servers. Only desperados will approve such architecture. The other problem with event callbacks are:
1) is that they require both caller & callee systems both online at the same time
2) they have no indication of the backlog of pending callbacks.

While conceptually handling chain reorganizations isn't terribly difficult it is also the feature that most often goes nearly completely untested. It is one of the most common failure modes in various solutions mentioned on this forum.

The other problem is purported scalability of having one blockchain and multiple wallet. Again the practice had show that people handle they wallet-synchronization code with tight poll loops and other similar lameness, that in practice multiple copies of bitcoind take less CPU and bandwidth resources only more storage. I think Stratum & Electrum is a good example: Stratum protocol in theory handles asynchronous notifications well, but its practical implementation in Electrum is plagued by horrible inefficiencies.

The last thing is that even if you solve all the above problems you still don't have a clean interface to the industry-standard accounting tools: Fix protocol in securities and various "middleware/message queueing" (like e.g. ZeroMQ) in general accounting. On the other hand the raw bitcoind can be run with -privdb=0 (and minimal patches) and allow to fully integrate with accounting back-ends thanks to the support for X/Open distributed transaction monitoring in BerkeleyDB than handles the wallet.

In summary: individual problems aren't difficult by themselves, the complexity is in integrating all of them. The sad state is that various folks peddle incomplete solutions plagued with tight poll loops and with untested/incorrect handling of chain reorganizations. The only exception was bitsofproof but it is back to being closed source. Almost nothing has changed in this space since I joined this forum sometime mid-2011.

You seem to be an advocate of Synala. I have one question: did your solution survived at least one accounting audit conducted to the prevailing accounting standards in your country?

Well, on the positive side I can at least observe that folks ceased to propose "use built-in accounts in bitcoind". Before those got officially marked as obsoleted this was the closest thing to the willful accounting fraud.

Just checked and confirmed, yes, the "listunspent" function does list all UTXOs from watch-only addresses added via the "importaddress" function.  Should note, the watch-only inputs have the "spendable" attribute set to false, but that's the only difference i can see.

Wouldn't really recommend that method for multiple reasons, and better to keep your own database of UTXOs for your addresses, but your software, so your choice.

EDIT:  If you're by chance using a HD wallet, you somewhat have to keep a list of UTXOs yourself anyway.  Otherwise, if you're going to rely solely on "listunspent", you're not going to know what key index was used to generate the child key / address from the master BIP32 key, and that piece of information is quite important if you ever want to send the funds.

I assume it indexes the UTXOs for watch only addresses in exactly the same way (the only difference being that you don't have the private keys) so yes it should work with the "listunspent" RPC command.

The watch only wallet mode of bitcoin core doesn't support listunspent?