Proposal: New RPC interface for bitcoind

This is the beginning of a proposal for a new RPC interface for bitcoind to query and receive notifications about the daemon status, block chain or specific transactions. This is incomplete but I am seeking comments if this is a good idea and worth the effort to implement.

This is intended to address the following issues

Allow separating the bitcoin-qt GUI and wallet code from the daemon code
- Allows multiple instances of a modified bitcoin-qt with wallet code (and other clients) to interact with one instance of the daemon. This is useful for multiple users on the same system or to allow a small set of daemons to be used by other computers on a trusted network. This also makes it easier for users to run the daemon all the time while running the GUI on demand.
- Removes all private keys from the daemon process space. This implementation would allow only having interested public keys (addresses) in the address space of the daemon for a short period of time for the extra paranoid.
Provide a robust mechanism to receive updates for anything relevant including new blocks and transactions.
Allow receiving all updates relevant to a specific wallet very quickly assuming the bitcoind is trusted by the connecting client.

The current JSON-RPC interface does not provide a mechanism for real time updates or to request specific transactions matching some criteria notably spendable by a certain keys or having a specific scriptPubKey. This proposal is primarily a description of messages between the client and server where the server is bitcoind. Google protocol buffers will be used here though it would be possible to provide similar functionality with JSON, XML or some other format. This proposal does not define the exact mechanism for sending these messages. Accordingly it can be integrated as new calls to the existing interface though this proposal does overlap with some of the existing RPC functions and has distinctly different conventions.

This interface allows a client to create, query and get notifications for a context containing a list of public keys, scriptPubKeys or transactions ids the client is interested in. A context is identified by an arbitrary byte string (should it be fixed or limited length) context identifier that could be derived from the keys in the context. This allows multiple clients to share the same context.

A single Context message can represent the full state of a context. A notification Context message contains a subset of a full Context message containing all new data. The message is structured so all changes can be represented by either updating a fixed field or adding a new message to a repeated field. An object is removed from a repeated field by adding it to another repeated field that is mutually exclusive from the original. The same message is also used to specify which aspects of the context the client will need notifications for. This enables a client to request just the required information for example allowing a client to receive an account balance without receiving the specific transactions.

In addition to client defined contexts each bitcoind will have single Global context containing connected peers, general block chain information or anything else not predicated on something specified by the client.

Code:

service Bitcoin {
// AddtoContext either creates or updates a context specified by past Context messages.
// The Context message sent can only contain fields labelled Load below.
rpc AddtoContext(Context) returns Context;

// Returns a Context message containing all data needed to restore the context without scanning the block chain
rpc DumpContext(context_id) returns Context;

// Remove all data associated with a context from memory.
rpc RemoveContext(context_id);

// SetContext specified what fields the client would like notifications for. Every field that is included in the passed Notification message will be set to receive notifications. This replaces the current notification set each time called.
rpc SetNotify(Notification);

// Get the complete context data filtered by SetNotify, commonly called before GetNotifications
rpc GetContext(context_id) returns Context;

// One mechanism to receive notifications. Each call would return a Notification message containing all data that has changed since the last call.
rpc GetNotifications() returns Notification;
}

message Notification {
optional Global global;
repeated Context contexts;
}

message Global {
optional PeerInfo peers;
optional BlockChain chain;
}

message Context {
required bytes context_id;

repeated PublicKey keys;
repeated PublicKey keys_delete;
repeated scriptPublicKey scripts;
repeated scriptPublicKey scripts_delete;
repeated Transaction transactions;
repeated Transaction transactions_delete;
// Add more repeated fields for additional transaction criteria

// For setting notifications (SetNotify) the deleted fields are not valid and each item can only be specified once making it equivalent to optional instead of repeated in that case.
}

message PublicKey {
// All PublicKey messages must contain either a public_key or i if the message is a child of another PublicKey message.
optional bytes public_key; // Load, not required if included as child in other PublicKey message
optional bytes chain_code; // Load, Specified for BIP 32 addresses

optional int32 i; // Load, Specified for BIP 32 addresses

optional int32 i_look_ahead; // Load, Specified for BIP 32 addresses

repeated PublicKey children; // Load, Only used for BIP 32 addresses, this field is not accepted to set notifications.

repeated Transaction tx_received;
repeated Transaction tx_sent;

repeated Transaction tx_received_defunct;
repeated Transaction tx_sent_defunct;
// Defunct list is mutually exclusive with regular list of transactions. If a transaction is made invalid (e.g. double spend) a notification will be sent with that transaction in the defunct list to signify it has been removed from the regular list.

// Contains total balance from all transactions including child transactions
optional int64 balance;
optional int64 received;

optional int32 confirmations; // Minimum confirmations of all transactions including children

optional int32 scan_progress;
}

message scriptPublicKey {
optional bytes script;
optional bytes script_hash;

repeated Transaction tx_received;
repeated Transaction tx_sent;

repeated Transaction tx_received_defunct;
repeated Transaction tx_sent_defunct;
// Defunct list is mutually exclusive with regular list of transactions. If a transaction is made invalid (e.g. double spend) a notification will be sent with that transaction in the defunct list to signify it has been removed from the regular list.

// Contains total balance from all transactions including child transactions
optional int64 balance;
optional int64 received;

optional int32 confirmations; // Minimum confirmations of all transactions including children

optional int32 scan_progress;
}

message Transaction {
optional bytes raw_transaction;

// Identify transaction
optional bytes txid; // Load, used to allow daemon to quickly find relevant transactions when loading context
optional bytes block_hash; // Load
optional int32 transaction_no; // Load, Transaction location in block

optional int32 confirmations;

optional int64 amount;
optional int64 fee;
}

message BlockChain {
optional uint32 height_from_peers; // Best block height based on response from peers
optional uint32 backtrack; // Used to fetch a specified number of blocks from the main chain to the current best block
repeated Block main_chain;
repeated Block orphaned;
// main_chain and orphaned is mutually exclusive so adding the same block to one implies it is not included in the other
}

message Block {
optional bytes raw_block;
optional uint32 height
optional byte hash;
optional byte previous;
optional byte root;

// More block info
}

message PeerInfo {
optional int32 count;
Peer {
optional bytes address;
optional int32 services;
optional int64 lastsend;
optional int64 lastrecv;
optional int64 conntime;
optional int32 version;
optional string subver;
optional bool inbound;
optional int64 releasetime;
optional int32 startingheight;
optional int32 banscore;
}
repeated Peer peers;
}

Examples

Setup context for a set of keys to match a current wallet implementation

Send the follow message to AddToContext RPC.

Code:

Context: {
context_id: Unique id for wallet possibly first key in pool
keys: [{
public_key: Public key matching wallet private key
}] // Repeat for each key in the wallet
}

Setup server to send notifications to the client by sending the following message to the NotifyContext RPC.
The presence of any value in a field signifies a notification is desired.

Code:

Notification {
global: {
peers: {
count: 1 // Receive notification anytime the number of peers changes
}
chain: {
height_from_peers: 1 // Receive notification of the expected newest block high in the chain
main_chain: {
height: 1 // Receive notification of the newest block height after that new block is accepted
}
}
}
contexts: [{
context_id: same context from AddToContext
keys: [
{
public_key: 1 // Include public key so the notification can be identified
tx_in: {
raw_transaction: 1 // Send full transaction when its output matches the public_key
// Could receive notification for other fields describing the transaction.
}
tx_out: {
raw_transaction: 1 // Send full transaction when its input matches the public_key
}
}]
}

Typical notification message would look like

Code:

Notification {
global: {
peers: {
count: 16 // Number of peers changed
}
}
contexts: [{
context_id: same context from AddToContext
keys: [
{
public_key:
tx_in: [{
raw_transaction: raw transaction data
}]
tx_out: [{
raw_transaction: raw transaction data
}]
}]
}]
}

Notifications on new best blocks, similar functionality to -blocknotify

Set the following notifications with SetNotify

Code:

Notification {
global: {
blocks: {
main_chain: {
hash: 1
}
}
}
}

Notifications will be sent like below for each new best block

Code:

Notification {
global {
blocks: {
main_chain: {
hash: Best block hash
}
}
}
}

Ecommerce site using Hierarchical Deterministic Wallets without any private keys. Only needs to know when coins are sent to specific accounts and how many confirmations.

Send the follow message to AddToContext RPC.

Code:

Context: {
context_id: Unique id for wallet possibly first key in pool
keys: [{
public_key: master node key
chain_code: chain code
children: [{
i: 0 // Look at the 0 chain
i_look_ahead: 0 // Don't look for any other keys at this level
children: [{
i: 0 // External Chain by BIP 32
children: [{
i: 0
i_look_ahead: 2
}]
}, {
i: 1 // Internal Chain by BIP 32
children: [{
i: 0
i_look_ahead: 2
}]
}]
}]
}] // Repeat for each key in the wallet
}

The 0 chain key could have been used in place of the master node key instead of deriving it from the master node key

Set notifications with SetNotify

Code:

Notification {
contexts: [{
context_id: same context from AddToContext
keys: {
i: 1

// We are only interested in the amount received for each key, not the full transactions
received: 1
confirmations: 1
}
}]
}

Typical notification message would look like

Code:

Notification {
contexts: [{
context_id: same context from AddToContext
keys: [
{
children: [{
i: 0 // Internal chain
received: 100000000 // One bitcoin received.
confirmations: 0 // New notification will be sent each time confirmations changes
children: [{
i: 0 // Transaction on first address on Internal chain
received: 100000000 // One bitcoin received.
confirmations: 0
}]
}]
}]
}

bitcoin library
I think it would be advantageous to develop a bitcoin library that uses an interface to the daemon like this in addition to providing tools to manage the private keys either through a local file store or another RPC interface to a private key daemon (TPM secured process, specialized hardware). Ideally this library would be the preferred interface to bitcoin.

0mq

This could be implemented over 0mq. It isn't evident to me if there are real world use cases to broadcast (PUB-SUB) notifications for specific contexts (small number of specific transactions). I see there is a 0mq pull request that publishes blocks, transactions and new host notifications and uses REQ-REP to provide access to the JSON-RPC. An ideal 0mq implementation would probably be capable of sending the same context notification messages to multiple hosts but would this ever be needed?

My current inclination is to implement this with 0mq and google protocol buffers. The result would be harder to understand on the wire than something like JSON-RPC but would be a little easier to implement and run faster. I would expect such an implementation would scale to anything needed. That said, authentication and encryption would be more important than just about anything offered by 0mq alone and will be a easier with an openssl secured TCP socket just like the current JSON-RPC.

Topic: Proposal: New RPC interface for bitcoind (Read 1381 times)