Topic: Stratum protocol documentation (Read 93202 times)

{"id": 0, "method": "client.reconnect", "params":["stratum-lb-usa48.btcguild.com",3333,0]} 

{"id": 0, "method": "mining.get_transactions", "params":["545198de00000000"]}

 Summary

The Stratum protocol reduces client-server communications to levels that are usable with very low bandwidth, and reduces the strain on servers drastically.
The key concept behind Stratum based mining is "push" based work, where the server pushes work to the miner, and the miner is able to utilize that work until the next push, regardless of the hash rate.
This is done by allowing the miner to increase a counter in the coinbase transaction, and build a new merkleroot for the block header, which effectively means the miner generates new work continuously without contacting the server.
Additionally, this new protocol utilizes a single asynchronous socket connection for all communication, rather than opening/closing HTTP connections with each communication.



Building Local Work

The mining pool provides a miner with two pieces of information upon connecting. The first is ExtraNonce1. The second piece of information is ExtraNonce2_size. This is how many bytes should be used for the ExtraNonce2 counter. ExtraNonce2 is a hexadecimal counter, and should be padded to fill up the number of bytes identified as the ExtraNonce2_size.

When the pool pushes work, it provides the coinbase transaction in two pieces. When the miner is building the block header to hash, it builds a merkleroot by creating a coinbase transaction. This is done by combining: Coinbase1 + ExtraNonce1 + ExtraNonce2 (padded) + Coinbase2.

This transaction is then passed through a double SHA256 to get the coinbase tx hash. You then combine that with first merkle branch provided by the work (if any), and double SHA256 the combined string. You then take that result, and do the same with the next merkle branch, repeating this process until all merkle branch hashes have been combined with previous results.

This process creates a unique merkle root for the new block header, which can then be pushed through the miner using the standard 32-bit nonce counter.



Initial Connection Communication

Client: {"id": 1, "method": "mining.subscribe", "params": []}\n
Server:  {"id": 1, "result": [["mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f"], "08000002", 4], "error": null}\n

Connection uses 'mining.subscribe' to subscribe to work from the server.
Server result response contains:
result[0] = "mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f" - Unique string used for the subscription
result[1] = ExtraNonce1, used for building the coinbase.
result[2] = Extranonce2_size, the number of bytes that the miner users for its ExtraNonce2 counter



Server Work Communication

Server: {"params": ["bf", "4d16b6f85af6e2198f44ae2a6de67f78487ae5611b77c6c0440b921e00000000",
"01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff20020862062f503253482f04b8864e5008",
"072f736c7573682f000000000100f2052a010000001976a914d23fcdf86f7e756a64a7a9688ef9903327048ed988ac00000000",
["c5bd77249e27c2d3a3602dd35c3364a7983900b64a34644d03b930bfdb19c0e5","049b4e78e2d0b24f7c6a2856aa7b41811ed961ee52ae75527df9e80043fd2f12"],
"00000002", "1c2ac4af", "504e86b9", false], "id": null, "method": "mining.notify"}\n

This is sent at regular intervals by the server. It should be sent almost immediately after an acknowledge subscription.
Work pushes contain:
params[0] = Job ID. This is included when miners submit a results so work can be matched with proper transactions.
params[1] = Hash of previous block. Used to build the header.
params[2] = Coinbase (part 1). The miner inserts ExtraNonce1 and ExtraNonce2 after this section of the coinbase.
params[3] = Coinbase (part 2). The miner appends this after the first part of the coinbase and the two ExtraNonce values.
params[4][] = List of merkle branches. The coinbase transaction is hashed against the merkle branches to build the final merkle root.
params[5] = Bitcoin block version, used in the block header.
params[6] = nBit, the encoded network difficulty. Used in the block header.
params[7] = nTime, the current time. nTime rolling should be supported, but should not increase faster than actual time.
params[8] = Clean Jobs. If true, miners should abort their current work and immediately use the new job. If false, they can still use the current job, but should move to the new one after exhausting the current nonce range.



Worker Authorization

Client: {"params": ["eleuthria_miner1", "password"], "id": 2, "method": "mining.authorize"}\n
Server: {"error": null, "id": 2, "result": true}\n

Client authorizes a worker using the method "mining.authorize". Client request contains:
params[0] = Worker Name
params[1] = Worker Password (can be "" or omitted if server does not require passwords)

Server response is "result": true for successful authorization, "result": false for unsuccessful authorization.



Work Submissions

Client: {"params": ["eleuthria_miner1", "bf", "00000001", "504e86ed", "b2957c02"], "id": 4, "method": "mining.submit"}\n
Server: {"error": null, "id": 4, "result": true}\n

Miners submit shares using the method "mining.submit".
Client submissions contain:
params[0] = Worker Name
params[1] = Job ID
params[2] = ExtraNonce 2
params[3] = nTime
params[4] = nonce
Server response is result: true for accepted, false for rejected.



Difficulty Adjustment

Server: { "id": null, "method": "mining.set_difficulty", "params": [2]}\n

The server can adjust the difficulty required for miner shares with the "mining.set_difficulty" method.
The miner should begin enforcing the new difficulty on the next job received. Some pools may force a new job out when set_difficulty is sent, using clean_jobs to force the miner to begin using the new difficulty immediately.



Cheat Sheet

Methods:
mining.subscribe     : Used to subscribe to work from a server, required before all other communication.
mining.authorize     : Used to authorize a worker, required before any shares can be submitted.
mining.notify        : Used to push new work to the miner.  Previous work should be aborted if Clean Jobs = true!
mining.submit        : Used to submit shares
mining.set_difficulty: Used to signal the miner to stop submitting shares under the new difficulty.

For mining software developers

Stratum protocol is based on JSON-RPC 2.0. In this chapter I expect that you're familiar with this protocol and you understand terms like "request", "response" and "notification". Please read JSON-RPC specification for more details.

For high level image of the Stratum protocol concept, please read Stratum protocol specification on Google docs. This document needs some care, but give you the basic examples how to connect to Stratum server.
Exception handling

Stratum defines simple exception handling. Example of rejected share looks like:

{"id": 10, "result": null, "error": (21, "Job not found", null)}

Where error field is defined as (error_code, human_readable_message, traceback). Traceback may contain additional information for debugging errors.

Proposed error codes for mining service are:

    20 - Other/Unknown
    21 - Job not found (=stale)
    22 - Duplicate share
    23 - Low difficulty share
    24 - Unauthorized worker
    25 - Not subscribed

Real-world example

This chapter contains real log of miner-pool communication which solved testnet3 block 000000002076870fe65a2b6eeed84fa892c0db924f1482243a6247d931dcab32
Miner connects the server

On the beginning of the session, client subscribes current connection for receiving mining jobs:

{"id": 1, "method": "mining.subscribe", "params": []}\n
{"id": 1, "result": [[["mining.set_difficulty", "b4b6693b72a50c7116db18d6497cac52"], ["mining.notify", "ae6812eb4cd7735a302a8a9dd95cf71f"]], "08000002", 4], "error": null}\n

Reminder: The newline character \n is a part of the message and must be added to the end of *every* JSON message. Server may wait to this magic character to start processing the message. This is the most common mistake which people implementing line-based clients do!

The result contains three items:

    Subscriptions details - 2-tuple with name of subscribed notification and subscription ID. Teoretically it may be used for unsubscribing, but obviously miners won't use it.
    Extranonce1 - Hex-encoded, per-connection unique string which will be used for coinbase serialization later. Keep it safe!
    Extranonce2_size - Represents expected length of extranonce2 which will be generated by the miner.

Authorize workers

Now let authorize some workers. You can authorize as many workers as you wish and at any time during the session. In this way, you can handle big basement of independent mining rigs just by one Stratum connection.

{"params": ["slush.miner1", "password"], "id": 2, "method": "mining.authorize"}\n
{"error": null, "id": 2, "result": true}\n

Server start sending notifications with mining jobs

Server sends one job *almost* instantly after the subscription.

Small engineering note: There's a good reason why first job is not included directly in subscription response - miner will need to handle one response type in two different way; firstly as a subscription response and then as a standalone notification. Hook job processing just to JSON-RPC notification sounds a bit better to me.

{"params": ["bf", "4d16b6f85af6e2198f44ae2a6de67f78487ae5611b77c6c0440b921e00000000",
"01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff20020862062f503253482f04b8864e5008",
"072f736c7573682f000000000100f2052a010000001976a914d23fcdf86f7e756a64a7a9688ef9903327048ed988ac00000000", [],
"00000002", "1c2ac4af", "504e86b9", false], "id": null, "method": "mining.notify"}

Now we finally have some interesting stuff here! I'll descibe every field of the notification in the particular order:

    job_id - ID of the job. Use this ID while submitting share generated from this job.
    prevhash - Hash of previous block.
    coinb1 - Initial part of coinbase transaction.
    coinb2 - Final part of coinbase transaction.
    merkle_branch - List of hashes, will be used for calculation of merkle root. This is not a list of all transactions, it only contains prepared hashes of steps of merkle tree algorithm. Please read some materials for understanding how merkle trees calculation works. Unfortunately this example don't have any step hashes included, my bad!
    version - Bitcoin block version.
    nbits - Encoded current network difficulty
    ntime - Current ntime/
    clean_jobs - When true, server indicates that submitting shares from previous jobs don't have a sense and such shares will be rejected. When this flag is set, miner should also drop all previous jobs, so job_ids can be eventually rotated.

How to build coinbase transaction

Now miner received all data required to serialize coinbase transaction: Coinb1, Extranonce1, Extranonce2_size and Coinb2. Firstly we need to generate Extranonce2 (must be unique for given job_id!). Extranonce2_size tell us expected length of binary structure. Just be absolutely sure that your extranonce2 generator always produces extranonce2 with correct length! For example my pool implementation sets extranonce2_size=4, which mean this is valid Extranonce2 (in hex): 00000000.

To produce coinbase, we just concatenate Coinb1 + Extranonce1 + Extranonce2 + Coinb2 together. That's all!

For following calculations we have to produce double-sha256 hash of given coinbase. In following snippets I'm using Python, but I'm sure you'll understand the concept even if you're a rubyist! It is as simple as:

import hashlib
import binascii
coinbase_hash_bin = hashlib.sha256(hashlib.sha256(binascii.unhexlify(coinbase)).digest()).digest()

How to build merkle root

Following Python snippet will generate merkle root for you. Use merkle_branch from broadcast and coinbase_hash_bin from previous snippet as an input:

import binascii

def build_merkle_root(self, merkle_branch, coinbase_hash_bin):
    merkle_root = coinbase_hash_bin
    for h in self.merkle_branch:
        merkle_root = doublesha(merkle_root + binascii.unhexlify(h))
    return binascii.hexlify(merkle_root)

How to build block header?

Now we're almost done! We have to put all together to produce block header for hashing:

version + prevhash + merkle_root + ntime + nbits + '00000000' +
'000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000'

First zeroes are blank nonce, the rest is padding to uint512 and it is always the same.

Note that merkle_root must be in reversed byte order. If you're a miner developer, you already have util methods there for doing it. For some example in Python see Stratum mining proxy source codes.
Server can occasionally ask miner to change share difficulty

Default share difficulty is 1 (big-endian target for difficulty 1 is 0x00000000ffff0000000000000000000000000000000000000000000000000000), but server can ask you anytime during the session to change it:
{ "id": null, "method": "mining.set_difficulty", "params": [2]}

This means that difficulty 2 will be applied to every next job received from the server.
How to submit share?

When miner find the job which meets requested difficulty, it can submit share to the server:

{"params": ["slush.miner1", "bf", "00000001", "504e86ed", "b2957c02"], "id": 4, "method": "mining.submit"}
{"error": null, "id": 4, "result": true}

Values in particular order: worker_name (previously authorized!), job_id, extranonce2, ntime, nonce.

{"id": 2, "method": "mining.suggest_difficulty", "params": [42]}

Network protocol specification
Stratum platform


draft


________________


Overview
        Stratum is basically a platform for creating lightweight Bitcoin clients (= clients without a blockchain, keeping only private keys). Absence of the blockchain on the client side provide very good user experience, client has very small footprint and still provide high level of security and privacy.
        More technically, Stratum is an overlay network on the top of Bitcoin P2P protocol, creating simplified facade for lightweight clients and hiding unnecessary complexity of decentralized protocol. However there’s much bigger potential in this overlay network than just providing simplified API for accessing blockchain stored on Stratum servers. For utilization of such potential, we definitely need some robust protocol providing enough flexibility for various type of clients and their purposes.


Some advanced ideas for Stratum network, which will need flexible network protocol:
* Integration of BTC/fiat exchanges into clients
* Wallet storages for diskless or extremely low-resource clients (AVR-based hardware wallets)
* Server-side escrows (sending bitcoins to email)
* Integration of bitcoin laundry
* Exchange calculators (for providing “fiat” equivalents of BTC in clients)
* Firstbits support
* Mining support for clients
* Various transport protocols (especially HTTP Push, which allows PHP websites to integrate with Bitcoin easily)


________________
Requirements
1. Protocol should be as simple as possible. Some clients aren’t capable to handle high-level message systems like Google’s Protocol buffers or Apache Thrift.
2. Protocol should be text-based. Some languages cannot handle binary data (javascript) or it’s pretty difficult to implement it correctly (PHP). It’s also easier to debug text-based protocol than binary data.
3. Protocol must support standard RPC mechanism (request-response). Request should contains method identifier and parameters, response should be able to transfer error states/exceptions.
4. Mapping between request and response must be clear. Don’t allow vague relation between request and response. Some message-based systems use only textual conventions to map requests and responses together, like ‘firstbits_resolve ’ expects ‘firstbits_response  
’ or ‘firstbits_error  ’. It creates ambiguous data flow, avoid it.
5. Protocol should support publish-subscribe mechanism. Client can subscribe on server for receiving some kind of information. After this request, server will proactively broadcast messages to subscribed clients until client disconnects or cancel it’s subscription.
6. Protocol must be bi-directional. In the opposite of standard client-server model, we sometimes need to allow server to initiate the communication. As an example, server can ask client to reconnect to another node (before switching to maintenance mode) or send some text message to user.


________________


Network layers
        Stratum network layer should be flexible on many levels. This document is proposing complete stack for Stratum networking, however such complex problem can be divided into three independent layers:


1. Transports
2. Application protocol
3. Services
Transports
        Transports are the way how to obtain bi-directional communication between a lightweight client and Stratum servers. All Stratum servers should support some basic range of supported transports like plain socket transport, HTTP Poll (for environments with restrictive network access), HTTP Push (for non-interactive clients like PHP scripts), websocket/socket.io (for javascript clients) etc.
        Transport is responsible for keeping the session, it’s not a responsibility of any upper layer. It means that once any kind of transport is established, upper layers of protocol can send/receive messages like it’s transferred over already established TCP socket.
 
* Socket transport
   * Standard TCP socket, clients initiate connection, then both sides can initiate transport of payload (application protocol) anytime.
* Websocket
   * It’s HTTP-based, socket-like transport, widely supported by javascript clients. It’s probably the best way how to provide instant notification to browser-based applications.
   * After initial handshake, it works like “Socket transport”.
* HTTP Poll
   * Client performs HTTP POST every N seconds, storing payload as request body, processing server payload from response body.
   * HTTP POST without a session identifier creates new session (cookie?), client is responsible for using this cookie in following requests during the session.
   * Server store all messages for delivery to client into server-side buffer, flush them to client in following poll request.
* HTTP Push
   * Transport for clients which cannot keep open connection and they’re unable to ask server frequently enough. Typical example is PHP-based website, which want to subscribe for some blockchain changes (notification about incoming transaction for given address).
   * On the beginning of the HTTP Poll session, client provide callback URL, turning the current HTTP Poll transport into HTTP Push.
   * When there’re some new data on Stratum server, server performs HTTP POST request to callback URL with payload in the request body.
   * Client must perform keep-alive HTTP Poll request for indicating it’s still alive and want to receive HTTP Push messages. It can be done by crontab entry with one hour period, for example.


HTTP Poll
Important HTTP headers in the request:
* Cookie should contains all cookies provided by the server in previous calls, especially the STRATUM_SESSION cookie which is identifying current HTTP session. Blank or missing STRATUM_SESSION will initiate new session, even on the same TCP connection. Using many different cookies in different HTTP requests is possible, which allow serving more independent clients over one TCP connection.
* Content-Type must be always “application/stratum” indicating that payload is following Stratum protocol. This is especially useful for filtering requests in the server configuration (Apache/Nginx), so ports 80 and 443 can still be used for non-Stratum traffic.
* X-Callback-Url Provides the URL (always in the full form, including desired HTTP/HTTPS protocol type) for HTTP Push transport. Occurence of this header in the request turns the session into the HTTP Push protocol.


Important HTTP headers in the response:
* Set-Cookie contains cookies identifying current HTTP session and client must send back all those cookies in following request. By sending the STRATUM_SESSION cookie, server is indicating that the request/response is part of newly created session. When the client receive this cookie in the middle of the communication, he must re-initialize the session (subscribe for all services, …). This may happen when client perform request with the session which is already expired.
* Content-Type is always “application/stratum”, indicating that response was created by Stratum application server.
* Content-MD5 contains the MD5 hash of the body, for checking that content wasn’t corrupted during the transmission.
* Server indicates the version of server implementation. 
* X-Session-Timeout contains lifetime of current session (in seconds). After this time, every consequential request with the same session ID is considered as new session and client must perform initial handshake (subscribing for all services, …)
* X-Content-SHA256 has the same purpose as Content-MD5, but provides checksum for clients which don’t handle MD5. SHA256 must be presented in the clients anyway for creating Bitcoin transactions, so the possibility of checking transport checksum with SHA256 is minimizing dependencies.


Important HTTP status codes in the response:
* 200 OK - server processed the request succesfully
* 5xx Server error - client should perform consequential request with some delay to prevent server overloading.


HTTP Push
        This transport works on the top of HTTP Poll transport and it is fully backward compatible. It is enabled by sending X-Callback-Url header in HTTP Poll request, indicating that server may actively send payload to the client using given URL as HTTP POST request.
Client can continue to work with the transport like it’s still the HTTP Poll transport, but broadcasts from the server are delivered instantly over callback URL, without waiting on consequential poll request.
Callback URL can be changed during the session, by providing updated URL in the header. HTTP Push can be also switched back to HTTP Poll, by providing X-Callback-Url with blank string value.
Client must perform occasional HTTP polling even with HTTP Push transport, to indicate he’s alive and interested in receiving callbacks. Don’t forget that those keep alive calls are still valid HTTP Poll requests and server can provide real data in the response.
Keep alive poll requests must be called before session on the server timed out (timeout is provided in X-Session-Timeout response header). It’s recommended to perform keep alive request few seconds before session will actually timed out to protect session before expiration thanks to temporary network failure. Don’t forget that occurence of STRATUM_SESSION cookie in the keep alive response indicates expiration of previous session on the server and client must perform re-initialization all previous subscriptions.


Important HTTP headers in callback:
* Content-Type is always “application/stratum”, indicating that request was created by Stratum application server.
* X-Session-Id contains session ID of HTTP Poll connection. This is useful for handling multiple connections over the same callback URL.
* X-Session-Timeout contains remaining time to session expiration in seconds.
Application protocol
        Protocol itself is the way how to exchange information between client and server, using already established transport. Protocol doesn’t understand the meaning of exchanged information, it only keeps the track on request-response and internal data format. Protocol format is the same for all supported transports.
        Basic structure of proposed protocol is line-based, json-encoded message. Every message is on separate line and there exists only two formats of messages - request and response. Those messages use the format of JSON-RPC 2.0 protocol (http://json-rpc.org/wiki/specification).
        JSON-RPC allows two types of requests. One is part of standard RPC mechanism when every request expects some response. Second type is the notification formatted as a JSON-RPC request, but it doesn’t expect any kind of response. Typical example is broadcasting of new block or transaction from server to clients. The difference between RPC request and notification is that notification always has id=null.
Request
Every RPC request contains three parts:
   * message ID - integer or string
   * remote method - unicode string
   * parameters - list of parameters


message ID must be an unique identifier of request during current transport session. It may be integer or some unique string, like UUID. ID must be unique only from one side (it means, both server and clients can initiate request with id “1”). Client or server can choose string/UUID identifier for example in the case when standard “atomic” counter isn’t available (multi-processing environment like PHP servers).


Examples:


* Retrieve transaction history of given Bitcoin address (client->server):
{“id”: 1, “method”: “blockchain.address.get_history”, “params”: [“1DiiVSnksihdpdP1Pex7jghMAZffZiBY9q”]}


* Broadcast transaction to Bitcoin network (client->server):
{“id”: 2, “method”: “blockchain.transaction.broadcast”, “params”: [“tx_payload”]}


* Subscribe for receiving information about new blocks in blockchain (client->server):
{“id”: 3, “method”: “blockchain.block.subscribe”, “params”: []}


* Unsubscribe from receiving information about new blocks in blockchain (client->server):
{“id”: 4, “method”: “blockchain.block.unsubscribe”, “params”: []}


Notification
Notification is like Request, but it does not expect any response and message ID is always null:
   * message ID - null
   * remote method - unicode string
   * parameters - list of parameters




Examples:
* Broadcast message about new block (server->client). Server don’t expect any response to this message. Client must perform call “blockchain.block.subscribe” to start receiving requests like this:
{“id”: null, “method”: “blockchain.block.new”, “params”: [“block_payload”]}


Response
        Every response contains following parts
   * message ID - same ID as in request, for pairing request-response together
   * result - any json-encoded result object (number, string, list, array, …)
   * error - null or list (error code, error message)


Examples:
* Received transaction history of an address (server->client):
{“id”: 1, result: [“tx1_id”: {}, “tx2_id”: {}], error: null}


* Received resolved firstbits address (server->client):
{“id”: 2, result: [“1DiiVSnksihdpdP1Pex7jghMAZffZiBY9q”], “error”: null}


* Resolving of firstbits address failed (server->client):
{“id”: 2, result: null, “error”: (1, “Firstbits cannot be translated to valid Bitcoin address”)}


Message signatures
#TODO
sign, sign_type, sign_id, sign_time
Replay attacks
#FIXME: Timestamp IS a part of signed data, it’s up to client to decide if cached message is good enough.
To protect messages against replay attacks, RPC commands vulnerable to such attack should contain some unique parameter. Because every signature is constructed from both request and response data, current timestamp in the request should provide enough security.
        Timestamp is a part of signature metadata. Although some types of RPC calls are immutable by design and adding timestamp to signature doesn’t provide additional security against replay attack, it’s up to client to decide if cached response (with timestamp in the history) is good enough for him. Caching of signatures is a reasonable way to improve server load and using signature with older timestamp shouldn’t be an issue for immutable calls.
Exceptions
Every RPC error returns 3-tuple with error code, human-readable error message and detailed traceback of exception (may be just blank string on production servers for security reasons). Error codes < 0 are protocol specific or unhandled exceptions. Error codes > 0 are exceptions generated by services and every service should offer detailed description of possible error states.


-1, Unknown exception, error message should contain more specific description
-2, “Service not found”
-3, “Method not found”
-10, “Fee required”
-20, “Signature required”, when server expects request to be signed
-21, “Signature unavailable”, when server rejects to sign response
-22, “Unknown signature type”, when server doesn’t understand any signature type from “sign_type”
-23, “Bad signature”, signature doesn’t match source data


#TODO
Services
#TODO Documentation for blockchain-related services (for Electrum)
Server side:


discovery.list_services()
discovery.list_vendors(service_name)
discovery.list_methods(service_name, vendor)
discovery.list_params(service_name, vendor, method)


node.peers.subscribe()
node.stop()
node.get_signature_pubkey()


# TODO:
blockchain.block.subscribe
blockchain.block.unsubscribe
blockchain.block.broadcast


blockchain.address.subscribe
blockchain.address.unsubscribe
blockchain.address.get_history
blockchain.address.get_balance


blockchain.transaction.broadcast
blockchain.transaction.get


blockchain.transactions.guess_fee
blockchain.transactions.subscribe
blockchain.transactions.unsubscribe


firstbits.resolve
firstbits.lookup
Client side:


node.get_version
node.reconnect(hostname)


Notifications:
blockchain.block.notify()
blockchain.transaction.notify()