Pages:
Author

Topic: [Stratum] Overlay network protocol over Bitcoin - page 3. (Read 37882 times)

legendary
Activity: 1386
Merit: 1097
Check out my answer here on why The Stratum protocol is secure.

Very nice, thank you :-).

I didn't commited into git for few days, because I'm finishing one project in my job, but then I'll work more actively on the Stratum again. I'm missing it!
legendary
Activity: 1358
Merit: 1003
Ron Gross
Check out my answer here on why The Stratum protocol is secure.
legendary
Activity: 1386
Merit: 1097
It's actually hosted by BitVPS.COM not .Net....

Fixed, thanks Smiley
sr. member
Activity: 419
Merit: 250
It's actually hosted by BitVPS.COM not .Net....

legendary
Activity: 1358
Merit: 1003
Ron Gross
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history¶ms=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.

Hmm, I'll fix and retry ... possibly next weekend.
legendary
Activity: 1386
Merit: 1097
Any documentation on what is and is not implemented right now?

get_history() works now, however the result data may be slightly changed in the future. I'm writing API for standard services, then will be much cleaner what method expect what parameters and what will be the result object.
legendary
Activity: 1386
Merit: 1097
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history¶ms=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA

Hi, your payload is wrong, you must use json-encoded data. So

{"id":1, "method":"blockchain.address.get_history", "params": ["14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA"]}

instead of url-encoded data.
legendary
Activity: 1358
Merit: 1003
Ron Gross
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Any documentation on what is and is not implemented right now?
ETAs?
legendary
Activity: 1358
Merit: 1003
Ron Gross
This also happens when I try a prodnet address:

POST / HTTP/1.1
Host: chicago.stratum.bitcoin.cz:8000
Content-Type: application/stratum
Connection: keep-alive
Accept: */*
User-Agent: NING/1.0
Content-Length: 84

id=1&method=blockchain.address.get_history¶ms=14obksEpeUTAKsdKsw3vTSUoiYeK9S3thA
legendary
Activity: 1358
Merit: 1003
Ron Gross
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?

What's the expected time that requests are supposed to take?

I sent a request to http://chicago.stratum.bitcoin.cz:8000, with a timeout of 60 seconds ... and the timeout expired.

content-type: application/stratum
id: 1
method: blockchain.address.get_history
params: mw5h6BDh77GUPc4DCmzjoU7ry2R4exoaYr   (this is a testnet address, so I guess it wouldn't work ... but I should get an error, not a timeout)
legendary
Activity: 1358
Merit: 1003
Ron Gross
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...

Another question - I haven't seen any reference to testnet in this thread ... are there testnet servers? Does the protocol support it?
legendary
Activity: 1386
Merit: 1097
Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

Assuming that such services are already implemented on the Stratum nodes (which isn't yet) - yes, it will work. But you're doing exactly the case b) from my previous post, so you don't need to use get_history at all, just subscribe the address in the time when you send it to the user.

But I understand you're mostly playing with the stuff so I'm ok with it...
legendary
Activity: 1358
Merit: 1003
Ron Gross
Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..


For context, this is the application I'm developing. I wanted to implement a polling client at first, that polls every minute or so, just as an initial proof of concept - I'd like to see the protocol works. I intend to move to a more efficient implementation soon after that.

Let me try to rephrase. Assuming servers don't rate-limit just yet - if I don't send any cookies, and just poll every minute on address.get_history, and once in a while do transaction.broadcast ... will this work?

I really would like to get all transactions on these addresses. The addresses should in theory only have 1-2 transactions on each of them, but I'd still like to know all the transactions history.

This is not the final implementation, just a temporary POC.
legendary
Activity: 1386
Merit: 1097
The first one is a meta question - how about using Shapado as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Yes, some better tools should be better in the future, but at this moment the forum thread is doing well. There is only few people so the discussion is pretty clean now. Especially I'll try to move that google docs to some wiki which allow me to separate whole document into more parts and include some additional info (as an examples).

Quote
The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?

What's your purpose of such calls? You probably don't want to download whole address history on every request (say 5 seconds), right? I'm not sure, because I don't know your application, but you probably want to download address history on the startup and then subscribe your client for changes on given address, to receive only new transactions. In this way you save server resources and bandwidth. It's also possible that server may block your requests when you start overflooding server with "get_history" requests every few seconds, because there's no reason to do that so frequently (rate limiting isn't implemented yet, but it is pretty good feature for the future).

There are mostly two main usecases:
a) The address you're interested in is already existing, so you need to ask for history + subscribe for new transactions.
b) The address is freshly generated by you, so you know that there's no history and you want ONLY to subsribe for new transactions. This is the case of some merchant tool, which created fresh address for receiving coins for the order...

case a)
first http call (one HTTP request can contain more RPC calls, divided by newline character):
blockchain.address.get_history('1SomeAddress') => list of transactions
blockchain.address.subscribe('1SomeAddress') => returns only True on a success, but server will send you data for incoming transactions in following HTTP responses.

following http calls: blank HTTP request (with the session) to the server

case b)
first HTTP call:
blockchain.address.subscribe('1SomeAddress') (no get_history() command is needed!)

following HTTP calls: blank HTTP request (with the session) to the server

I think there's big misunderstanding about the "polling" thing. Polling does not mean that the connection is stateless and you need to ask for everyting on every request. Polling mechanism is only the workaround for the situation when only the client can initiate the communication and is giving a chance to counterparty to send pending data back in the polling response (like information about incoming transaction)...

So yes, you need to "care" about the session even in your use case. But almost every http client library can do cookie management internally, when you perform more calls ("http polling requests") in the row..
legendary
Activity: 1358
Merit: 1003
Ron Gross
slush,

I'm finally starting to work on this, and have some questions.
I think I'll fire them one by one on this thread, instead of accumulating them.

The first one is a meta question - how about using Shapado as a Q&A forum? It's "setup in 8 seconds", and better than a forum thread. It might be better than just firing these questions away at the Stack Exchange. The advantage of a dedicate forum is that you (and other Stratum devs) could follow that website and quickly respond to questions, and the answers would be findable later via Google.

Another question is about a sessions while polling:

From the Google Doc:

Quote
Cookie should contains all cookies provided by the server in previous calls, especially the STRATUM_SESSION cookie which is identifying current HTTP session. Blank or missing STRATUM_SESSION will initiate new session, even on the same TCP connection

The APIs that I need right now appear to be only address.get_history and transaction.broadcast (I'll start by implementing the simple & naive polling protocol). For these purposes, I'm not sure why I need to maintain a session. As a matter of fact ... can you give an example when a session is useful?
legendary
Activity: 1386
Merit: 1097
Can you explain a bit more your decisions?

Of course. HTTP is just a transport carrying abstract Stratum payload. There is TCP transport already and there can be a lot more transports in the future. Stratum protocol is designed to be bi-directional, so server can send some payload to client (information about new block, new transaction) at any time. It's not the part of application layer itself to decide HOW transport will deliver such payload to the client. So server just tell "send to the client" and don't care about it anymore.

HTTP Poll transport is just the alternative for the environment where real bi-directional communication (like TCP socket or websocket) is not possible. So every poll request just give a chance to send pending messages from the server to the client in the HTTP response, simulating bi-directional communication, just with some delay (=polling interval).

From the view of this idea, creating RPC command for polling is going against the whole conception, because RPC commands are handled by different layer than HTTP transport mechanism. And there's nothing like 'poll' RPC command for standard bi-directional transports anyway.

Quote
Also, how would that work with TCP?

In the TCP transport, which is bi-directional, polling is not needed at all. Server will send the payload when it's ready, in real time. It's the beauty of pub-sub mechanism.
legendary
Activity: 1896
Merit: 1353
new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.


Can you explain a bit more your decisions?
Performing blank HTTP requests is an interesting idea, but I would like to know why you decided that polling should be performed that way. is this in order to minimize traffic?
Also, how would that work with TCP?
legendary
Activity: 1386
Merit: 1097
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.

What exactly are you missing? I see only three session-related calls in server.py (new_session, update_session, poll_session) which are useless in Stratum protocol.

Edit: To be more verbose:

new_session: You just don't send session cookie to the server and it will create new session for the client automatically. No RPC command needed.

update_session: Replaced with blockchain.address.subscribe()

poll_session: Perform just blank HTTP request to Stratum server, it will put pending messages to the response. No RPC command needed.
legendary
Activity: 1386
Merit: 1097
I do not know. The initial idea was to propose a new name for the communication protocol, so that it can be distinguished from the software.

Thomas, I proposed the protocol and asked you for the opinion. You told me that you don't understand and/or don't see any point in it and you want reference implementation of the protocol. So I created Stratum server implementation. What's unclear in this?

Quote
Now it seems that slush has decided to rewrite client and server from scratch, and that "Stratum" is being used to designate his software, not the protocol.

It's still Stratum protocol + server reference implementation + I'm writing some "default" services into it. Protocol implementation is already here and you know that I decided to separate Stratum from Electrum project, because I see some potential for lightweight clients generally, not only for Electrum.

But of course I'm open to help you with integrating Stratum into Electrum client, if you want. We already had the discussion about Twisted on the IRC and I accept that you don't like Twisted in the client, so I'll provide you clean-python implementation of Stratum client.
legendary
Activity: 1896
Merit: 1353
I used the rpc calls that you provided in your document where it was possible, but some calls are missing for session management.
So does this mean that electrum servers will eventually be part of the stratum network? That's what I was hoping, I just want to make sure.
I do not know. The initial idea was to propose a new name for the communication protocol, so that it can be distinguished from the software.
Now it seems that slush has decided to rewrite client and server from scratch, and that "Stratum" is being used to designate his software, not the protocol.
So I guess we might want to find another name for the protocol...
Pages:
Jump to: