[Stratum] Overlay network protocol over Bitcoin - page 7.

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

Quote from: slush on December 31, 2011, 10:22:04 AM

Quote from: ripper234 on December 31, 2011, 09:31:17 AM

You can checkout my initial skeleton of an java binding here - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!

I'll do HTTP, efficiency really isn't relevant to my use case, and leave the TCP implementation for the future.

I don't really use IRC a lot. Do you plan to have something more than a google doc page (e.g. a homepage, wiki, reddit, ...) ?

I still don't know exactly how to generate the offline TX, and haven't yet gone over your doc - sadly the work week is starting tomorrow for me (Israeli work weeks are Sunday-Thursday), and I won't have time to work on it during the week. Perhaps next weekend.

I'm really looking forward to getting an alpha implementation up & running.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: ripper234 on December 31, 2011, 09:31:17 AM

You can checkout my initial skeleton of an java binding here - subject to change, of course.

Excellent! Are you going to implement HTTP transport (push or poll?) or TCP socket transport? I encourage you to implement TCP socket because of lower overhead, although I understand that HTTP can be easier to implement.

Btw yesterday I implemented message signing and succesfully tested HTTP Poll and Push transports, although Google document isn't updated yet (thanks to 2112 suggestions, I added Content-Length, Content-MD5 and X-Content-Sha256 headers, there's also X-Push-Url for turning HTTP Poll transport into HTTP Push).

I'll try to commit all changes soon so you'll be able to test your client against working server, but I still need to cleanup the code. If you have any questions or suggestions, join #stratum channel anytime!

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

Quote from: slush on December 31, 2011, 08:44:40 AM

Quote from: ripper234 on December 30, 2011, 11:44:21 PM

(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).

BTW, you can checkout my initial skeleton of an java binding here - subject to change, of course.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: ripper234 on December 30, 2011, 11:44:21 PM

(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Don't worry, I already see there's some interest for such solution even between developers. And I'm sure that people (=users) will love solutions based on lightweight clients (and developers will be enjoying writing websites using Stratum API).

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

BTW, I posted this thread on reddit and it got downvoted for some reason.
(Some) Bitcoin lovers don't understand that for Bitcoin to succeed it can't remain just a P2P network used by hackers and computer geeks.

Red Emerald

hero member

Activity: 742

Merit: 500

Quote from: slush on December 30, 2011, 05:02:10 PM

Quote from: Red Emerald on December 30, 2011, 04:06:11 PM

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...

This looks interesting too. It doesn't seem to match some of the other specs I've seen though.

https://github.com/jchris/canonical-json#readme

slush

legendary

Activity: 1386

Merit: 1097

Btw I closed contest about name for this overlay network. I picked name "Stratum", which is latin word for "layer".

slush

legendary

Activity: 1386

Merit: 1097

Quote from: Red Emerald on December 30, 2011, 04:06:11 PM

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

Thanks for great link, it's exactly the answer what I has been searching for. It's actually very similar to the solution which I discussed with genjix on IRC, but it's nice to see that others found same solution independently. It looks that I'll choose those conventions described on stackexchange. It is slightly complicating all the signing the stuff, because it de-facto needs custom object encoder. However it is necessary to implement only in clients which require signing of proxied calls; common clients (using only blockchain service directly on overlay server) probably won't care about signing at all.

Edit: excellent, I found that "canonical JSON" is widely used term and there are some implementations in various languages already, like jsonical for pyhon (http://pypi.python.org/pypi/jsonical/). I just need to test if those implementations are compatible, but it looks like step in good direction...

Red Emerald

hero member

Activity: 742

Merit: 500

Quote from: ripper234 on December 30, 2011, 11:39:11 AM

Quote from: EhVedadoOAnonimato on December 30, 2011, 10:39:43 AM

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.

This.

I'm not interested in my thin client downloading and saving anything. I'm working on a phonegap application that is pretty much just one html file, one css file, and one javascript file. Private keys are stored encrypted in local storage (like GLBSE) or generated deterministically when the client is opened (we'll see how well js can do this /me crosses fingers). There is no room to download even a lightweight blockchain.

As far as signing, we should look at how other people have done it. OAuth alphabetizes the fields and then does some more. I don't like the idea of having to implement a "canonical_encode" but it looks like it might be necessary.

http://stackoverflow.com/questions/4670494/how-to-cryptographically-hash-a-json-object

kjj

legendary

Activity: 1302

Merit: 1026

Quote from: ripper234 on December 30, 2011, 11:41:36 AM

Quote from: kjj on December 30, 2011, 11:20:43 AM

Timestamps have their own problems. Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability. You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.

Until someone adds a new RPC call that actually is dangerous. And why not add some? The messages are signed and secure.

A system that is 90% safe is a trap waiting to spring on the first guy that forgets about the other 10%.

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

Quote from: kjj on December 30, 2011, 11:20:43 AM

Timestamps have their own problems. Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability. You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

What types of API call are vulnerable to replay anyway? Transactions can only be made once, and once they're made there's no use trying to replay them. Since a malicious server can't do anything too serious except lie in responses or delay things, a malicious traffic sniffer can't do anything worse in a replay attack.

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

Quote from: EhVedadoOAnonimato on December 30, 2011, 10:39:43 AM

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

To developers it's not convenient enough. I wanted to use bitcoinj for a java web app, but then discovered I would have to maintain the blockchain in order to use it. This means consuming a lot of traffic for a simple web app that will be idle 99.9% of the time.

Having an external API can make me practically stateless - I just have to store keys, that's all - which simplifies my life considerably.

ripper234

legendary

Activity: 1358

Merit: 1003

Ron Gross

Quote from: slush on December 30, 2011, 09:31:11 AM

Quote from: ripper234 on December 30, 2011, 07:51:21 AM

slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.

Thanks, I asked this on Stack Exchange and emailed Mike Hearn, so I imagine I'll have some direction soon enough.

kjj

legendary

Activity: 1302

Merit: 1026

Quote from: slush on December 30, 2011, 09:59:48 AM

Quote from: kjj on December 30, 2011, 09:18:16 AM

For the usual case, you don't care that multiple messages might be equally valid. The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

No, I get it. What I'm saying is that you need another function, json_encode_canonical() such that canonical_data = json_encode_canonical(json_decode(arbitrary_data)) and you need to implement it the same way on all platforms, so that when you calculate digital_signature(canonical_data) you are always doing it on the same data. The json_encode_canonical() function can even ignore fields that don't factor into the signature.

Quote from: slush on December 30, 2011, 09:59:48 AM

Quote

As for the ID, don't leave it out of the signature unless you enjoy replay attacks. Just force the client to make it globally unique. There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.

Timestamps have their own problems. Either the clocks need to be synchronized very well, or you need a window of validity, but that window of validity is also a window of vulnerability. You'd think that synchronizing clocks would be an easy assumption, but it turns out not to be at all.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: EhVedadoOAnonimato on December 30, 2011, 10:39:43 AM

I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

I have experience with BitcoinJ client on my android phone. Yes, it's better than downloading whole blockchain, that's without a discussion. But it's not *so* good as BitcoinSpinner client, which is lightweight. As and example, in bitcoinj I still need to wait to synchronize block headers, which is especially annoying when I'm using this wallet only time by time for face2face transactions. And lightweight clients cleanly wins in such use cases...

Personally I love freedom of choice, when everybody can choose solution which fits his needs. And I see that infrastructure for lightweight and ultra-lightweight clients is still weak, so I'm trying to fix it. Then integration into existing applications will become extremely easy, because those applications won't need to care about blockchain at all, just use some internet service with standard API which will *completely* mask the complex world of bitcoin.

EhVedadoOAnonimato

hero member

Activity: 630

Merit: 500

Quote from: slush on December 30, 2011, 09:40:26 AM

Quote from: EhVedadoOAnonimato on December 30, 2011, 08:40:28 AM

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

Fair enough. I just want to emphasize again that you can have speed and convenience with a lightweight blockchain. Just see BitcoinJ, it runs even on mobile phones.

Quote from: slush on December 30, 2011, 09:40:26 AM

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.

True, not everything can be done on p2p. But why those different services need to be done by the same entity anyway? You could plug your MultiBit client on that escrow.

Anyway, I don't want to sound like a naysayer. You're all doing a great work and not asking anything in return. That's noble, my sincere congratulations.
I just like the lightweight p2p solution better, that's all.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: kjj on December 30, 2011, 09:18:16 AM

For the usual case, you don't care that multiple messages might be equally valid. The signature is for "this message", not for "any message bearing the same content as this message".

I probably failed I expressed my worries. So I'll try again.

Main benefit of JSON is that it can be serialized/deserialized in almost every language in transparent way, just obj = json_decode(ser_data). However, those json_encode/decode implementations don't care about order of variables in serialized data, because order isn't a part of json specification.

So json_encode(json_decode(data)) can provide different stream than original data, which is necessary for validating json message. Thanks to this issue, you need to fiddle with serialized data directly as with text stream and remove signature-related data from this stream, instead of deserializing message, removing signature-related data, serializing it back and checking signature.

Quote

As for the ID, don't leave it out of the signature unless you enjoy replay attacks. Just force the client to make it globally unique. There are already well established techniques for doing this.

As a prevention for replay attacks I'm proposing "timestamp" field. It's also the reason why I think that request should be contained inside the signature in some form.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: EhVedadoOAnonimato on December 30, 2011, 08:40:28 AM

But one thing that I don't understand is why are people so in a rush to implement client-server solutions, when a p2p lightweight solution is perfectly possible, and we already have BitcoinJ to demonstrate.

Because both technologies solve different problem. Bitcoin p2p is distributed database, however I'm trying to create protocol for *services* on top of this distributed database.

If you want privacy and security, you can every time use blockchain. If you need speed and convenience, use some overlay network, which masks compexity for you and provide only necessary service. It's giving freedom of choice, obviously.

By the way, not all services offered by overlay must be related to blockchain and they cannot be provided in decentralized way. For example you probably want to pick some escrow service and use it's API directly or over trusted channel, not using some anonymous peer in p2p network.

slush

legendary

Activity: 1386

Merit: 1097

Quote from: ripper234 on December 30, 2011, 07:51:21 AM

slush, how would the transaction mechanism work?

Creating transactions is done in few steps:

1. Choose destination address (or addresses)
2. Choose source addresses (=coin selection)
3. Calculate transaction fee
4. Sign inputs
5. Combine everything into transaction as a stream.

All those steps must be addressed in bitcoinj already, because those are necessary steps for every client. Those steps are implemented also in Electrum client (python).

However things can be slightly easier, because steps 2,3 and 5 can be done on server side (it's usually called super lightweight client, because it depends on server even with coin selection). BitcoinSpinner (java based android client) is doing it already.

kjj

legendary

Activity: 1302

Merit: 1026

Quote from: slush on December 30, 2011, 08:32:35 AM

Quote from: kjj on December 29, 2011, 09:49:26 PM

Add a "signature_algorithm" key to the response. Also, allow the request to specify a list of acceptable algorithms, like {"id": "1234", "signatures_accepted": "ecdsa: SECP256r1, SECP521r1; pkcs 1", "firstbits": "1payBTC"}

good idea

Quote

As for actually calculating it, there are two ways to go. In a binary protocol/format or when the length of the signature is known in advance, it is common to construct the response using zeros to fill in the signature field, then calculate the signature based on that, then overwrite the zeros with the actual signature. In a text stream like this, I would build the response without the signature key and value, then construct a new response that includes them. Or just insert them, modern languages have no problem chopping and splicing strings, and C programmers already have lots of practice doing it the hard way.

Yes, I was thinking about this, too. Unfortunately it's getting slightly complicated, because there are also some other fields which can be changed without corrupting the signature, like message "id". It's because ID must be unique in particular session, but this signed response can be created as a response to some proxy call to backend (in the firstbits example, signature is created directly by firstbits service, but signature must be valid also as a response from overlay server to client.

Another issue is that fields in json-rpc don't have strict order, so two identical and valid json-rpc may have different signature (well, 2112 must be laughing now :-) ). This is problematic for validating messages, because you need to extract fields which are not part of signed message, but not affect message payload.

This leads me to conclusion that trying to sign/verify message itself can be very fragile. I also don't want to go against json-rpc specification and force such things like order of fields. Currently I don't know how to decide this issue, is there any best practice to doing this? How other text-based protocols solve such problem?

For the usual case, you don't care that multiple messages might be equally valid. The signature is for "this message", not for "any message bearing the same content as this message".

If you are worried that a proxy might replace the message with a message that is different but equivalent, then you need to have a canonical form, and you should only emit the canonical form. When the client receives a message, it must convert it into canonical form before checking the signature.

As for the ID, don't leave it out of the signature unless you enjoy replay attacks. Just force the client to make it globally unique. There are already well established techniques for doing this.

Topic: [Stratum] Overlay network protocol over Bitcoin - page 7. (Read 37890 times)