The resultant Frankenstein monster (or family of monsters) is going to try to strangle its creator.
I agree with you that projects like this are tending to be over complicated. Many times I found that I'm engineering some monster project just because I wanted to hide internal complexity of the task. That's exactly the reason why I'm forcing myself to keep this KISS: basic transports, simple RPC protocol, pubsub mechanism, services on top of this. If you know how to make it even simplier, tell me, seriously!
I'm trying to avoid "full stack" RPC mechanisms like SOAP+WSDL, because, honestly, it's pretty hard to implement everything correctly, on both ends.
It is hard to argue with what you proposing point by point because of the contradictory nature of the requirements. I'm going to group my response into the following points.
I'm fully aware that those requirements are contradictory. It's pretty common that some solution needs to find some compromises. But correct me if you think that some of those requirements are wrong (for such purpose of overlay network).
1) Protocol can be either RPC or bidirectional, but not both.
Well, what's wrong on creating communcation channel and then have a possibility to call services on both ends, from any connected side?
I'm aware that it *can* be overcomplicated in the end, but it fully depends on designing of services. This idea is taken from jabber protocol, where both sides are providing some services. (And yes, I thought about building overlay network on top of jabber protocol, because it already provides service paradigm, but I think jabber is great example of overcomplicated stuff.)
The RPC paradigm (request-response, master-slave) is mutually contradictory with a pair of peers exchanging asynchronous messages.
I agree. I like RPC much more than message-based protocol, because it's making communication much cleaner. Just a request and its response. It's the reason why I'm not proposing communication based on async messages, because dependencies of various types of messages can turn into total mess.
Bitcoin protocol itself is from its origin asynchronous and cannot be squeezed into the master-slave architecture.
I partially agree (not fully, because everything can be transformed into master-slave architecture, but of course it doesn't necessary mean that it's effective), but I'm not talking about bitcoin network. Difference between Bitcoin network and "Overlay" network is that Bitcoin is distributed database, but Overlay is network of services.
2) Your target market (low-end consumer-level devices) demands that there's checksuming and framing at the application layer. Precisely because cheap NAT gateways and cheap DSL/Cable/WLAN modems are known to mangle the transport-level frames due to bugs (in the implementation of NAT) and excessive buffering (to improve one-way file transfer benchmarks).
If you think you can add CRC later you are going to loose by not detecting corruptions early.
Hm, thanks to you, I'm thinking about this a little more than before. Is that really an issue? Isn't TCP checksumming and TCP retransmissions on both ends enough to "fix" corrupted information? More generally, keeping transmission working is task for the transport layer, not for protocol itself. And I agree that transport implementations should use it's internal mechanisms for checking that transmission was successfull. Like using Content-Length or content checksum in http headers.
RPC proposal itself contains unique IDs of requests to link requests to response. If transport layer fail, it will probably appear as disconnection, which also closes the session (on TCP transport), so client need to reset it's internal state after reconnecting (ask for balance, history etc, to be sure it is again in stable state).
Side note: I'm running json-based protocol on the pool over a year, I had over 3300 rq/s in June peak. I agree that mining protocol is stupid and ugly (although I understand the reason how and why it has been designed in such way - don't worry m0m ;-) ) and that there are some isssues with DDoSing etc. But I definitely didn't have any problems with corrupted packets like you're suggesting.
3) In my experience JSON is probably the close to being the least resilient encoding possible. I can't disclose proprietary data, but I have over a decade's worth of reliability statistics for various remote services (RPC-like) that are sold by organizations I consulted. But the rough ranking is
as follows: (from the least errors to the most)
3.1) ultra-lean character-based protocol similar to FIX, designed originally for MNP5 and V.42bis modems, currently used through a simple TCP/IP socket
3.2) SOAP (RPC over XML) with Content-Length, Content-MD5 and DTD verification enabled
3.3) SOAP and plain XML-RPC without the above strenghtening options
3.4) JSON-RPC
3.5) RPC over e-mail with various human-readable encodings
Fair summary, thanks. I'm familiar with SOAP (although I never used it in real life). As I mentioned above, I agree that Content-Length and Content-MD5 should be implemented, but it's part of transport in my concept (because HTTP is only one of many ways how to transfer bytes from one side to another, and Content-* are headers of HTTP), not a part of protocol so we are in agreement here.
About DTD - it's definitely better face of XML concept and I see some benefits in implementing DTD as formal specification of protocols. Although I personally dislike XML, I'm open to change my mind at this point.
Is here, as an example, standardized way how to serialize data such lists or dictionaries? I picked JSON because it is providing pretty compressed serialization of some standard structures in transparent and understandable way. I can call json_encode(any_data) and don't care about "how it works" in almost any programming language. But if there is some similar, widely accepted XML specification for serializing such objects, I'll elaborate. But handling XML streams on low level is usually pain.
JSON is also infamous for letting people easily make byte-endianness mistakes, very much like current "getwork" which is neither big-endian not little-endian.
Well, endianess in getwork isn't the mistake of json, but of layer above. However, I agree, it is a pain.
4) You somehow read the my earlier suggestions about IPsec imply high-end large-volume target market. The reality is quite opposite: Windows support IPsec since 2000, Linux for a long time, Netgear ProSafe family has several models in $100-$200 range, L2TP and PPTP are available for free on all iPhones,Blackberries,Androids; many Nokias and other smartphones. The real hindrance is the HTTP(S)-uber-alles mindset, not the actual difficulty and cost of the implementation.
Thanks for explanation. I have only one experience with IPSec ten years ago and that experience was painful. Good to hear it gets better over the time. I feel I'm repeating myself, but I see that "transport" concept is one of the most powerful thing on my proposal. If somebody want to fiddle with IPSec, give it to him. But most of people will be happy with ssl-hardened TCP socket or even HTTP poll, which is very familiar for them. Both for programmers and for end users.
In summary I'd like to say that you wrote a very interesting and thought provoking proposal. I just think that the range of the targets you are hoping to cover is way too broad ($3 AVR processors, shared hosting plans, home computers, etc.).
Actually I'm already in touch with two groups of people who're developing hardware based wallets, so my proposal was created with some specific projects in mind. I agree that final audience is pretty wide, but everything I can do is to keep KISS attitude and hope I won't overcomplicate it in some way.
I have my personal litmus test for the technological implementation in the Bitcoin domain: it has to support (and be tested for) chain reorganization. Preferably it should correctly retry the transactions upon the reorg. Absolute minimal implementation should correctly shut down with a clear error message and a defined way to restart the operations.
Interesting stuff. I feel like overlay network should be "stateless", which mean that it will "forget" transaction once transaction has been succesfully broadcasted to Bitcoin network. The implementation of retransmissions should be more the task of clients built on top of overlay network. As far as overlay notify it's clients that their balance has been changed (because of blockchain reorg) and provide correct address history (using actuall chain branch), it's job should be done. The reason why I don't think overlay network should try to "fix" such issues by retransmissions is that it can turn into really complicated stuff. As an example - end user don't want to re-broadcast some transaction, because it has been created in the context of previous incoming transaction in "wrong" chain, which isn't actually stored in currect branch... But thanks for suggestion, I'll think about it more.
In the end, I really appreciate that our discussion is factual and we're discussing specific points more than "you're doing everything wrong". I think this is constructive and helps me in my project.
