Topic: End the Confusion - Organizing the Protocol development effort (Read 2069 times)

I want to thank everybody for their feedback on this issue.  I've been hesitant to get involved on the bitcoin.it wiki mainly because I don't want to seem like intruding into somebody else's project without at least discussing the issues first and knowing where they were coming from.  If that is going to be at least the new place that can be used to document the protocol, I'm encouraged by the thought and will try to move over the rest of the information on the Doku wiki that is applicable.

I believe that once this protocol is fully documented, it will be a great benefit to the community as a whole in ways that are not seen at the moment.

So far I count 3 votes for the new Wiki, and none for the other options. I'll go ahead and mark the Wiki and the stub on http://code.google.com/p/pybitcoin/wiki/BitcoinProtocol as obsolete, and start referring to bitcoin.it instead.

+1

the protocol needs to be independent of the 'official' client. ideally the client is updated frequently for performance and ui improvements, the protocol changes little or not at all.

Yeah, I guess there's no point in changing the protocol in depth, but we can still add places to create backward compatible changes, make the protocol extensible.

I had some pretty big issues implementing the protocol in Java, due to the endianness, but I worked around it, so that's fine, even if it's a bit of a byte array copy hell

What I do hate however is the ever increasing version number. A new version of the main client increases the version and it is impossible to distinguish whether another implementation is still compatible, the version number should be used for the protocol version and not for the client version (there's the version string for that).

Yeah, I can imagine future new protocol messages being done differently, but the way BitCoin works practically ensures there's no point in trying to change the existing protocol. The hashes are all calculated over raw messages pretty much so it's impossible to (say) make everything big endian, or change the sizes of certain fields, without starting a new block chain.

Fortunately most successful protocols in the world are fairly quirky so I don't think this will stop BitCoin from going places

I have come to the conclusion that the protocol 'as implemented' is pretty much set-in-stone.  While small modifications may be made in the future for security, the block will be the same as now in 50 years time.

The only thing that will change, is how we distribute and share the blocks between clients.

A brand new protocol is quite the new and separate effort from simply documenting the existing protocol.

As the original author of the pybitcoin document my vote goes to the Wiki too.  As for the evolution of the protocol it can only improve, little endian, strange design choices like unsigned int 64 length fields and then fixed size fields are a really strange mix.

Also we need to create a minimalistic core protocol with places to extend (see the capabilities discussion on this forum) : make it easy for people to implement the core and make it hard to get it wrong!

I was even starting to think about creating my own protocol with gateways to the existing one just to get it right with a well thought through structure.

In addition to documenting the on-the-wire protocol, I think we should document the rules a peer should follow. Some rules are for behaving cooperatively, such as participating in the flood-based broadcasting. Others are for using the network safely, how and when to validate incoming blocks and other data. I think you have some of this in the original wiki page RHorning, but I'm not sure it has all propagated to the new wiki.

I'm referring to the definition of "forum" meaning "a place to get together and discuss things of a common interest", not something mechanical or specific to any given piece of software or any configurations like setting up another section in the current forums.  We just need a common place to communicate about the networking protocols, and yes the technical forum is fine.

There is the mega-thread about the protocol development that is here which serves the purpose of the technical side of things:

https://bitcointalksearch.org/topic/bitcoin-protocol-specification-1860

Until that thread starts to become fractured in the discussions and needs multiple threads to keep up, that should serve our purposes for now.  I am open, however, to discuss any other venues or places to communicate.  As I said, this is a political and not a technical issue, as it is about how the future of Bitcoin ought to be developed.  I'm referring to the meta discussion about how we should proceed and where discussion ought to happen.

Good idea.  I vote for the new wiki as the right spot, with the Discussion pages there as the place to bring up issues with the documentation/etc.

I'll put a pointer to the new wiki at the beginning of my bitcointools NOTES.txt.


Do we need a separate forum specifically for network-related stuff?  Seems to me the Development forum here is the right place for those kinds of discussions.

For many of those who are forum regulars, you should know that I started a documentation effort with the old doku wiki here:
http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1

In addition, with the move of the "documentation" to bitcoin.it, there is now a new effort here:
https://en.bitcoin.it/wiki/Protocol_specification

Furthermore, there are additional efforts to formally document the protocol at the following locations:

• http://code.google.com/p/pybitcoin/wiki/BitcoinProtocol
• https://github.com/gavinandresen/bitcointools/blob/master/NOTES.txt
• https://github.com/bitcoin/netspec/blob/master/protocol_specification.txt

In addition, there is always the C++ code of the reference implementation of the Bitcoin client itself that "documents" the protocol by way of the source code itself.

Mind you, I'm not here telling anybody that they shouldn't try to start their own documentation effort for the protocol, but I think it would be to a great advantage to everybody involved if we all worked together rather than having this accomplished as a bunch of different and separate efforts.  More significantly, I would like to get a forum or some place where we as a community can intelligently debate and discuss the internal workings of the software and specifically do either a "cleanup" of the protocol or to rationally make changes to the overall protocol that can be beneficial.

Some of this is planning ahead and trying to anticipate the needs of the community, but I would like to point out that this network protocol really is bitcoin in its most raw and base level.  From long experience I've seen how coding first and documenting later can be a bad thing once you are trying to put some stability to some software.

Yes, there is a need to do rapid prototyping when you aren't sure about how to proceed with a software development project, and sometimes you can get ahead of yourself where you keep coding, but with the thousands of dollars (and other kinds of currencies) now invested into Bitcoins and (we all hope) soon to be millions involved, it is about time we also discuss just how the "protocol development" ought to proceed.  Some new structures were introduced into the Bitcoin software v 0.3.17 and while there has been some introduction to those structures, I think it would be wise to debate those structures in a public forum before they get implemented in software more completely.  There are also other ideas which have been discussed elsewhere, including perhaps things related to the "Bitcoin-DNS" projects and other "extensions" to Bitcoin that may prove to be useful.

In other words, I am also calling for a more formal approach to the protocol development.  This is not really a technical issue so much as a political issue.  Enough of the protocol has now been documented that alternate implementations of the protocol are now possible based strictly off of this documentation, and Bitcoin is about to hit a new level of organization where people using this documentation will take the concept of cryptocurrency to many new directions.  With web protocols, we have the "world wide web consortium", and of course with more general network protocols we have the IETF.  How can or should the "governance" of the bitcoin protocol happen, and in what ways can changes to bitcoin help benefit the community?

More specifically, I'd like to end the confusion and make at least one documentation effort more or less "semi-official" at the moment by general community consensus, but at the same time we ought to be discussing where we can or should go from here with this documentation effort too.  To me it seems to be the wrong way to get things accomplished by publishing code and expecting the documentation to catch up... at least if the goal is to stabilize the network.  There are also some "flaws" in Bitcoin which also need to be intelligently discussed, including known weaknesses with the current specification.