Topic: Bitcoin protocol standarization (Read 3825 times)

implementations of what?  machines that communicate in a predetermined agreed language implement a specification, regardless of how that spec is defined, unless it is just one program.  saying that a program implements a spec defines by solely by its own code isn't really an implementation of anything.   

People who are pro-standards, or anti-standards, are both extremists.  There are good standards, bad standards, and mediocre standards.  We learn and improve.  How did the good standards come about and avoid becoming bad standards?  Bitcoin is built on many standards which, if you like Bitcoin, you hopefully agree they are good standards to have, such as TCP.   


True of any communications protocol.  It's a good thing HTTP works, or we wouldn't be here right now discussing this. 


It is true there is greater security risk with bitcoin.  But, that doesn't mean that new code preceding specification is inherently more secure than having a spec to define and discuss a change beforehand.  In fact, if you really felt comfortable with the way new code is being written to create new implementations of bitcoin, you wouldn't be proposing a VM to address this concern.  To be sure, I think the VM idea is a very good one.  I'm just using it as an example because you clearly don't believe things are perfect the way they are today and the potential risk of new implementations.  If it aint broken, don't fix it, right? 

Politics sure is a fuzzy word in any scientific conversation.  Let's reverse this.  Are you saying that open source projects don't have politics?  Are all developers always in perfect agreement.  That's why I like the early days of the IETF.  That was about as minimal on the politics as you could get.  It was engineers who worked together very well to create new things together.  It wasn't until about the mid 90s that the culture began to change.  To some extent, I think we both agree that culture is always a risk.  But, I believe that risk is just as real without a standards dialog.  As for some of your concerns, that's about WHO is influencing the standards, or what type of person, not the fact that the standardization process is visible and perhaps a bit formal.  That said, I haven't proposed any type of process.  I agree that if one were to develop a process, it might need to be a bit unique for Bitcoin to optimize the primary objectives of Bitcoin: Security, Scalability and De-centralization, with a high bar for any direct impact ability (e.g., voting) to ensure technical credibility and common ground on core principles.  Yet, the non-direct impacting open discussions could prove incredibly value-able towards Bitcoin's objectives.  And the visibility on the process cannot possibly hurt.       


+1 and I understand your concerns.  Keep it scientific with the core team dedicated demonstrating technical excellence and a common vision.  Note, the latter hasn't always been clear to-date, such as with anonymity and fungibility. 

Like SSL and PKI? 

You again make it sound ilke a choice between math and politics.  Math and politics will always be there.  The only question is HOW you manage the politics.  Do you hide it and pretend it's not there, because there is so much math?  Or do you limit it in your core charter with checks and balances and hopefully an agreement on guiding principles?     


True, except the "..." part.  hahaha 


ROFL!  Not sure how you think Bitcoin is as complex as physics computations, or how you think people will die if the current political status quo has increased visibility.


Except Bitcoin isn't a virtual universe. 


I know.  I was digressing a bit into a vision that in itself has nothing to do with Bitcoin.  I do see a new M2M virtual currency being required to grease the wheels of trustless resource and task distribution, developed from the ground up to meet the needs of M2M.  Examples of how requirements would differ from Bitcoin: transactions would be incredibly micro.  Yet, because these are virtual machines in a virtual universe, they could settle debt in much larger time spans, such as once a day, decreasing the load on the primary order book.  It could be a combination of trusted and trustless, as well. 

I appreciate your passion for the success of Bitcoin.  You do bring a lot of insight and I do empathize with your concerns even when we don't totally agree on whether or not the current political status quo is enough to ensure the success of Bitcoin over the next 10 years.     

It wouldn't be "just for Bitcoin", but would be "where Bitcoin-related programs are particuarly short". I would presume that it would have (at least some) 256-bit registers and the elliptic curve operations would be single instructions.

For the similar projects please skim through the zk-SNARK paper to get a high-level overview of the Tiny-RAM machine (and related GNU C compiler back-end) implemented there.

https://eprint.iacr.org/2013/507.pdf

The goals are also somewhat similar to the goals of SystemC, a C++ subset/superset.

http://en.wikipedia.org/wiki/SystemC

The main objective is that the architecture must be so precisely defined as to allow automatic synthesis of equivalent digital circuits and machine-generated proofs about some programs expressed in that language/bytecode.

It has function call operator (used in P2SH) that allows implementation of iteration through recursion, so this old chestnut should better die.

Personally I would hope not - I was involved in a hugely funded project to build a stack based VM system before Java existed in the early 90's and as a programmer (who worked on that project) I can say that I am no fan of such stack based languages (they are far less elegant than using a more traditional CPU style approach).

That's why I said "something like". Script would indeed be completely unsuitable. But the specification language would have a similar purpose, and maybe a similarly-simple stack-based language would be appropriate. (I don't know much about how something like this should actually be designed, though.)

I very much doubt that Bitcoin's script could be of any use as it is not Turing complete and although actual Bitcoin transactions are handled by it things like re-orgs cannot be handled by it at all (nor could it handle any of the p2p stuff).

With the sort of VM that gmaxwell is talking about, all of Bitcoin's consensus rules would be specified in a simple and exact programming language (something like Bitcoin's Script, maybe), and then every full node would include this consensus specification verbatim in its code and directly interpret it when checking things against the consensus rules. The specification would not describe or dictate the rules -- it would literally *be* the rules. This ensures that all full nodes have exactly the same behavior, but since each implementation designs its own VM/interpreter for processing the specification, different designs for the lower-level storage operations (etc.) would be possible.

This seems to me like a very good/interesting idea.

I am also keen to understand the VM suggestion and why @gmaxwelll thinks that would be better than just creating a C/C++ lib with wrappers for other languages to use.

Are we talking about a new VM that is just for Bitcoin?

I've been thinking about all this, about a standard, the code and the available documentation and I have to admit that there were several misunderstanding from my side.

1) Source code is the more detailed specification you can find. This is always true.

2) The reference implementation is not messy code, in fact it is pretty clear now. Last time I took a look at it was in 2011/2012.

3) Available documentation is very good. I didn't know about the developer guide before this post and it is rather detailed (the wire protocol and other low level data structures are well described in other places).

I think that my misinformation stole time to @gmaxwell and others however maybe it is a good thing because other developers could have the same questions about standards.

I'd love to have a library (a .dll) to link in my project with all (or at least the hard part) what is needed to implement a node and I am sure it is the way to move forward. If a VM is a better option I would like to understand why.

   

Wow, great post.  Learning much from it.  Perhaps worth perserving somewhere on the wiki?

What I was talking about there isn't about plug-ability or extensibility; it's orthogonal to it and wouldn't itself offer any additional flexibility. The motivation there is being able to achieve absolute consistency in the parts of the system which MUST be absolutely consistent while still allowing for multiple implementations by reducing the surface area where things could go wrong.

The kind of flexibly you're thinking of might also benefit from some of the same tools... but a consensus system like Bitcoin has some pretty incredible overheads that preclude a good bit of application space by their very nature.

That's true.  Yet, good specs do mature to standardization and an open process independent of any one implementation. 

http://arstechnica.com/information-technology/2014/10/html5-specification-finalized-squabbling-over-who-writes-the-specs-continues/

Bitcoin has come a long way in 5 years.  Kudos to all the great innovators and hard laborers who made it happen.  The question is how best to objectively advance it over the next 10 years.  Should a standard drive a reference implementation, or a reference implementation drive the standard? 

That said, you could say there is some hair splitting in this discussion, as bitcoin has naturally taken on traits of an open standard through people's drive to improve implementations and the long-term health and viability of the protocol.  With open discussions, it has been behaving recently like the early days of the IETF. 

gmaxwell, i was replying to one guy's statement, but didn't want to be too obvious by quoting him.  here's what he said that I didn't think the OP deserved:

"Any developer with basic knowledge of software engineering, c++, and cryptography concepts are all that is required to read and understand the vast majority of the reference code."

The documentation you linked to, on the other hand, is very useful, and reads a lot like a spec. 

Now that you pointed out the VM definition here, that is ironically what I have been thinking of developing for years for P2P, but with a broader context, a re-usable a service layer.  I'd like to create a P2P platform that can be used to create all kinds of novel things. 

I like the core functionality that retroshare has.  but, unfortunately, it's not spec'd to be a VM'd P2P platform, although they have tried to make it plug-able and have built good re-usable functionality like its cache services.  I'd like to help build a platform that can provide vast capabilities -- building blocks that can be used to create higher level craziness limited only by imagination.  A VM could provide standard access to the building blocks and simplify development of p2p apps, while protecting the machine running it.  To promote rapid adoption, I'd like a reward system for those running nodes dedicating resources, with perhaps a coin dedicated completely to machine to machine task interchange and resource sharing.   

I can attest that it's possible to write a client without looking at the reference code. @erik777 - what do you think a standardization process would add? Bear in mind that HTML 5 doesn't have an official spec and is still used by most of the internet.

It wouldn't: You'd use a authenticated data structure so that all the input could come from an unverified domain. It could fail completely but it would know it... even a not found would be expected to prove non-membership by providing fragments for a search query that failed against the current root.  The UTXO doesn't even need to be committed in the blockchain for this to work, e.g. the bytecode machine could just keep track of the root itself.

(If you wanted to be especially paranoid you could use path-oram to make sure that there was no visible access pattern or data sequence outside of the sandbox thta was common on multiple hosts which could cause a systemic failure).

The communications are already speced out in reasonable enough detail that multiple people have implemented talking to it from just the documentation. (e.g. from he developer guide and Bitcoin Wiki/

You're accusing people of being dishonest?  Really?  Try engaging in a little professionalism. Next time someone gives you an explanation you don't understand, you could try not accusing them as dishonesty as a first step. It certainly doesn't motivate people to spend their time educating you on a subject about which you are clearly ill-informed when you take that approach.

Sometimes people perceive value incorrectly. It's not that the code is "fine" its that for the normative operation of a consensus system it's all that actually matters. Additional informative documentation is useful too, and people have written a fair amount of it.

Funny, I don't see how thats an example of better peer review of the protocol at all: It requires only an understanding of the system at the level in the original Bitcoin whitepaper (which has been available since 2009). OTOH, I've used that work before as an example of how academia needs more peer review from industry-- because that design breaks decentralization which is the whole point of Bitcoin, and Amiller proposed a very similar design (and arguably better, since there were some additional attacks it resisted) back in 2011 which the work should have cited.

A bytecode simulator, like the JVM. One intentionally constructed to be very simple in order to avoid the extreme reimplementation risk in consensus code.

I agree that it would be a good idea to define a language agnostic spec for the communications protocol, and I haven't heard a single valid argument against it.  If you are against defining an open standard, at least be honest and just say no one is motivated to do it or share whatever concern you have would come out of such an effort.  Honestly is ok.  But, obfuscating lack of motivation with "the code is OK" is just a distraction that undermines the value others perceive.  Regardless of whether or not someone currently seasoned in C++ finds it OK the way it is, an open specification would provide value to those of us not so motivated to pick apart a C++ program just to understand the communications going over the wire.  This could also enhance peer review by increasing the respective peer review audience, and possibly foster more innovation around the network.    

EDIT: The next thing I read after posting this happened to be an excellent real example of the value of better peer review of the protocol:

https://bitcointalksearch.org/topic/new-paper-accelerating-bitcoins-trasaction-processing-359582

Can someone tell me what a VM means in this conversation?  I certainly doesn't sound like a virtual machine that runs an OS, nor does it sound like a JVM, unless I'm just not connecting the dots.    


  

This VM would need to be able to have enough memory for the UTXO set.  All the crypt functions would presumably be opcodes.

Accepting blocks into the memory pool presumably doesn't need to be handled by the VM, since that won't cause a fork.  Each miner would check their own blocks before mining.

With UTXO set commitments, block verification could be more self-contained.  The VM would run on a block + Merkle path based metadata + the header chain.

I agree with this - Linux has had a long history of libs written in C that end up being available for other languages - why the need to go against that and create a Java thing (as I assume is being suggested)?

That's exactly what I mean.

I've heard people saying that one of the developer's perversion is that they want to rewrite all in another programming language, everybody wants to have a library available for being used in their favorite language. It is easy to understand why devs want to use a confortable language for their projects so, even when that could be too hard in the case of bitcoin full node, new implementations are going to appear and that could be a problem.

It is clear that bitcon is not a commnon thing and there are lost and lots of details to have in mind and tests (like security). However, if I am rigth and new implementations are developed then the bitcoin core team will need to do something.
 

I think that separate library (or specialized libraries) is an very good idea. I don't understand why a VM could be required.   

One of my longer term hopes around the refactoring of Bitcoin core into a separate library for consensus is that it allows us to compile the consensus parts into a simple bytecode with a narrow interface which can be executed by an easily implemented and testable virtual machine, so every implementation can just use the same bytecode and be confident that they'll be consistent.  Considering how hard it has been to get people to understand the unusual requirements of consensus, it may turn out to be hard to get people to accept the performance hit (and to avoid doing crazy JIT stuff which breaks the obvious-correctness of the approach).