Topic: [ANN] BitEN - bitcoin erlang node (Read 6902 times)

I am now even more interested in BitEN for this reason

  http://www.paracoin.org/home/depth_l1/network_mesh

Looking forward to updates, but will also be taking these from the repo pretty soon I hope.

Status update:
Implemented parallel blockchain download. For now, only headers are downloaded. Download time ~4 min, CPU-bounded (need to investigate exact cause). Works fine, but not resistant to malicious nodes.
Implemented some EC cryptography in pure Erlang. Slow, but easy to understand/verify.
Switched from OpenVZ to XEN hosting on development node (still 256 MB Ram, but now with swap). Memory usage <30 MB resident, 100 MB virtual.

Plans: download whole blocks and do actual validation. Maintain UTxO set, with rollback capability.

No doubt that it has a potential benefit. It would however have lowest priority on the wish list if ranked by commercial user.

If Satoshi client was not the reference but the reference was independent tests, then such extension was permissible. Now it would compound the problem for minuscule upside.

UDP has the potential to deliver blocks and transactions more rapidly throughout the network.  The former benefits miners, and the latter benefits all bitcoin users.

Yes it is hard to get the comprehensive tests, but insisting on the entire Satoshi client as reference is not sustainable.

UDP is an example of the tremendous misalignment of incentives between "core dev" and commercial use.

Sure, I'd love it if the Bitcoin protocol was specified entirely by comprehensive tests. It's just really hard to get there with confidence.

By the way, UDP support isn't merged, it's just a proposal. So that isn't a great example.

the Satoshi implementation is much much more than needed to describe the behavior crucial for consensus, worse it is a Spaghetti and gets extended with any feature the core dev team thinks fancy to ad. Just think of the latest UDP initiative of jgarzik. Satoshi client is a moving target to other implementations and even to itself as it evolves. Although it is standard by its number of installations, this does not seem to be able to hold, as requirements diverge with more advanced use. All major Bitcoin services use modifications of Satoshi, BitcoinJ (soon bitsofproof) or proprietary implementations of the protocol.

I plead for a standard defined by text, code examples and extensive set of tests. That standard should be derived and often purposefully extracted from Satoshi code but even future versions of Satoshi code should be measured on that testbed.

Wait - this is actually we move toward. The growing list of test cases passed by Satoshi, BitcoinJ and bitsofproof are the building blocks of that new standard!

And YES, the BF should use a lion share of its resources to define the standard set as its first goal at inception.

We've talked about cleaning up some of them after a hard fork. At the same time, hard forking is (currently) a fairly tricky thing to do, and there's a desire to avoid dumping "everything and the kitchen sink" into a forking event.

The way we seem to be going is - let's try and make hard forking easier and less intimidating. If we prove it can be done without lots of disruption, then suddenly fixing (known) bugs and quirks seems much less risky and lower cost. But it does have to be low cost because, frankly, there just isn't much point in fixing most of them. They aren't dangerous. Did you know the "Referer" field in HTTP is mis-spelled? Well, nobody ever tried to fix it because it's just not worth the hassle.

Even if some of these bugs were resolved and cleaned up, I'm not sure it'd change things much. Like I said, every time we think the "spec" is complete, someone discovers a new and unexpected quirk. So how do you know you got them all? Is C++ really such a terrible language for precisely specifying behaviour? Worse than English?

And do you personally feel that keeping bugs in original code forever is the right thing? What's BF stance about it, especially "standardization of bitcoin" being major stated goal?

There have been many, many threads on this topic. Look at the bitsofproof thread by grau for some long debates on it.

Basically, everyone who has worked on Bitcoin for a long time believes the code is so complicated that it isn't currently possible to document every weird quirk and edge case that must be supported correctly. Every time we think the spec is complete, someone discovers another piece of unexpected functionality or behaviour that then must be incorporated into the description. This leads to the question of how you know you've got it all written down, and precisely enough for re-implementation.

Eventually, you come to realize - wait. We already have a very precise specification of how Bitcoin behaves. That specification has as much detail as you'd ever need, because it's written in C++ and executed directly by the CPU. So if you have a question about the precise manner in which transactions that have invalid SIGHASH_SINGLE flags are treated, the best way to get an answer is just read the code. Then you know for sure. Reading an English language derivative of it means you can never be sure it was totally correct.

Just be aware of the usual warnings. If you produce something that seems to theoretically "work" but doesn't match the behaviour of the C++ client exactly (including bugs), you can get split off the chain or end up being banned by the anti-DoS code.

Over time the documentation has improved a lot, but trying to produce a 100% compatible version of Bitcoin from it is probably still not possible.

Functional languages like Erlang or Scala are best suited to create robust multithreaded programs since their patterns avoid mutable objects and side effects. Using such language definitely helps writing thread-safe validation and service of lots of parallel requests.

A Bitcoin node performance however is constrained by block storage and EC signature validation, if you use some low level implementations for those, you get best of both worlds.
This is what I also did in bits of proof using LevelDB (native C++) and plan to use for EC.

I agree that it doesn't solve any efficiency issues or make anything faster (it is hard to beat C++ there). Erlang is very much aimed at reliability, so I'm more hoping for a bulletproof, crashless, seamlessly upgradable implemementation.

Well, example version message on the wiki has incorrect checksum - was beating my head against the wall for a hour.
Parsing was really easy (thanks for Erlang binary pattern matching), in this respect docs are ok. It's a little lacking in protocol details like state diagrams, timeouts, mempool algorithms, but I think I'll manage to produce something working (maybe not very compatible) from the spec.

Not really. If you just want to do something simple like connect, send and receive version and verack messages, then you could have something up and running in an hour. I know because I've done it. It's simply a matter of implementing the code to parse messages which is described very clearly on the wiki, no trial and error required.

Incorrect.  There are several implementations.


Incorrect.  Gavin actively encourages alternative implementations, and I've written two alternate implementations.  At least one was linked earlier in this thread.

Seems to me Bitcoin's biggest problem at the moment is that there is only one implementation. It seems that there is absolutely no desire by the core dev team to ever have more than one client, since that would actually require freezing and documenting the protocol long enough and doing interop testing.

So instead, we will continue to have a mono-culture where there is no bitcoin protocol, other than "whatever some specific version of bitcoind currently speaks".

For that reason alone, I think it is important to back this and other project and if necessary completely ignore the developers who have little interest in standardization or interoperability with more than one implementation. "Bitcoin" the standard, should be a protocol, a money API, and a clean, minimal reference implementation, not a chunk of C++ code that does everything from node to GUI.

Of course, the usual answer is: go build it yourself. I notice those who try get shouted down or thwarted by a moving target, because the dev team is off pursuing a new extension, fad or GUI feature, instead of standardizing the existing protocol.

Yeah, let's disagree.

Just wanted to echo Matt's comments here. When he talks about "verifying transactions", that is another way to say "write a full Bitcoin implementation". As many people have discovered, that is a huge amount of very difficult work. If you don't verify transactions then simply connecting to lots of nodes and blasting messages out doesn't actually achieve anything (I know this is unintuitive).

The reason is, as Matt noted, the variance in transaction propagation time doesn't seem to be to do with insufficiently dense connectivity but rather things like not all transactions that are broadcast actually being valid, skew in relay/IsStandard rules, and a few other things (old nodes hurt too but they'll be slowly evicted from the system come May 15th anyway).

Without verifying transactions just echoing them around will result in your nodes eventually getting banned, because connections that send invalid data are assumed to be malicious.

Cool though Erlang is, I'm not sure it has many benefits for something like Bitcoin. Of course you're welcome to disagree. When I look at the causes of inefficiency in the current C++ codebase, it's not clear that the language has anything to do with it. There are various things we could optimise to make propagation of data faster, although it doesn't seem to be Bitcoin's biggest problem at the moment.

I like the idea of using the Erlang platform for bitcoin. It sounds like a perfect fit, a safe language aimed at distributed systems.

Make it three.

I see great potential value in the work you are attempting if Bitcoin ever comes under widespread attack, and have seen erlang used to great effect for complex problems.