Topic: ANN: Announcing code availability of the bitsofproof supernode - page 8. (Read 35112 times)

I promised some comparison of leveldb and relational model and have some feature updates:

Performance
bootstrap node from scratch (Merkle root and POW check and store until block 210.000 thereafter full validation of scripts)

in Minutes:
                    until block 210.000      then until most recent block
LevelDB               93                        61
Relational         3664                       297

That roughly 2.5 hours against 2.5 days.

I run this on:
Linux bitsofproof 3.2.0-31-generic #50-Ubuntu SMP Fri Sep 7 16:16:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
with 4 2GHz processors 16GB RAM, Derby 10.9.1 with 400MB cache.

New features
I split the code base into

• supernode - the protocol implementation and storage engine
  
• supernode-api - a java interface that defines (not yet complete) BCSAPI and basic primitives needed in local code base of clients e.g. Hash, sign.
  
• supernode-testclient - a small java app that uses the BCSAPI to demonstrate connection and work with running server
  

The remote calls are accomplished with Java RMI using two way authentication and encryption over SSL.

An example use of securely connecting to remote bitsofproof supernode:
start the server (running on a linux server in a datacenter in Germany):
    nohup java -server -Xmx4g -jar target/supernode-0.7-SNAPSHOT.jar production relational &

ask for the balance of the bitsofproof donation account using the test client (running on my MacOS laptop at home in Hungary):
    java target/supernode-testclient-0.7-SNAPSHOT.jar -a 13xhxy21to93Lrb7d4ssZVaMnFtQVvRkSk
prints:
13xhxy21to93Lrb7d4ssZVaMnFtQVvRkSk balance: 1400000000
thats 14 BTC, retrieved in a roundtrip of 72 miliseconds.

the same momentary balance for 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp balance: 67391770
roundtrip time 658 seconds. (ca. 1.5 million transaction outputs retrieved on server side)

BitCoin Server API

BCSAPI == Bean Context Services API
or
BCSAPI == Business Connectivity Services API
?

The plan is to unlock innovation and new uses by offering a robust and compliant kernel that is modular and extensible as a service.

The primary interface to the kernel will be defined as a Java interface (BCSAPI), available throgh usual ways of remote invocaton just by configuration. For debug insight I prefer implementing JMX.

JSON RPC calls of bitcoind could be implemented on top of that by those seek backward compatibility with existing tools. I may also provide some as it seems appropriate or for demonstration.

I will try hard not to make a specific use part of this code base.

Wallets, client servicing, connection to other enterprise software and whatever people will come up with should build on that BCSAPI kernel interface, preferably remote invoking it.

I think that miners are a targetgroup in itself.

I have always wanted to provide bitcoinclients to other people, so they don't need to stay connected all the time and don't have to worry too much about the security of their computers. I wanted to run several instances of the satoshiclient, as I like how it encrypts the wallet and especially the powerful JSON RPC server. There is already so many software that works with the json-commands, it would be a great match to give a user access to a server and let them chose their own front end of their liking.

You already said that you wanted to incorporate the json rpc in your and I am wondering if you'll make it so that it would have access to multiple wallets as well?

Mining is already split to the process of block creation and POW since the second is no longer on CPU. Some will continue running both processes but I suspect more small ASICs and GPUs will trust pool operators they connect to for simplicity. Pool operators might differentiate themselves by having checks against several versions and implementations as added value service.

I think I will implement the proposal since it is useful for testing and make an instance of bitsofproof supernode accessible at bitsofproof.com for anyone to run checks against. Technical details will follow as it settles in the proposal I suggested.

Please participate in the design of this protocol extension proposal in the thread https://bitcointalksearch.org/topic/bip-proposal-automated-compliance-check-and-block-template-exchange-130327
I would like to refocus this thread to bitsofproof only.

I think miner will just stop running an implementation or version as they think it is marginalized in user population or if they think it will not get tracktion. The check is however cheap.

I think the proosal would increase network security against split and as a byproduct create a testing interface accross versions and implementations.

Since challenges would no longer be in the source tree compliant implementations could not just learn them since they have to reply correctly to any new challange.

You're right, but the upgrade vs alt-impl scenarios differ in a key way - if you're upgrading from one release of the Satoshi client to another, and everyone is doing so simultaneously, then the period of time where you want to run both in parallel is limited. Once the (vast) majority upgraded to the new release, you can stop running the old one. Also, typically there's only two versions in wide usage simultaneously. And as Pieter points out, many releases aren't at risk of changing block validation (eg, if only the gui or config file loading or whatever changes).

With re-implementations the other users on the network will never "upgrade" to the version you're running, so you would have to run both (or 3 or 4) in parallel forever. Not great.

Sure block templates are a very nice tool for miners who want to run untrusted block creation code (and probably very useful for testing), but it would be sad if evolution forced miners to run several nodes at the same time. It makes it harder to know which rules the network actually uses, and requires a lot more resources (each node will at least require maintaining its own UTXO set or equivalent).

I'm not sure why you think the 0.8 release would require any conversion. As far as I know, no reference client has ever changed the block validity rules (except for some very early bugs, and backward-compatible changes like BIP16 and BIP30). Nobody intends to make a release that risks causes a block chain fork.

It is primary interest of the miner that blocks they create are accepted by the majority of nodes, right?
Now assume Satoshi 0.8 comes out but conversion of the majority user did not yet achieved. The approach would ensure blocks will be accepted before mining work spent to them by the populatin mix.

This primary interest would be leveraged to facilitate compliance test. Big miner could also run pre-releses and provide feedback on compatibility of future releases before they are released to public.

Note that block templates would be checked that is cheap. A block template is just like the final block only that the work is not yet done (nounce does not produce a nice hash)

Under that model miners and high value targets would need to run, say, 12 implementations at at least 12x the cost (and probably more because some implementations will be much less efficient).  ... and the effective feature set of the network would be degraded to whatever the most popular broken implementation runs.  …

There can be uses for running multiple nodes for comparison— I do today. It may become a regretful requirement in the future if broken node software becomes commonplace. But this is a _very_ bad answer to the fact that your software is broken while it is not widely used.  Please fix your software before spending time on this.

+1
This is the way to go forward.

I have to say that the feature list of the supernode sounds very promising and I am afraid people are really going to use it. I for one will (altough not for mining).

Yes, credit to you for the block template.

Thanks again and I will come back to your offer and you have mine.

This idea goes a bit further to facilitate testing and perhaps compatible to BIP 23. The requirements could however be close enough for a merged implementation.

Please contribute to the thread https://bitcointalksearch.org/topic/bip-proposal-automated-compliance-check-and-block-template-exchange-130327 on this so this refocuses to the bitsofproof supernode.

Except this has been around for months now already... it's the BIP 23 Proposals I suggested you implement earlier.

If implementations support an efficient block template exchange and mutual verification then it could be cheap for the miner to run several implementations to test block template against before mining work is spent. They could as well run different versions of Satoshi code to ensure backward compatibility.

The actual implementation of such block template exchange would also be the testing interface implementations could challenge each other. I suggested this idea to Gavin about a month ago and he liked it. Now I see a further significant use of it.

Such use would however only be valuable if it is jointly supported across implementations, therefore suggest to evolve it as a BIP for block template challenge/response.

I am ready to work on the definition and implement it since it fits well to the deliverable of testing. How about you core team ?

+1, quoted for emphasis.

No.

Are you basically saying that some evil guy can install some modified Satoshi client on his average botnet and bring the network down? How many nodes would he need? ...

There is quite a difference between BDB and OpenSSL here. Depending on third party libraries is not a problem as such - there's no way we can get around depending on a C library, the OS kernel, ... as you say, that would stop evolution altogether.

The problem is that the rules (as defined by Satoshi's implementation) simply pass data directly to OpenSSL, so the network rule effectively is "whatever cryptographic data OpenSSL accepts", which is bad. OpenSSL has all reasons for trying to accept as much encodings as possible, but we don't want every client to need to replicate the behaviour of OpenSSL. In particular, if they add another encoding in the future (which, again, is not necessarily bad from their point of view), we might get a block chain fork.

So what I aim for is that the network rules are at least just defined by the implementation of just bitcoind. Ideally, we'd have some form of formal specification, but I don't see that happening anytime soon. This should make it easier for alternative full node implementations to have some assurance they do not have forking behaviour. And yes, I agree that a more diverse landscape with multiple trusted full implementations would be better, under the condition that those are well tested. I would find it a pity (but perhaps inevitable) if large miner setups would need to run multiple node implementations, and compare their constructed block against all...

And yes, fraudulent code in BDB can break things too, but that won't cause a massive fork - just a few nodes that were attacked. That is by no means comparable to an implicit rule "yeah, just make sure OpenSSL likes it".