Topic: ANN: Announcing code availability of the bitsofproof supernode - page 5. (Read 35169 times)

Thanks, I know the effort in it

A member of my team works on a mobile client, preview likely available in two weeks.

For now, I suggest to look at this integration test to get a feel how a client would work:

https://github.com/bitsofproof/supernode/blob/master/server/src/test/java/com/bitsofproof/supernode/test/APITest.java

Wow, that's a lot of new stuff

Is there a proof-of-concept client to play with?

bits of proof beta release 0.9 now features:

 - support for SPV clients using bloom filter
 - reduced disk footprint increased speed on server side:
       . block chain with indices is 10GB now.
       . loads from scratch under 4 hours
       . retrieval of transaction history for a set of addresses in a few seconds roundtrip over the bus.
 - a redesigned API
       . with bloom filtered subscription to validated transactions on the bus
       . encrypted wallet local file with several accounts (BIP32) and colors

Thank you for detailed response.

Bouncy castle. Just measured on my laptop with 2.5GHz i5 single threaded validation of ECDSA: 830 sigs/second.

Beware that
1. bits of proof has multithreaded transaction validation, therefore this scales about linear with number of processors available and their speed.
2. In practice signature validation is distributed over time since the server usually receives and validates transactions while unconfirmed, therefore block validation only triggers validation of those transactions it did not receive before.

We currently have a few hundred transactions per block, 95+% of them already validated in the minutes between blocks, therefore block validation is a fraction of a second even with this implementation of ECDSA.
Storing the block is much more effort than validating it in regular production.

The only use case where validation speed plays a role at the moment is initial block chain load and verification. There it can be up to about 2 seconds per block in worst case.
This is addressed by both Satoshi and bits of proof by not validating signatures until a checkpoint height (well behind current last block).

In summary, although this pure Java implementation of the signature validation is about a magnitude slower than the C heavily optimized for secp256k1, I do not consider it a bottleneck for now.
If it ever becomes one, it should not be a big deal to use JNI to call such an optimized native one. bits of proof already uses JNI to drive native LevelDB.

Thanks for your answers. What ECDSA implementation do you use and how many signature checks/sec it can perform?

Yes, bits of proof runs since months in parallel on bitsofproof.com, with a few restarts for upgrades or data migration.
I run it with peers 20 regularly, sometimes 50.
There are a few ban rules similar to bitcoind and limits on message size (identical to max block) also limits on caches of orphan blocks and unconfirmed transactions. There is a limit on unsolicited connections.

Do you have host running bitsofproof 24/7?
What are you using for ECDSA? How many sigchecks/sec does it perform?
What's maximum number of peers you tested it with?
What anti-DoS rules do you have?
Thanks.

Bits of proof website launched http://bitsofproof.com background info, presentations and documentation coming soon.

You might be interested to check the Jobs section.

keep up the great work
(will donate next time a re-up at mt. gox)

Note that I incorporated bits of proof zrt. and transferred the copyright on the bits of proof implementation of the Bitcoin protocol to it.

The license terms remain the same, that is Apache License, Version 2.0.

bits of proof zrt. offers custom extensions and commercial support for this implementation of Bitcoin.
The implementation is at beta stage, features will be further extended.

See further info at: https://github.com/bitsofproof/supernode

You are right, ideally I would care about build process on platforms other than Linux and OS X.
I do not have bandwidth.

I deleted my rant above, but you managed to quote it before I recognized that it was a non-productive outbreak of my long standing hate relationship with Windows. That system did cause frustration to me every time I had to face it.

Your help is appreciated. I understand your frustration (sounds familiar), but I did everything to make the build a no-brainer on any unix system. If you were figuring out build instructions for windows, that would be great to avoid similar frustration.

I believe that this system will be deployed in unix like environments in virtually all cases of production. Even if production environment would be windows it could use the built artifacts without recompiling them, so the issue is really limited to developer who have to use windows for some reason.

i find your suggestions a bit dissatisfying from a build perspective. You are using platform independent language and a platform independent build stack and yet you are suggesting your actual build may in fact be platform dependent. I understand you have other priorities bringing up the code to a high level of quality, but you should be in the frame of mind where your code and build will be platform independent at some point. PRab swapping out versions of protoc is a perfect example of something Maven is supposed to unequivocally disambiguate. At a casual glance, everything else in the build is looks good, which is why build issue a bit confusing.

I did do some research the other day, and I can see you are including protoc in your build as it has been recommended in various forums. My main issue is that the recommendation is incomplete (some posts even acknowledge this), which is why PRab and I are unable to reproduce your build without a non-trivial amount of effort.

I feel I can contribute to helping you create a platform independent build or at least a clearer build path, so I will try to help you to that end. I'll need to find some more time to look into this and will contact you if I find I need any more information.

Looks like maven would not build the right path for tests, since it can not load the log config. This could be also the explanation for not picking up the right security provider (bouncy castle). You likely have some unique environment or build of maven. Local hosted repositories or patched?

The:
[debug] execute contextualize

lines in your maven output suggests this is some patched maven.

Ok, I think I a pretty close to having it. I swapped out protoc to version 2.4.1 (was at 2.5.0). Looks like it compiles but then fails while running a self test.

http://pastebin.aquilenet.fr/?7810ca4c946a733a#cvCNsQFIP0bzRKaEzQIIqB6DiEbmiK4zeqmgen61Bf8=

Any suggestions?

I thought this server will be basis for unique implementations, not a commodity client for end user. Tell me a bit more about the use case you address please.

i'm interested in implementing this on aws and building a pristine community image and good documentation or even automated security automation on setup, and also getting running in a worker role on azure...anyone interested in collaboration?

Recent versions of java and protobuf, that is Java 6 or 7, protobuf 2.x are needed.
The protobuf compiler generates BCSAPIMessage.java.

I am sure you find better sources with google on how to install java, protobuf or use maven on windows than my answers, since I avoid using windows whenever I can.


You see the errors because the protobuf compiler is not installed or not in path.
Let me know if protobuf compiler dependency and execution can be formulated more elegantly and I will amend.

The build works as documented on https://github.com/bitsofproof/supernode/wiki/Build. Basically a one liner.

I am confident that this is the case, since the build is executed on a fresh git clone with every commit using random machines supported by the travis-ci project. Also all unit test are running through there. See recent (and historic) build and test outputs here: https://travis-ci.org/bitsofproof/supernode. The travis configuration is also trivial, just asking for default protobuf installation, see here: https://github.com/bitsofproof/supernode/blob/master/.travis.yml

@PRab If you just want more scroll back you have a few options:
-change the command prompt properties in the title bar and increase the "screen buffer height"
-pipe it out to a file, i don't know what mvn goals you are using so these are just examples "mvn clean package -l mvn.log" or "mvn clean package > mvn.log", unfortunately you can only redirect output and won't see the output in the CLI
-use a proper IDE like Eclipse or IntelliJ CE which should give you as much scroll back as you need and code URLs

I haven't tried figuring out what version of protoc he's using either, which is where I'll have to turn my attention to grau

@grau I have some experience as a build manager and going to say "bad grau, bad", repeat after me "i must not include implicit external dependencies in maven builds". In maven you must be explicit in your pom files in such situations. That is the price you pay when including such dependencies, whether you use maven or something else.

If you can't include it due to licensing or any other reason, use a stub pom that declares what version you are using and a link to any licensing terms and/or where to get it. It's painful, but less painful than another developer trying to read your mind.

Here's the default log level error I get:

Also, seeing an ant-run makes me a sad panda, but it's adequate if it's not causing an build problems.

What versions of Java, Maven, and protoc are required?

I just tried building ("mvn package" - Windows 7 64bit) from git and it failed. I can't see the exact message because the mvn window closes automatically. It appears to make the \api\target directory, but there are just a couple of subdirectories (no actual files). In addition, it modifies \api\src\main\java\com\bitsofproof\supernode\api\BCSAPIMessage.java by quite a bit.

Any suggestions?