Pages:
Author

Topic: ANN: Announcing code availability of the bitsofproof supernode - page 5. (Read 35176 times)

hero member
Activity: 836
Merit: 1030
bits of proof
Wow, that's a lot of new stuff Smiley

Is there a proof-of-concept client to play with?
Thanks, I know the effort in it Smiley

A member of my team works on a mobile client, preview likely available in two weeks.

For now, I suggest to look at this integration test to get a feel how a client would work:

https://github.com/bitsofproof/supernode/blob/master/server/src/test/java/com/bitsofproof/supernode/test/APITest.java
hero member
Activity: 731
Merit: 503
Libertas a calumnia
Wow, that's a lot of new stuff Smiley

Is there a proof-of-concept client to play with?
hero member
Activity: 836
Merit: 1030
bits of proof
bits of proof beta release 0.9 now features:

 - support for SPV clients using bloom filter
 - reduced disk footprint increased speed on server side:
       . block chain with indices is 10GB now.
       . loads from scratch under 4 hours
       . retrieval of transaction history for a set of addresses in a few seconds roundtrip over the bus.
 - a redesigned API
       . with bloom filtered subscription to validated transactions on the bus
       . encrypted wallet local file with several accounts (BIP32) and colors
jr. member
Activity: 42
Merit: 11
Thank you for detailed response.
hero member
Activity: 836
Merit: 1030
bits of proof
Thanks for your answers. What ECDSA implementation do you use and how many signature checks/sec it can perform?
Bouncy castle. Just measured on my laptop with 2.5GHz i5 single threaded validation of ECDSA: 830 sigs/second.

Beware that
1. bits of proof has multithreaded transaction validation, therefore this scales about linear with number of processors available and their speed.
2. In practice signature validation is distributed over time since the server usually receives and validates transactions while unconfirmed, therefore block validation only triggers validation of those transactions it did not receive before.

We currently have a few hundred transactions per block, 95+% of them already validated in the minutes between blocks, therefore block validation is a fraction of a second even with this implementation of ECDSA.
Storing the block is much more effort than validating it in regular production.

The only use case where validation speed plays a role at the moment is initial block chain load and verification. There it can be up to about 2 seconds per block in worst case.
This is addressed by both Satoshi and bits of proof by not validating signatures until a checkpoint height (well behind current last block).

In summary, although this pure Java implementation of the signature validation is about a magnitude slower than the C heavily optimized for secp256k1, I do not consider it a bottleneck for now.
If it ever becomes one, it should not be a big deal to use JNI to call such an optimized native one. bits of proof already uses JNI to drive native LevelDB.
jr. member
Activity: 42
Merit: 11
Thanks for your answers. What ECDSA implementation do you use and how many signature checks/sec it can perform?
hero member
Activity: 836
Merit: 1030
bits of proof
Do you have host running bitsofproof 24/7?
What are you using for ECDSA? How many sigchecks/sec does it perform?
What's maximum number of peers you tested it with?
What anti-DoS rules do you have?
Thanks.

Yes, bits of proof runs since months in parallel on bitsofproof.com, with a few restarts for upgrades or data migration.
I run it with peers 20 regularly, sometimes 50.
There are a few ban rules similar to bitcoind and limits on message size (identical to max block) also limits on caches of orphan blocks and unconfirmed transactions. There is a limit on unsolicited connections.
jr. member
Activity: 42
Merit: 11
Do you have host running bitsofproof 24/7?
What are you using for ECDSA? How many sigchecks/sec does it perform?
What's maximum number of peers you tested it with?
What anti-DoS rules do you have?
Thanks.
hero member
Activity: 836
Merit: 1030
bits of proof
Bits of proof website launched http://bitsofproof.com background info, presentations and documentation coming soon.

You might be interested to check the Jobs section.
full member
Activity: 211
Merit: 100
"Living the Kewl Life"
Note that I incorporated bits of proof zrt. and transferred the copyright on the bits of proof implementation of the Bitcoin protocol to it.

The license terms remain the same, that is Apache License, Version 2.0.

bits of proof zrt. offers custom extensions and commercial support for this implementation of Bitcoin.
The implementation is at beta stage, features will be further extended.

See further info at: https://github.com/bitsofproof/supernode

keep up the great work Cheesy
(will donate next time a re-up at mt. gox)
hero member
Activity: 836
Merit: 1030
bits of proof
Note that I incorporated bits of proof zrt. and transferred the copyright on the bits of proof implementation of the Bitcoin protocol to it.

The license terms remain the same, that is Apache License, Version 2.0.

bits of proof zrt. offers custom extensions and commercial support for this implementation of Bitcoin.
The implementation is at beta stage, features will be further extended.

See further info at: https://github.com/bitsofproof/supernode
hero member
Activity: 836
Merit: 1030
bits of proof
Looks like maven would not build the right path for tests, since it can not load the log config. This could be also the explanation for not picking up the right security provider (bouncy castle). You likely have some unique environment or build of maven. Local hosted repositories or patched?

Or you are running into Windows unable to deal with paths longer than X byte or paths with space in it or similar.

General suggestion:
Install some unix based operating system before trying to build software.
i find your suggestions a bit dissatisfying from a build perspective. You are using platform independent language and a platform independent build stack and yet you are suggesting your actual build may in fact be platform dependent. I understand you have other priorities bringing up the code to a high level of quality, but you should be in the frame of mind where your code and build will be platform independent at some point. PRab swapping out versions of protoc is a perfect example of something Maven is supposed to unequivocally disambiguate. At a casual glance, everything else in the build is looks good, which is why build issue a bit confusing.

I did do some research the other day, and I can see you are including protoc in your build as it has been recommended in various forums. My main issue is that the recommendation is incomplete (some posts even acknowledge this), which is why PRab and I are unable to reproduce your build without a non-trivial amount of effort.

I feel I can contribute to helping you create a platform independent build or at least a clearer build path, so I will try to help you to that end. I'll need to find some more time to look into this and will contact you if I find I need any more information.

You are right, ideally I would care about build process on platforms other than Linux and OS X.
I do not have bandwidth.

I deleted my rant above, but you managed to quote it before I recognized that it was a non-productive outbreak of my long standing hate relationship with Windows. That system did cause frustration to me every time I had to face it.

Your help is appreciated. I understand your frustration (sounds familiar), but I did everything to make the build a no-brainer on any unix system. If you were figuring out build instructions for windows, that would be great to avoid similar frustration.

I believe that this system will be deployed in unix like environments in virtually all cases of production. Even if production environment would be windows it could use the built artifacts without recompiling them, so the issue is really limited to developer who have to use windows for some reason.




member
Activity: 89
Merit: 10
Looks like maven would not build the right path for tests, since it can not load the log config. This could be also the explanation for not picking up the right security provider (bouncy castle). You likely have some unique environment or build of maven. Local hosted repositories or patched?

Or you are running into Windows unable to deal with paths longer than X byte or paths with space in it or similar.

General suggestion:
Install some unix based operating system before trying to build software.
i find your suggestions a bit dissatisfying from a build perspective. You are using platform independent language and a platform independent build stack and yet you are suggesting your actual build may in fact be platform dependent. I understand you have other priorities bringing up the code to a high level of quality, but you should be in the frame of mind where your code and build will be platform independent at some point. PRab swapping out versions of protoc is a perfect example of something Maven is supposed to unequivocally disambiguate. At a casual glance, everything else in the build is looks good, which is why build issue a bit confusing.

I did do some research the other day, and I can see you are including protoc in your build as it has been recommended in various forums. My main issue is that the recommendation is incomplete (some posts even acknowledge this), which is why PRab and I are unable to reproduce your build without a non-trivial amount of effort.

I feel I can contribute to helping you create a platform independent build or at least a clearer build path, so I will try to help you to that end. I'll need to find some more time to look into this and will contact you if I find I need any more information.
hero member
Activity: 836
Merit: 1030
bits of proof
Looks like maven would not build the right path for tests, since it can not load the log config. This could be also the explanation for not picking up the right security provider (bouncy castle). You likely have some unique environment or build of maven. Local hosted repositories or patched?

The:
[debug] execute contextualize

lines in your maven output suggests this is some patched maven.
member
Activity: 98
Merit: 10
Ok, I think I a pretty close to having it. I swapped out protoc to version 2.4.1 (was at 2.5.0). Looks like it compiles but then fails while running a self test.

http://pastebin.aquilenet.fr/?7810ca4c946a733a#cvCNsQFIP0bzRKaEzQIIqB6DiEbmiK4zeqmgen61Bf8=

Any suggestions?
hero member
Activity: 836
Merit: 1030
bits of proof
i'm interested in implementing this on aws and building a pristine community image and good documentation or even automated security automation on setup, and also getting running in a worker role on azure...anyone interested in collaboration?
I thought this server will be basis for unique implementations, not a commodity client for end user. Tell me a bit more about the use case you address please.
newbie
Activity: 51
Merit: 0
i'm interested in implementing this on aws and building a pristine community image and good documentation or even automated security automation on setup, and also getting running in a worker role on azure...anyone interested in collaboration?
hero member
Activity: 836
Merit: 1030
bits of proof
What versions of Java, Maven, and protoc are required?

I just tried building ("mvn package" - Windows 7 64bit) from git and it failed. I can't see the exact message because the mvn window closes automatically. It appears to make the \api\target directory, but there are just a couple of subdirectories (no actual files). In addition, it modifies \api\src\main\java\com\bitsofproof\supernode\api\BCSAPIMessage.java by quite a bit.

Any suggestions?

Recent versions of java and protobuf, that is Java 6 or 7, protobuf 2.x are needed.
The protobuf compiler generates BCSAPIMessage.java.

I am sure you find better sources with google on how to install java, protobuf or use maven on windows than my answers, since I avoid using windows whenever I can.

@grau I have some experience as a build manager and going to say "bad grau, bad", repeat after me "i must not include implicit external dependencies in maven builds". In maven you must be explicit in your pom files in such situations. That is the price you pay when including such dependencies, whether you use maven or something else.

You see the errors because the protobuf compiler is not installed or not in path.
Let me know if protobuf compiler dependency and execution can be formulated more elegantly and I will amend.

The build works as documented on https://github.com/bitsofproof/supernode/wiki/Build. Basically a one liner.

I am confident that this is the case, since the build is executed on a fresh git clone with every commit using random machines supported by the travis-ci project. Also all unit test are running through there. See recent (and historic) build and test outputs here: https://travis-ci.org/bitsofproof/supernode. The travis configuration is also trivial, just asking for default protobuf installation, see here: https://github.com/bitsofproof/supernode/blob/master/.travis.yml
member
Activity: 89
Merit: 10
@PRab If you just want more scroll back you have a few options:
-change the command prompt properties in the title bar and increase the "screen buffer height"
-pipe it out to a file, i don't know what mvn goals you are using so these are just examples "mvn clean package -l mvn.log" or "mvn clean package > mvn.log", unfortunately you can only redirect output and won't see the output in the CLI
-use a proper IDE like Eclipse or IntelliJ CE which should give you as much scroll back as you need and code URLs

I haven't tried figuring out what version of protoc he's using either, which is where I'll have to turn my attention to grau

@grau I have some experience as a build manager and going to say "bad grau, bad", repeat after me "i must not include implicit external dependencies in maven builds". In maven you must be explicit in your pom files in such situations. That is the price you pay when including such dependencies, whether you use maven or something else.

If you can't include it due to licensing or any other reason, use a stub pom that declares what version you are using and a link to any licensing terms and/or where to get it. It's painful, but less painful than another developer trying to read your mind.

Here's the default log level error I get:
Code:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.3:run (compile-protoc) on project bitsofproof-server-api: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "protoc": CreateProcess error=2, The system cannot find the file specified -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Also, seeing an ant-run makes me a sad panda, but it's adequate if it's not causing an build problems.
member
Activity: 98
Merit: 10
What versions of Java, Maven, and protoc are required?

I just tried building ("mvn package" - Windows 7 64bit) from git and it failed. I can't see the exact message because the mvn window closes automatically. It appears to make the \api\target directory, but there are just a couple of subdirectories (no actual files). In addition, it modifies \api\src\main\java\com\bitsofproof\supernode\api\BCSAPIMessage.java by quite a bit.

Any suggestions?
Pages:
Jump to: