@Gavin Andresen
Thanks for the information. Don't get worried about the tests. I'm fully aware they are incomplete. The tests can always be fully implemented later on. I welcome anyone that is interested to contribute to the testing code.
So I'm following "make it work for a couple of test cases then move on and come back later to fully implement the tests".
I'll also need to come back later and complete the documentation. I'm documenting to some degree as I go along but there are parts that I'll be better able to document later.
Topic: cbitcoin - Bitcoin implementation in C. Currently in development. - page 8. (Read 20312 times)
June 04, 2012, 10:42:18 AM
June 04, 2012, 10:23:33 AM
If I had to write something in straight-C, I would use the D language.
if i had to use X, i would use something different and unrelated Y.WTF? the reason for using C is that its simple, small and portable. which i suspect that D is not.
June 04, 2012, 10:17:45 AM
Hopefully this is the right way. Am I right thinking that the signatures are 70 bytes and the public keys are 66 bytes?
Signatures are BER-encoded data structures, and can be an arbitrary number of bytes (if they're DER-encoded, which is the strict subset of BER encoding, then they're 70-something bytes).
Public keys are either 33 or 65 bytes (not counting the "push the next N bytes onto the stack" CSCript opcode).
I've got to say you make me nervous; you seem to be following a "make it work for a couple of test cases then move on" style of development, which is a bad way to create a secure, robust codebase.
PS: I sympathize with you RE: OpenSSL's lack of documentation....
June 04, 2012, 10:03:42 AM
Except D isn't C. 
OpenSSL is terribly undocumented. I'm currently looking at checking transaction ECDSA signatures using:
The first parameter is the 70 byte signature, the second is the 32 byte SHA256 digest and the last is the public key data. For testing I need to implement this function. I'm using OpenSSL for the tests but it doesn't document converting public key bytes to an EC_KEY object. Not good at all! But I looked at the bitcoin source code and I saw o2i_ECPublicKey so I'll use that...
Hopefully this is the right way. Am I right thinking that the signatures are 70 bytes and the public keys are 66 bytes?
OpenSSL is terribly undocumented. I'm currently looking at checking transaction ECDSA signatures using:
Quote
bool (*ecdsaVerify)(u_int8_t *,u_int8_t *,const u_int8_t *); /**< Function for verifying ECDSA signatures for a 70 byte signature against a 32 byte hash using a 66 byte ECDSA public key. */
The first parameter is the 70 byte signature, the second is the 32 byte SHA256 digest and the last is the public key data. For testing I need to implement this function. I'm using OpenSSL for the tests but it doesn't document converting public key bytes to an EC_KEY object. Not good at all! But I looked at the bitcoin source code and I saw o2i_ECPublicKey so I'll use that...
Quote
bool ecdsaVerify(u_int8_t * signature,u_int8_t * hash,const u_int8_t * pubKey){
EC_KEY * key = EC_KEY_new_by_curve_name(NID_secp256k1);
o2i_ECPublicKey(&key, &pubKey, 66);
ECDSA_verify(0, hash, 32, signature, 70, key);
}
EC_KEY * key = EC_KEY_new_by_curve_name(NID_secp256k1);
o2i_ECPublicKey(&key, &pubKey, 66);
ECDSA_verify(0, hash, 32, signature, 70, key);
}
Hopefully this is the right way. Am I right thinking that the signatures are 70 bytes and the public keys are 66 bytes?
June 04, 2012, 09:22:05 AM
If I had to write something in straight-C, I would use the D language.
June 04, 2012, 08:05:52 AM
Unfortunately from what I see it doesn't help with my SIGHASH issue. I made a question here: http://bitcoin.stackexchange.com/questions/3890/for-sighash-single-do-the-outputs-other-than-at-the-input-index-have-8-bytes-or
June 04, 2012, 01:49:15 AM
I'm assuming with SIGHASH_SINGLE the output scripts are blank but the VarInt is still there, so each output up to before the input index will have 9 bytes, 8 for the value and 1 for the VarInt. Can anyone verify that? Or is it just the value with 8 bytes? Trying to track it down in the C++ client was taking too long.
Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.
Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.
I'm not sure I understand exactly your problem, but maybe it could help if I tell you that I had lots of difficulty trying to understand the bitcoin serialisation protocol, until I read Gavin's bitcointools python code. It is much easier to understand than C++. It helped me a lot for my perl library.
June 03, 2012, 11:59:57 AM
I'm assuming with SIGHASH_SINGLE the output scripts are blank but the VarInt is still there, so each output up to before the input index will have 9 bytes, 8 for the value and 1 for the VarInt. Can anyone verify that? Or is it just the value with 8 bytes? Trying to track it down in the C++ client was taking too long.
Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.
Edit: Apparently the C++ client is platform dependent on the x64 architecture meaning the signed integers should be twos compliment and little endian.
Also how are output value signs represented as serialised data? Similar to the script, except as little endian? Twos compliment? The serialisation code in the C++ client is just too over my head for me to understand.
Edit: Apparently the C++ client is platform dependent on the x64 architecture meaning the signed integers should be twos compliment and little endian.
June 03, 2012, 07:35:46 AM
(registering to this thread)
June 02, 2012, 05:49:04 PM
Now I've implemented transaction serialisation and deserialisation. Here is the test code for that: https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/test/testCBTransaction.c
Next I'll move onto transaction signing and validation. After that I can work on the blocks, block chain, block validation and then the networking.
Once again, if anyone wishes to aid in anyway the best way is to scrutinise the code for bugs and problems and improve the tests.
Next I'll move onto transaction signing and validation. After that I can work on the blocks, block chain, block validation and then the networking.
Once again, if anyone wishes to aid in anyway the best way is to scrutinise the code for bugs and problems and improve the tests.
I just had a brain wave. There is no need to do copying of data for serialisation and de-serialisation of bitcoin messages like bitcoinj does. I can store the serialised byte data and just access that through pointers. Should be more efficient.
It's problematic trying to use a Java library as a basis for a C library. :-)
I need to store integers separately in the systems endianness.
It's problematic trying to use a Java library as a basis for a C library. :-)
I need to store integers separately in the systems endianness.
The transaction tests are here: https://github.com/bitcoin/bitcoin/blob/master/src/test/transaction_tests.cpp
Hopefully I'll get around to starting transaction validation next week, though I'll be busy over the next few days.
If anyone would like to contribute, improving the test code would certainly be helpful.
Hopefully I'll get around to starting transaction validation next week, though I'll be busy over the next few days.
If anyone would like to contribute, improving the test code would certainly be helpful.
Thanks.
I note that the difficult ones, CHECKSIG, etc, aren't included. I guess that's because there is no transaction for the script to operate on?
Is there a plan to include those difficult ones in the unit tests later? Or will they be part of the transaction unit tests?
I ask only so I know whether to consider creating a fake transaction for the test scripts to run inside?
I note that the difficult ones, CHECKSIG, etc, aren't included. I guess that's because there is no transaction for the script to operate on?
Is there a plan to include those difficult ones in the unit tests later? Or will they be part of the transaction unit tests?
I ask only so I know whether to consider creating a fake transaction for the test scripts to run inside?
I'm not familiar enough with Satoshi client internals to easily understand what script_tests is doing. Could you confirm my understanding of the format of the two JSON files?
The format is a list of pairs, where the first item is the scriptSig required to spend and the second is the scriptPubKey. The third item is ignored (useful for comments).
script_valid.json contains only valid scriptSig/scriptPubKey pairs.
script_invalid.json contains only invalid scriptSig/scriptPubKey pairs.
valid/invalid are defined by the rules of transaction validation, and the unit test actually constructs transactions and runs the verification routine on them to make sure that they succeed or fail, as expected.
sup
Thanks Gavin. I do need to implement more tests, I will do this at a later time, first I need to implement the transaction structures properly and OP_CHECKSIG. I'll have to use some of the tests you posted!
Your unit tests look like they'll miss lots of edge cases.
I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).
JSON format tests are here:
https://github.com/bitcoin/bitcoin/tree/master/src/test/data
(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )
I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).
JSON format tests are here:
https://github.com/bitcoin/bitcoin/tree/master/src/test/data
(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )
I'm not familiar enough with Satoshi client internals to easily understand what script_tests is doing. Could you confirm my understanding of the format of the two JSON files?
Obviously it's an array of arrays, but some of the inner arrays have two elements and some have three. I guess the first two elements are the two component scripts? Which one is which? Is the third element simply an error message to be issued if that test fails? What is considered a fail -- i.e. is the script meant to always reduce to one answer and that answer should be TRUE (or FALSE depending on which file is used)?
Your unit tests look like they'll miss lots of edge cases.
I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).
JSON format tests are here:
https://github.com/bitcoin/bitcoin/tree/master/src/test/data
(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )
I've been working on cross-implementation unit tests for Script, and am actually working on more tests today (and am working on a testnet reset that will embed the "should validate" tests into the testnet block chain).
JSON format tests are here:
https://github.com/bitcoin/bitcoin/tree/master/src/test/data
(read by https://github.com/bitcoin/bitcoin/blob/master/src/test/script_tests.cpp )
The script interpreter is implemented except for the signature checking codes. If anyone wants to help, the interpreters should be scrutinised and in particular checked against the bitcoin protocol to ensure it operates correctly. This is it:
https://github.com/MatthewLM/cbitcoin/tree/master/cbitcoin/src/structures/CBScript
The interpreter is here:
https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/src/structures/CBScript/CBScript.c#L105
All documentation is in the header file.
The tests are in this file:
https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/test/testCBScript.c
Next I'll implement the transaction structures before moving on to signature checking. Then I can implement the transaction validation and move onto blocks.
https://github.com/MatthewLM/cbitcoin/tree/master/cbitcoin/src/structures/CBScript
The interpreter is here:
https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/src/structures/CBScript/CBScript.c#L105
All documentation is in the header file.
The tests are in this file:
https://github.com/MatthewLM/cbitcoin/blob/master/cbitcoin/test/testCBScript.c
Next I'll implement the transaction structures before moving on to signature checking. Then I can implement the transaction validation and move onto blocks.
Well it doesn't matter that I cannot do any test. While it would be interesting Boehms GC may be insecure and reference counting is not too much of a headache using the right tools; it didn't take me long to fix the memory leaks. So now I'll move onto just testing the virtual functions vs ordinary functions before moving on to the script.
I was able to test removing the redundant virtual functions and it gives a speedup of about 3% which is not worth the change. Now that I did that I'll move onto the script.
Edit: Just have the arithmetic and cryptography codes to implement in the script interpreter. If anyone wants to help, the script interpreter certainly does need scrutinising.
I was able to test removing the redundant virtual functions and it gives a speedup of about 3% which is not worth the change. Now that I did that I'll move onto the script.
Edit: Just have the arithmetic and cryptography codes to implement in the script interpreter. If anyone wants to help, the script interpreter certainly does need scrutinising.
Jump to: