There are 2 links relevant to DER encoding, which helped me to get along:
https://bitcoin.stackexchange.com/questions/41525/regular-expressions-for-der-signature-hex-also-req-address-txids
https://bitcoin.stackexchange.com/questions/12554/why-the-signature-is-always-65-13232-bytes-long?rq=1
Topic: Bitcoind does not like ECDSA (r, s) pair produced by OpenSSL (Read 363 times)
it's not hard to encode DER correctly but it takes a little effort that people don't bother to put in.
You're right, it was not hard at all to encode it correctly once I got all the facts right. Been using this "own encoder" for a while now and had no other issues whatsoever.
I encountered some other weird situation in the meantime if you want to take a look -- https://bitcointalksearch.org/topic/scriptpubkey-not-verified-upon-locking-but-when-trying-to-unlock-it-5060723
Regards
The DER encoder in OpenSSL is correct, but the above examples apparently weren't using it. The result of "I made my own DER encoder" is usually wrong... it's not hard to encode DER correctly but it takes a little effort that people don't bother to put in.
Even once you get the encoding right, you'll still have to obey the lowS rule: https://bitcoin.stackexchange.com/questions/59820/sign-a-tx-with-low-s-value-using-openssl
Even once you get the encoding right, you'll still have to obey the lowS rule: https://bitcoin.stackexchange.com/questions/59820/sign-a-tx-with-low-s-value-using-openssl
Well, there is some built-in OpenSSL function that takes a signature and DER-encodes it, but for the function that just signs there is no encoding whatsoever so my second "if" is needed and is actually hit like 50% of the time.
Haven't gotten any other rejected TX so I suppose everything is fine, the only question still open is whether that 0-byte "if" is needed for S as well (third "if" in the pseudocode) or just for R (I have them both right now but for S didn't get any hits inside the if so far).
Haven't gotten any other rejected TX so I suppose everything is fine, the only question still open is whether that 0-byte "if" is needed for S as well (third "if" in the pseudocode) or just for R (I have them both right now but for S didn't get any hits inside the if so far).
I don't see anything wrong.
But I don't know two things, your full code and how OpenSSL works. For instance I am not sure appending that 0 is even needed. Usually when these functions return a value like this they also take care of that initial 0 for you. So your second if may never be true!
But I don't know two things, your full code and how OpenSSL works. For instance I am not sure appending that 0 is even needed. Usually when these functions return a value like this they also take care of that initial 0 for you. So your second if may never be true!
First off thanks for your help
I am doing the encoding myself but 2 days ago when I wrote the question I figured if I followed the "<30><02><02>" scheme and still get rejected there's something wrong in the S/R produced by openssl, hence why I came up with the shitty code above 
. Obviously, I was wrong.
I made some progress in the meantime and out of the 10-20 txes I managed to broadcast not any of them was rejected, but this might be a coincidence so for extra checking I am posting yet another pseudocode of how I do things now:
Does this look better? Am I being lucky that my transactions made it or the code is actually correct? Thanks
I am doing the encoding myself but 2 days ago when I wrote the question I figured if I followed the "<30>
. Obviously, I was wrong. I made some progress in the meantime and out of the 10-20 txes I managed to broadcast not any of them was rejected, but this might be a coincidence so for extra checking I am posting yet another pseudocode of how I do things now:
Code:
signature = ecdsa_sign(txHash, privateKey);
// enforce a low S
curveOrder = ecdsa_get_order(secp256k1Group);
halfCurveOrder = curveOrder / 2; // actual division done with openssl's right shift BIGNUM function as well as comparison and subtraction operations below
if (signature->s > halfCurveOrder)
{
signature->s = curveOrder - signature->s;
}
// check for LSBit of the MSByte for R
if (signature->r[0] >= 128)
{
signature->r = 0x00 + signature->r; // appending by having a new buffer memcpy(buff, &zero, 1); memcpy(buff + 1, rBuffer, rLength)
}
// the same check/appending done for S -- with this Im not sure it needs to be done since I should have a low S already??
// ...
return <30><4 + rLength + sLength><02><02>
Does this look better? Am I being lucky that my transactions made it or the code is actually correct? Thanks
I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?
2 questions.
Are you doing the DER encoding yourself or is OpenSSL does it for you? If it is the later then it is doing it correctly.
And what do you mean by "non-DER-compliant"? DER is just an encoding, you have some data that you encode using a form of TLV encoding scheme. This way of encoding can handle any kind of value. From 32 byte integers (r and s) to anything else like booleans, strings,...
Quote
How should I overcome this? I am thinking of something along the lines of (pseudocode):
But isn't that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?
Code:
Pair (r, s);
do
{
(r, s) = sign(hash, pvtkey);
} while (r[0] >= 128 || s[0] >= 128); // where r[0], s[0] should be the very first byte of each value
But isn't that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?
If OpenSSL is doing the DER-encoding then let it be like that. If you are DER-encoding yourself then first you have to understand why that initial 0 is appended to the result.
For humans when we want to represent sign we put a symbol behind the number showing its sign: -2 or +2
For computer there is no "sign" under the hood so the way it does it is with manipulating the bits (zeros and ones). The way it works for big numbers (like r and s here) is that the computer looks at the highest-order bit of the last byte (little endian order) if that is set the number is negative, if it is not set then the number is positive.
1 byte is 8 bits: 0000 0000 you look at the bold bit and if it was set then you decide about the sign.
127 = 0111 1111
128 = 1000 0000
129 = 1000 0001
...
So if the last byte (or first byte since DER encodes in big endian) is bigger than or equal to 128 and you want the number to be positive (which is the case with r and s) you append a new byte (0) to it to make the computer treat that number as positive.
So this code you posted is wrong because if the first byte of the byte array of 'r' or 's' is bigger than 128 that doesn't make them invalid that you want to try again. They can still be valid, you just have to append a first 0 byte to make it positive.
What you need to check are listed in sec1 pdf section 4.1.3 (!=0 &
So here technically you always* end up with a value for s that is small, hence doesn't have its highest-order bit of the first byte (big endian order) set so you don't need that appended 0.
* I haven't really thought about this much but I believe it may be possible to find a smaller s (eg. 31 bytes instead of 32) and that s is lower than N/2 but can have its highest-order bit set (>=128 will fail and reject this valid s) which is another reason why the posted code is wrong.
Did you choose the correct elliptic curve to make the calculations ? Bitcoin is using secp256k1.
To generate a private key using openssl:
To generate the public key related to the previously generated private key:
To generate a private key using openssl:
Code:
openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out $PRIVATE_KEY
To generate the public key related to the previously generated private key:
Code:
openssl ec -in $PRIVATE_KEY -pubout -out $PUBLIC_KEY
I'm using the OpenSSL headers and linking against the shared library, not generating stuff from the command line, but as I said for like maybe 1 in 5 signatures everything is fine (because I am lucky enough that the produced signature's S value has not its first byte >= 128).
I read some other threads which said that if that happens then you need to prepend an additional 0x00 byte in front of S but after doing that now I'm getting SCRIPT_ERR_SIG_HIGH_S heheh... so now I'm trying to get S = N-S if S >= N/2 where N is secp256k1's order... any idea if that's the way to go?
Did you choose the correct elliptic curve to make the calculations ? Bitcoin is using secp256k1.
To generate a private key using openssl:
To generate the public key related to the previously generated private key:
To generate a private key using openssl:
Code:
openssl ecparam -genkey -name secp256k1 -rand /dev/urandom -out $PRIVATE_KEY
To generate the public key related to the previously generated private key:
Code:
openssl ec -in $PRIVATE_KEY -pubout -out $PUBLIC_KEY
I am writing transactions manually and have stumbled across a rather bizarre situation.
Only one in a few of the transactions I broadcast to bitcoind is accepted, otherwise I get a (Non-canonical DER signature).
So I got my hands dirty and tracked the rejection to be originating from this line: https://github.com/bitcoin/bitcoin/blob/9c5f0d542d1db507b3b9c87bd9de6d0d758d51c1/src/script/interpreter.cpp#L163
I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?
How should I overcome this? I am thinking of something along the lines of (pseudocode):
But isn't that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?
Only one in a few of the transactions I broadcast to bitcoind is accepted, otherwise I get a
Code:
REJECT_NONSTANDARD
So I got my hands dirty and tracked the rejection to be originating from this line: https://github.com/bitcoin/bitcoin/blob/9c5f0d542d1db507b3b9c87bd9de6d0d758d51c1/src/script/interpreter.cpp#L163
I read about DER encoding and checked how IsValidSignatureEncoding is enforcing it, but I do not know why OpenSSL generates not-DER-compliant (r, s) values?
How should I overcome this? I am thinking of something along the lines of (pseudocode):
Code:
Pair (r, s);
do
{
(r, s) = sign(hash, pvtkey);
} while (r[0] >= 128 || s[0] >= 128); // where r[0], s[0] should be the very first byte of each value
But isn't that kind of redundant? Can I give OpenSSL any flag to produce a valid DER (R, S) pair in the first place?
Jump to: