Unfortunately the problem might be a generic one (that unsigned values cannot be used in Bitcoin scripts).
In this conversation sipa explains how Bitcoin accepts unsigned values, but then passes signatures directly to OpenSSL, and OpenSSL just parses everything as unsigned. Therefore you have to read the byte string and if it's not already zero padded, add a 0x00 byte.
https://download.wpsoftware.net/bitcoin/wizards/2013/03/13-03-11.log19:00 < HM> how do signed values even work
19:00 < sipa> they're stored as 2's complement
19:00 < sipa> so if the highest bit of the first byte is set, it's a negative vale
19:00 < sipa> but OpenSSL just parses everything as unsigned19:02 < HM> sure, but how did these transactions get accepted ?
19:02 < sipa> because OpenSSL parses everything as unsigned
19:02 < sipa> and every bitcoin full node has used OpenSSL to parse signatures
19:04 < sipa> imagine you want to store the value 0x9999
19:04 < sipa> so the positive integer 39321
19:04 < sipa> the correct DER encoding is 0x009999
19:05 < sipa> as 0x9999 is interpreted as -2621519:06 < sipa> the problem is, because OpenSSL knows it expects an unsigned integer, even if you store 0x9999, it will interpret that as 39321 and not as -26215
19:06 < HM> how is sane to encode a perfectly reasonable 2 byte unsigned value in 3 bytes with 1 useless 0x00 byte?
19:06 < sipa> because it is not an unsigned value
19:06 < sipa> DER doesn't have an unsigned integer type19:09 < sipa> anyway: bottom line: every implementation _must_ accept 0x9999 as 39321, even though a standards-compliant DER parser would interpret that as a negative number, which would cause ECDSA to reject the signature as out of range
19:12 < sipa> well in a way it makes sense: "ok you give me this signature *parse* ok, syntactically correct. wait... this R value is negative? i don't expect a negative number here... let's assume you just missed a 0 byte in front"
19:13 < sipa> the only problem is that bitcoin passes signatures directly to OpenSSL19:13 < HM> so basically you have to read the byte string and if it's not already zero padded, add a 0x00 byte Your last comment on Github asks if 5 bytes are allowed.
okay - but does your change allow for 5 bytes?
(if it does then I guess the problem is resolved)
clearly we need it to work for: 0x017f, 0x02ff00 and 0x03ffff00 at the very least
It says in a comment here that CScriptNum is coded to accept up to 5-byte bignums, which are good until 2**39-1, well beyond the 2**32-1 limit of the nLockTime field itself.
https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp#L355-L358 // If we kept to that limit we'd have a year 2038 problem,
// even though the nLockTime field in transactions
// themselves is uint32 which only becomes meaningless
// after the year 2106.
//
// Thus as a special case we tell CScriptNum to accept up
// to 5-byte bignums, which are good until 2**39-1, well
// beyond the 2**32-1 limit of the nLockTime field itself.
const CScriptNum nLockTime(stacktop(-1), fRequireMinimal, 5);