Topic: Why is OpenSSL needed in the official client? (Read 4060 times)

Thanks to everyone for their informative replies, this has provided useful insights into some of the thinking that goes on in the development process.

Actually ever since Herb Sutter joined MS their compiler has been extremely standards compliant (although you do have to flick some compiler switches to get it to adhere correctly).

I don't actually use the IDE myself (being a console guy). The only big problem with not doing that is the lack of auto-dependencies (at least in the free version) although it isn't really that hard to construct your own make system (which I did for my own project) and there are other free ones out there.

Not too surprising, it's one hell of a weakness right now. BDB is just not the right fit for how we use it.


I was talking about the storage system, not wallet semantics (which is what you want changed to conform to those rules). Even if we move to an append-only wallet file, that doesn't mean anything will observably change.


Mike Hearn also implemented a first Bitcoin-on-LevelDB port himself (see pull request 1619), which was abandoned after I modified it to work on top of my rewrite of the validation engine (see pull request 1677).

The problem you probably were referring to is the fact that Bitcoin relied on reading uncommitted data during block validation, something that isn't supported by LevelDB (it just has atomic writes, no real database transactionality). Mike solved that in his port by writing a tiny caching layer around LevelDB. I solved it by avoiding the need for such operations altogether, with a nice performance improvement along the way.


He wasn't talking about the storage layer. He was referring to your request for changing the wallet semantics. I disagree that it would take that long, by the way, but I disagree it's within our scope right now. There are enough alternative wallets already, those are in a perfect place to experiment with different types of wallets.

By the way, changing the wallet storage (but just that) to use an append-only format is also already implemented (by me), but it had some issues left, and I felt other things were more important to work on.

That's simply false.

Keywords such as "slots:" are defined away with a macro when compiling the actual code. There is no specific preprocessor for Qt (there is the "moc" compiler, but it adds class introspection information and such that is linked as a separate file, it does not transform source code).

I have compiled Bitcoin-Qt with visual studio 2010 express. I have no time to maintain a parallel build system though, mingw works perfectly for our purposes.  I'm using Qt Creator as IDE, which is similar enough to VS but without need to buy an expensive license after 30 days.

But if you'd like to maintain a VS build system I can send you what I have.

BTW@ 2112 you can use bitcoin with leveldb right now if you want. See pull request https://github.com/bitcoin/bitcoin/pull/1677

It seems like almost every technical thread about bitcoin{d,-qt} needs to take a detour into the DB-land.

I just wanted to stress that the "append-only" is the key concept to understand what is required architecturally to implement Bitcoin efficiently. Incidentally it is also a key ingredient to make any Bitcoin implementation GAAP-compliant.

It is alway scary experience to post here a link to some Microsoft's web property. But this thread already has at least 2 posters who aren't scared of MSFT, so here it goes:

http://blogs.msdn.com/b/pathelland/archive/2007/06/14/accountants-don-t-use-erasers.aspx

LevelDB unfortunately will not be "exactly what we need" unless a significant reachitecturing is undertaken.

Mike Hearn had explained this succintly. I'll find the link and post it here.

https://bitcointalksearch.org/topic/m.1048149

Gavin Andressen had estimated that the rearchitecturing of storage layer could take up to 3 years anywhere from 1 to 3 years while seriously impacting other intended deliverables. Again I'll find the link and post it here.

https://bitcointalksearch.org/topic/m.1170970

Right. SEC defines the compressed (0x02/0x03 + 32-byte X coordinate) and uncompressed (0x04 + 32-byte X coordinate + 32-byte Y coordinate) formats, but doesn't specify the hybrid format (0x06/0x07 + 32-byte X coordinate + 32-byte Y coordinate). From that link, it seems that the hybrid format may have been specified by that ANSI X9.62, but I won't spend $100 to buy the standard...

Anyway, it's that hybrid format that's bothersome. Making an implementation support it is trivial though - just change the 0x06 or 0x07 byte to a 0x04 and pass it that way to the crypto library.

I just learnt this from the OpenSSL source code, and the SEC document (see the secg site).

Pieter knows moe about this. Basically OpenSSL has the uncompressed and compressed ECDSA public key formats which are defined in the SEC standards but it also has a hybrid format. http://openssl.sourcearchive.com/documentation/1.0.0e-3/crypto_2ec_2ec_8h_aa2c7ec2902c397d59393d357921e14b5.html Literally that is all the documentation gives you. I have no idea where Pieter Wuille learned all this stuff.

There's a function DSAConvertSignatureFormat() which handles the various formats. Not sure about the hybrid keys, can you go into more detail?

Thanks

And it supports the hybrid keys too?

Well regardless I looked into CryptoPP and it supports all the standard ECDSA signature formats as well as the odd one used in OpenSSL. So it would be a suitable alternative to OpenSSL (for ECDSA).

Yes, a lot (close to 100000, IIRC).

We're not doing signature verification right now for things buried deep enough in the chain and protected by a checkpoint. Once a significant part of the chain has only standard signatures left (I haven't seen any non-standard pubkey, except on testnet where I tried it myself), code to support the non-standard signatures is only required for implementation that want to do full validation of the entire history (which, imho, there should be). It's more of a benefit for lighter types of clients, which don't verify history.

You could do that and hard code unspent outputs (That exist before the checkpoint) for validation purposes.

I think a new client could include a check point, validate the block hashes and then just assume that any tx / sig was valid (without checking it) if it precedes the check point?


Yeah I'm not saying backward compatibility is bad, but that it is already necessary for such a young project.

So have these non-standard keys or signatures already been included in the block-chain? That would cause a problem if ever a protocol change was wanted. You'd be stuck requiring to validate the signatures and keys as OpenSSL does it.

I think users with older clients, holders of older bitcoins quite appreciate the struggle to maintain backwards compat.

Nobody wants to wake up in the morning, to discover that their money is unspendable outside of a required upgrade.

Sad that for such a new piece of technology, we're already struggling to maintain backward compatibility with older hacks.

Rejecting keys and encodings which are not compliant with SEC standards in blocks could cause forking problems. There would need to be miner support to prevent the bad signatures or keys being included in blocks, before a client change was to take place.

Obviously you must know this already but I was just mentioning it.

It would be an improvement to get this through.

Yes, really. I'm on Windows. And I consider downloading and building external dependencies to be part of the build. Looking at build-msw.txt, I see:

Need to download boost, miniupnpc, BDB, and OpenSSL
Follow the build procedures for each of these. OpenSSL in particular requires Perl, a bunch of other stuff.

Then there's mingw32-make. What's that? I don't have that, I use Visual Studio. MinGW is a hassle.


Trolling aside, Visual Studio is currently the most productive environment for building C++ desktop applications (for me at least). At least once a year I evaluate the alternatives and nothing has dethroned it yet, although I desperately wish it would happen.

That's somewhat of an exaggeration. Let me explain.

Bitcoin uses DER encoding for signatures and EC private keys, and a serialization defined by the SEC2 standard for EC public keys. OpenSSL follows both these standards perfectly. It creates fully compliant ones, and parses them perfectly.

However, as DER-encoded signatures and SEC encoded public keys end up in the block chain, they are effectively part of Bitcoin's (implied) protocol specification as well.

Unfortunately, OpenSSL also supports signatures that do not follow the strict DER encoding (it supports anything encoded in BER, and even that is not required). For public keys, it accepts both encodings defined by SEC, plus a weird third own one. Because these non-standard signatures and public keys have always been accepted by every full node on the network (as those simply used OpenSSL), they became part of Bitcoin's implied protocol as well. For a long time nobody noticed this, but today there are alternative implementations that systematically create non-standard signatures for example.

There's an effort being done to outlaw these, as it will make the live of developers for alternative implementations easier. See pull request 1742 for more information.

Really?


Perhaps with optional build-openssl and build-bdb steps first, if your distro does not provide.  

If your OS makes it more difficult, get a better OS ;p