Topic: Why is OpenSSL needed in the official client? - page 2. (Read 4115 times)

HOLY!!!!!

Thanks for this heads up!!!!

I do not know if this was mentioned already but it is important. Any bitcoin implementation that does not use OpenSSL must be compatible with OpenSSL if it is doing full validation of the block chain. The reason is because OpenSSL has a broken version of ECDSA, ie. it doesn't follow the ECDSA standards and does it own thing. Other ECDSA implementations may not take the OpenSSL differences into account and if a bitcoin implementation does not successfully implement ECDSA as OpenSSL does, then it could allow someone to create a fork between the two implementations.

I've done that myself. I have a few projects where all I have are a handful of blobs, without any structure to the data. I still use SQLite because it is transactional and preserves the integrity of the writes, without me having to roll my own solution. LevelDB is a good alternative for that as well but if you ever want to start using relational database features then you'd either need two DBs or migrate code (both of which are acceptible, if you absolutely need relational features).

Good for you, but it certainly doesn't convince me for the wallet use case. The data being stored in there consists of complex data structures (keys, transactions), for which a serialization framework already exists. The only thing being written would be byte sequences. The advantage of SQL-based systems is easier aggregation of data from the database, but if the database layer can't inspect the data, that doesn't make much sense.

You could write an SQL-based wallet implementation that splits out wallet information over several SQL tables (I believe genjix' libbitcoin does that), and take advantage of what SQL adds, but at that point I think you need to rewrite the wallet implementation pretty much from scratch.


I still prefer not needing to integrate anything at all.


Yes, this is what we're going to do for LevelDB, as it doesn't even come as a dynamic library. Dependencies certainly complicate development - especially for portable applications - but in some cases there is just no better solution.

Agreed - the simpler the better IMO (have developed my own object DB but have yet to make it ACID).


Also agreed - you might be interested to take a look at the DB I developed as it works with streaming operators and fits very nicely with C++.

I use SQLite for all document formats, no matter how simple. Because it is transactional, robust, and performs well. I have taken the soci C++ wrapper and remodeled it to provide a very nice system of binding to C++ that takes advantage of all the object oriented metaphors. So it is simple to map primitive data types and object types to and from database entries.

As for integration and dependencies, SQLite is quite easy to integrate it comes as a single .c / .h file pair that you just add to your existing project. No Makefile, no build settings, nothing.

Let me point out that I don't like external dependencies. My opinion is that a repository should stand on its own. This is done by bringing in the sources for external dependencies directly into the source tree for the repo (I use "git-subtree" for that). I'm not a fan of using dynamic libraries at all. For desktop applications I much prefer to have everything "baked in" to a single .exe. Then there is no question about what people are running. Yes it is true that if there is a security vulnerability the software needs to be updated (you can't just update some shared .DLL) but I prefer this over the alternative.

To build an application it should be possible to clone the repository, open the associated IDE project file (Xcode or Visual Studio), press "Build", and get a correct result. Nothing more. No extra tools (no Python, no CMake, no Perl, nothing).

In my opinion, even sqlite is overkill for the wallet. It's yet another dependency (something you don't like, right?), and all we need is a simple key-value store that is read at startup and loaded into memory. Probably we'll move to a very simple custom append-only format with checksums.

For the blockchain: performance. LevelDB is exactly what we need: not more than a key-value store with atomic writes, with very good performance and consistency.

Interesting - my problem was actually with multi-threading not different processes - will have to look into that again if they have improved the locking (I last looked at that a few years back).

SQLite works great with multithreaded apps you just need to set the appropriate locking model option. If you mean concurrency in the sense of different processes accessing the same database then yeah I agree (but that's not SQLite's target audience).

Although not familiar with LevelDB I can say I was quite impressed with SQLite (used to use it for my own project although unfortunately it's not designed for concurrency due to its rather global locking approach).

Why not sqlite for the wallet (and maybe the blockchain) ?

Faster, less prone to corruption (in our setting), better compatibility between versions. At least, hopefully.

Also see this SE question.

What are the advantages of LevelDB vs BDB?

The tool for VC++ that I am using just scans all the quoted #include's from the source files to work out the dependencies.

Provided you stick to a consistent coding style it solves this problem easily and without anything weird (just requires a couple of tools and the makefile template).

To be more specific lets look at the official client sources. The qt/ directory contains all of the source files for the associated user interface classes:

https://github.com/bitcoin/bitcoin/tree/master/src/qt

We have

aboutdialog.cpp
aboutdialog.h
addressbookpage.cpp
...
walletmodel.h

Right now each of these is individually mentioned in the Makefile. What we could do instead is create a new "unity" .cpp / .h pair of files and manually #include these sources:

qt_ui.h

qt_ui.cpp

Given, it looks a little strange with .cpp files as includes, but the results are quite impressive. You can remove all of the #include lines from every associated .cpp and .h and just merge them into one non-repeated group of includes at the top of qt_ui.h and qt_ui.cpp. Preferably the bulk of includes would go into qt_ui.cpp (so they have a smaller scope).

It is instantly possible to see what the group of classes external include dependencies are (they are all in one place).

Adding, renaming, or removing individual UI component source files is done by editing the .h and .cpp, no changes to the Makefile are required. All of the individual .cpp / .h in the Makefile are replaced with the one qt_ui.h and qt_ui.cpp pair.

It is true that the granularity of compilation now becomes this larger set of classes, but the trade-off is well worth it.

Link not working in China (perhaps blocked?) - will try and have a look at that tomorrow though.

Okay - the system I have developed works with an XML like "makefile" that is minimal (only source files and required libraries for each dependency).

It then uses OS specific "templates" that take it's structure and turn it into an actual platform specific makefile (my system can do VC++, BCB and g++ makefiles currently).

Maintenance of the system is actually very minimal (don't think I've had to change it much in the last few years actually). Header dependencies (only for VC++ in my system as both BCB and g++ handle these automatically) are handled via a specific tool (also not that complicated).

Unity build is a system for organizing source files, it is not a separate tool and doesn't affect your Makefile or project file, except to make it orders of magnitude smaller. You can read about it here:

http://buffered.io/posts/the-magic-of-unity-builds

If you have Visual Studio or Xcode you can open up my "SimpleDJ" application (in my sig) and see how this all works.

I fully agree, and I think the other developers do as well.

It's just that all experiments with more automated build systems failed - either they didn't work for all currently supported platforms, or weren't maintained after being written.

Again, help is welcome.

I'm sure there are probably several good build systems out there - am comfortable with my own as it works perfectly for Windows and Linux - but manually putting together a large makefile is definitely a huge PITA (especially if it needs to be specifically told about headers).