What about making your stack modular such that you could swap in and out alternative components.
For example, use portions of btcd instead of libbitcoin.
Topic: Combine projects: libbitcoin/obelisk libraries, Electrum, and Hive (Read 9484 times)
November 02, 2013, 02:57:31 AM
November 01, 2013, 11:11:15 AM
I've had the feeling this was the direction we've been moving in for a while. Now the eager response of Hive is a welcome addition.
The Linux operating system came together when Linus' kernel was combined with the GNU system. If we wish to build a resilient and successful opensource ecosystem, then it will need the combination of projects responsible for a small piece of turf. It will force us to serve each other and create something very powerful.
When I got involved with Electrum, my goal was to replace the backend with libbitcoin. I saw Electrum as the future. Electrum's community and technology get me very excited. It's an open community, with lots of solidarity (despite the small size) and people helping out with infrastructure in a decentralised self-organised way.
As I started to develop on Electrum (which involved refactoring the server code to support pluggable backends), I became concerned with the UI. Over 2-3 months I did my own usability studies with friends and people, and started to focus on how Electrum should look like. Then there was more refactoring of the client code so we could integrate another GUI. During this time also saw Animazing becoming involved in development and he also had lots of ideas.
The original libbitcoin Electrum backend, was a fullnode integrated straight into Electrum server. However I didn't believe this was a scalable model. After the bitcoin2012 conference, I took a big break from Bitcoin for some months (the conference was too much). During this time, I started to formulate new ideas about a backend, slowly hacking prototypes at first, theorising, writing more code, rewriting, trying different approaches, ... The moving from Apache Thrift to ZeroMQ was a massive leap. I have a hero worship problem with the ZMQ creator Pieter Hintjens
I think he's a genius.
http://zguide.zeromq.org/page:all#toc7
And obelisk was born.
The community around libbitcoin started to grow, and we slowly developed our software. We started to plan between ourselves the tools and code needed to build a functional wallet. I still wasn't sure whether we'd go with Electrum or make something new. I remember when trying to integrate the light GUI into Electrum, it wasn't so modular (tight coupling between GUI and wallet).
Then we had the Barcelona Electrum developer meeting, and Electrum's new release was impressive. Thomas had re-architected Electrum into a modular framework, and was rapidly integrating features like HD wallets. We also saw the Electrum Android preview from Andy which was awesome. Never seen such a great wallet on mobile phones - and it's based off Electrum and Python! Andy is working with Kivy developers to build the thing.
The new aim for Electrum is a modular framework for creating wallets. The framework provides the features and you just hook them into an interface. Thomas demoed a terminal Electrum app to illustrate this.
I spoke with Pablo because I wasn't sure how to support Electrum. Before my intention had been to make an adapter that provided the current Electrum API (so clients could remain compatible with minimal changes to the Electrum codebase), but this was sub-optimal and doesn't take advantage of the parallel asynchronous scalability features (which needs some knowledge from the API user to use effectively). Also with multiple workers you need to be able to handle failure cases to retry a different host or know how to properly query results.
Pablo suggested the way forwards in replacing the client side Electrum blockchain query code with obelisk API. This was a problem though since Electrum is a Python-only project and the current obelisk was a C++ library. Also even if we made a Python obelisk then it will still depend on ZMQ. Since there was the Electrum Android Kivy project (and we're interested in phones too), we tried building ZMQ on the mobile phone. It was a nightmare and we never got it working (I don't think it's ever been managed). Pablo did the unthinkable and rewrote ZMQ in pure Python! Pieter Hintjens (project maintainer for ZMQ) was excited and wants to make it an official part of ZMQ. So we have a pure Python obelisk library.
By moving some of the Electrum core utils (such as bitcoin.py) into his project, we have a full Python wallet software library. Also Vitalik made Pybitcointools for working with keys, and Robert (a libbitcoin contributor) created a good abstraction layer for the blockchain which makes it look like a Django database ORM (for the blockchain). So we can move this functionality into a library, and prefer friend's code (so they own that piece of the ecosystem).
The technology in Electrum is solid, but it needs a decent UI. My only concern is that if we have worker -> balancer -> desktop core -> interface, that's a lot of overhead for transferring wallet history but maybe it's not such a big deal in the end. Also I'm working on some important issues to do with the blockchain database. I've been developing a custom database.
This is database read and write speed for Bitcoin transactions vs LevelDB. Note these results are merely illustrative. There's a lot more complex things happening underneath that's difficult to explain (I will write a manual). But by exploiting the properties of the blockchain leads to some interesting results such as hot backups, lock-free reads and lack of a separate database cache (instead use madvise() to tell the kernel how to cache from disk). Anyway this is experimental stuff. Read speed is my priority.
Together we can create a big group of talented developers freeing up the others to focus on our particular development interests. I see the potential in Hive for security features and concepts Thomas is working on in Electrum. There is the old conflict between the top-down cathedral with the ability to mobilise huge resources through command hierarchies, vs the bottom-up bazaar that provides the creative and open environment for merit to rise to the top. Debian is the Linux kernel, the GNU system tools, the graphics layer, the Gnome desktop environment and the various applications. I am optimistic that we can create the same resiliency and strength for the future of money.
The Linux operating system came together when Linus' kernel was combined with the GNU system. If we wish to build a resilient and successful opensource ecosystem, then it will need the combination of projects responsible for a small piece of turf. It will force us to serve each other and create something very powerful.
When I got involved with Electrum, my goal was to replace the backend with libbitcoin. I saw Electrum as the future. Electrum's community and technology get me very excited. It's an open community, with lots of solidarity (despite the small size) and people helping out with infrastructure in a decentralised self-organised way.
As I started to develop on Electrum (which involved refactoring the server code to support pluggable backends), I became concerned with the UI. Over 2-3 months I did my own usability studies with friends and people, and started to focus on how Electrum should look like. Then there was more refactoring of the client code so we could integrate another GUI. During this time also saw Animazing becoming involved in development and he also had lots of ideas.
The original libbitcoin Electrum backend, was a fullnode integrated straight into Electrum server. However I didn't believe this was a scalable model. After the bitcoin2012 conference, I took a big break from Bitcoin for some months (the conference was too much). During this time, I started to formulate new ideas about a backend, slowly hacking prototypes at first, theorising, writing more code, rewriting, trying different approaches, ... The moving from Apache Thrift to ZeroMQ was a massive leap. I have a hero worship problem with the ZMQ creator Pieter Hintjens
I think he's a genius.http://zguide.zeromq.org/page:all#toc7
And obelisk was born.
The community around libbitcoin started to grow, and we slowly developed our software. We started to plan between ourselves the tools and code needed to build a functional wallet. I still wasn't sure whether we'd go with Electrum or make something new. I remember when trying to integrate the light GUI into Electrum, it wasn't so modular (tight coupling between GUI and wallet).
Then we had the Barcelona Electrum developer meeting, and Electrum's new release was impressive. Thomas had re-architected Electrum into a modular framework, and was rapidly integrating features like HD wallets. We also saw the Electrum Android preview from Andy which was awesome. Never seen such a great wallet on mobile phones - and it's based off Electrum and Python! Andy is working with Kivy developers to build the thing.
The new aim for Electrum is a modular framework for creating wallets. The framework provides the features and you just hook them into an interface. Thomas demoed a terminal Electrum app to illustrate this.
I spoke with Pablo because I wasn't sure how to support Electrum. Before my intention had been to make an adapter that provided the current Electrum API (so clients could remain compatible with minimal changes to the Electrum codebase), but this was sub-optimal and doesn't take advantage of the parallel asynchronous scalability features (which needs some knowledge from the API user to use effectively). Also with multiple workers you need to be able to handle failure cases to retry a different host or know how to properly query results.
Pablo suggested the way forwards in replacing the client side Electrum blockchain query code with obelisk API. This was a problem though since Electrum is a Python-only project and the current obelisk was a C++ library. Also even if we made a Python obelisk then it will still depend on ZMQ. Since there was the Electrum Android Kivy project (and we're interested in phones too), we tried building ZMQ on the mobile phone. It was a nightmare and we never got it working (I don't think it's ever been managed). Pablo did the unthinkable and rewrote ZMQ in pure Python! Pieter Hintjens (project maintainer for ZMQ) was excited and wants to make it an official part of ZMQ. So we have a pure Python obelisk library.
By moving some of the Electrum core utils (such as bitcoin.py) into his project, we have a full Python wallet software library. Also Vitalik made Pybitcointools for working with keys, and Robert (a libbitcoin contributor) created a good abstraction layer for the blockchain which makes it look like a Django database ORM (for the blockchain). So we can move this functionality into a library, and prefer friend's code (so they own that piece of the ecosystem).
The technology in Electrum is solid, but it needs a decent UI. My only concern is that if we have worker -> balancer -> desktop core -> interface, that's a lot of overhead for transferring wallet history but maybe it's not such a big deal in the end. Also I'm working on some important issues to do with the blockchain database. I've been developing a custom database.
This is database read and write speed for Bitcoin transactions vs LevelDB. Note these results are merely illustrative. There's a lot more complex things happening underneath that's difficult to explain (I will write a manual). But by exploiting the properties of the blockchain leads to some interesting results such as hot backups, lock-free reads and lack of a separate database cache (instead use madvise() to tell the kernel how to cache from disk). Anyway this is experimental stuff. Read speed is my priority.
Together we can create a big group of talented developers freeing up the others to focus on our particular development interests. I see the potential in Hive for security features and concepts Thomas is working on in Electrum. There is the old conflict between the top-down cathedral with the ability to mobilise huge resources through command hierarchies, vs the bottom-up bazaar that provides the creative and open environment for merit to rise to the top. Debian is the Linux kernel, the GNU system tools, the graphics layer, the Gnome desktop environment and the various applications. I am optimistic that we can create the same resiliency and strength for the future of money.
November 01, 2013, 05:10:30 AM
I am glad that you noticed my efforts to make Electrum modular, and I welcome this idea.
For me, the next step is indeed to have a socket api between the core and gui.
I am doing this so that I can focus on the core of Electrum library, while leaving as much freedom as possible to gui developers.
For me, the next step is indeed to have a socket api between the core and gui.
I am doing this so that I can focus on the core of Electrum library, while leaving as much freedom as possible to gui developers.
Thomas are you coming to Milan?
November 01, 2013, 05:08:44 AM
I am glad that you noticed my efforts to make Electrum modular, and I welcome this idea.
For me, the next step is indeed to have a socket api between the core and gui.
I am doing this so that I can focus on the core of Electrum library, while leaving as much freedom as possible to gui developers.
For me, the next step is indeed to have a socket api between the core and gui.
I am doing this so that I can focus on the core of Electrum library, while leaving as much freedom as possible to gui developers.
October 31, 2013, 07:01:25 PM
Though the devil is in the details, we could potentially get behind something like this. You are rightly identifying the strengths and weaknesses of each group. Our long-term goal is to make a product that is ethically sound, easy to use, and which defaults to excellent privacy and security decisions for the user. If we can accomplish that together, we most likely should.
October 31, 2013, 04:43:01 PM
Hi,
All 3 of these projects are lacking things that we each have.
The server part of the libbitcoin stack is well developed, but we're still working towards client-side software for managing wallets. I was very impressed by Thomas' presentation about how he's turning Electrum into a framework. However the backend for Electrum is lacking, and also the servers are unreliable. We have built the client side libraries (Pablo's pure Python ZeroMQ + Pablo's obelisk-client + Vitalik's Pybtctools + Robert's BlockAlchemy). This removes any need for dependencies (pure Python) and can easily be deployed on Android (through Kivy).
I'm imagine a core on the desktop with an API over sockets that can only be called on localhost (although maybe do something else). Exchange the Electrum interface and start using the new library (move Electrum code over to it too, and provide a nice API).
Hive is even further in front of the user. The Electrum user interfaces suck (despite the technology being very good). And they're really thinking on different things. If we have a separate Python core with a socket, then it's possible to integrate with their C#. They already have their server backend and Electrum has one too, and so no point us all duplicating the same work when I've been doing backend/impl stuff for ages.
libbitcoin - server / core API stuff
* libbitcoin/obelisk server - me, Robert Williamson
* Client side library: Pablo, Vitalik and Robert.
** ZMQ (networking library) Python port by Pablo.
** Daemons like broadcaster or mixer written in Python for rapid development. This is one of Pablo's specialities.
Electrum - features, key management
* ThomasV, Animazing
* wallet, interface API - ThomasV
Hive - interfaces
* Wendell, Taylor Gerring
We can search for some designers, and we can provide infrastructure like servers and stuff. And contribute towards paying for packaging, Q&A, testing, ...
For the network stuff, I have help from Hintjens. He wants to make Pablo's Python an official port of ZeroMQ. The origin for ZeroMQ is from when he was working on networked financial software.
All 3 of these projects are lacking things that we each have.
The server part of the libbitcoin stack is well developed, but we're still working towards client-side software for managing wallets. I was very impressed by Thomas' presentation about how he's turning Electrum into a framework. However the backend for Electrum is lacking, and also the servers are unreliable. We have built the client side libraries (Pablo's pure Python ZeroMQ + Pablo's obelisk-client + Vitalik's Pybtctools + Robert's BlockAlchemy). This removes any need for dependencies (pure Python) and can easily be deployed on Android (through Kivy).
I'm imagine a core on the desktop with an API over sockets that can only be called on localhost (although maybe do something else). Exchange the Electrum interface and start using the new library (move Electrum code over to it too, and provide a nice API).
Hive is even further in front of the user. The Electrum user interfaces suck (despite the technology being very good). And they're really thinking on different things. If we have a separate Python core with a socket, then it's possible to integrate with their C#. They already have their server backend and Electrum has one too, and so no point us all duplicating the same work when I've been doing backend/impl stuff for ages.
libbitcoin - server / core API stuff
* libbitcoin/obelisk server - me, Robert Williamson
* Client side library: Pablo, Vitalik and Robert.
** ZMQ (networking library) Python port by Pablo.
** Daemons like broadcaster or mixer written in Python for rapid development. This is one of Pablo's specialities.
Electrum - features, key management
* ThomasV, Animazing
* wallet, interface API - ThomasV
Hive - interfaces
* Wendell, Taylor Gerring
We can search for some designers, and we can provide infrastructure like servers and stuff. And contribute towards paying for packaging, Q&A, testing, ...
For the network stuff, I have help from Hintjens. He wants to make Pablo's Python an official port of ZeroMQ. The origin for ZeroMQ is from when he was working on networked financial software.
Jump to: