Topic: mastercoind proposal to speed development: Please read, vote, and comment - page 2. (Read 6059 times)

For transparency, here's my PM exchange with grazcoin regarding potentially using his library (before I had checked out zathras stuff). grazcoin gave me permission to post:

Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs   With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.

Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Please don't be discouraged by any of the posts (mine included) - I'm all for having more people & development on board

Re the name, that's simply a label but seems to have been adopted by the community at large as the naming for a given projects' daemon - hence my desire to keep it free for when a reference is officially nominated - for my own personal opinion I'd actually like to see 'mastercoind/qt' as a reference client in it's sincerest form - ie without any external dependencies (websites or bitcoind etc).  Mastercoind/qt should be a standalone reference client that connects to the bitcoin network and performs all the necessary bitcoin functions as well as Mastercoin functions without needing anything else.  I see a fork of bitcoind/qt as the simplest (not necessarily best, but simplest) path to this, but that's just my two cents.

Re the libraries, none of them is 'the' correct library at present - for one I can tell you my stuff alone is not ready for stable - the functions do what they're supposed to but we're always coming up with new fringe cases and bugs that can knock things off track.  We currently develop to a consensus where by we discuss & agree areas of the spec that are open to interpretation (or just outright suggested changes/additions) and the multiple different implementations are critical to using RAD (rapid application development) techniques (basically everything can change fluidly with no change control) in these early stages.  

For example only yesterday when I was about to release the new version of my library (and thus broadcast a few example transactions) did I find out my implementation was dropping a byte in the minfee as we crossed packets and not prefixing a leading 0 for the timelimit int if the hex was single digit.  Using my own implementation I would never have known (as I was encoding/decoding the same way), but being able to test against Tachikoma's meant I was able to pick up the error and fix the bug.  This happens A LOT  For example Mastercoin-explorer & Masterchest may differ on a transaction so Tachikoma & I will investigate - sometimes he's got a bug, sometimes I have - but it's those differing implementations that actually help us to identify those bugs & correct them while everything is so green.  Tachikoma even has a verification API to automate these checks for differences coming I believe.  EDIT: Sorry, not to downplay Grazcoins & Bitoys work either, I check my errors against their code too

TL:DR; contribute what you think will be of value.  If Grazcoin's libraries give you a good foundation for something you want to build & you think it will benefit the community, then go right ahead  Just try to keep in mind we're young and there are still multiple ways of doing things - we're not at the stage of saying any one implementation is the 'right' one yet and need to be careful about community confusion.

P.S.  Re Linux support I have actually tested (and rewritten in places) my wallet to work under wine on Linux.  I'm only testing with a couple of distros while I remove the kinks but to date I've worked around most bugs and have it working successfully (just with the odd out of place/badly colored label etc).  I haven't done this with the engine yet but since I'm not using as many of the cryptographic providers or doing any GUI work it should be reasonably easy to get going if that became a requirement.

Thanks!

And actually, I'm going to stop my work on this until I have a clearer direction of which library/libraries I should be using, and if Mastercoin even wants this. I was a bit premature, sorry. Just let me know.

Good point on the "mastercoind". I'm good with whatever JR says here. My goal with this thing was to get it to a point and turn it over anyhow.

The optimal library for this purpose would be python, would interface with bitcoind via JSON RPC (so it had good cross-platform compatibility), and would take care of encoding/decoding of transactions, and maybe even processing to a embedded leveldb database (if it didn't do the latter, I could add that easily enough).

We don't have this optimal case today, so I have a few options:

* Zathras' libraries -- Windows-only it appears, but sound like it's the right focus. I just skimmed your code. If I found a way to interface the DLLs well to Python (I think it's possible), I could use them under Windows. Looks like I could use masterchest-library alone if I wanted to use something different than SQL for the serialization (e.g. write something higher level that used that to parse the blocks and serialize to leveldb, for instance), or combine in using masterchest-engine and a SQL backend. I really do like how you're interfacing around bitcoind here.

* grazcoin's stuff looks good for Linux, at least. However, it's still a bit rough around the edges for working with as a 3rd party library... mostly around how it's coupled to the web site interface he has, and how he serializes things. I have a few messages to him, still undecided here (the good news out of this is that I made some setup and init.d scripts that will be included in 'sx' though).

* A clean port of masterchest-library to python is another option here, too, since that interfaces with bitcoind, it could be used on both platforms.

* I looked at Tachikoma's stuff too.... ruby though (same situation).

Optimally, I'd want to use the same library and serialization strategy under both Windows and linux, at least for starters.

It may seem like this is a piece of cake, but my goal is something that is very easy to set up and works across multiple platforms, like bitcoind. With that, things become a lot more complicated - code/script wise. I already have a good amount of work into this (and will share my code once I get a basic version of it up), but if this is something one of you thinks you could do up to this level, and make available for both windows and Linux, just let me know and I'll stop my work on it. I don't want to waste your time or my time, I just want to see this done, which is why I started working on it.

Basically, guys, I'm asking for your advice. I'm willing to write up against something I can use for both Windows and Linux (or possibly take two different approaches). I'm also willing to package this up and make it easy to deploy on both platforms. Just let me know how I can best work with you all to get this done, and what you think the best library choice is for me here, given my parameters.

If this starts becoming political, or if I've ruffled feathers with this, then I'll just dump it on a github repo for folks to potentially pick over and move on. I'm not looking to establish a name or identity in any of this, I just want to see it done in one form or another, to help Mastercoin out, as well as my investment in it. That's all.

Hey rbdrbd and welcome to Mastercoin development

I thought I'd chime in on a couple of points - I have to agree with Tachikoma on this one, whether or not you state it's the reference implementation, the choice of name (mastercoind) will lead some members of the community to perhaps think it is.  I would love to see any reference implementation written by JR too.

Simply put, the project will likely have a reference mastercoin daemon at some point and I think it would be courteous to leave the 'mastercoind' moniker for that purpose - you could always rename your output binary later if Grazcoin/your work ended up becoming the reference implementation.

Good luck with your wrapper!

P.S. (sorry for plugging my own stuff in your thread ) but masterchest_engine & masterchest_library have been around for a while to offer devs a low barrier to entry - the library already does decoding/encoding/parsing/bitcoind integration etc on behalf of the dev and the engine processes all transactions to arrive at a known/current state so for example web devs can query balances/transactions/exchange state etc without knowing anything other than a bit of SQL.  They're of course works in progress but since your post kind of implies before your wrapper devs had no choice but to start from scratch, I wanted to clear up that misconception.

Thanks!

rbdrbd,

Thanks for taking the initiative on this effort. I agree that a reference client for development purposes can be very helpful to new developers getting involved in Mastercoin.

I hear where Tachikoma is coming from with his concerns though and lets make sure that what gets developer here is sensitive to consensus from developers and the community. I wonder if the reference client would be another good place for the Proof of Stake voting to be involved or if the community thinks we should keep Proof of Stake at the Protocol Spec level?

I'm sure J.R. will chime in on the effort. But not likely today, as he does the family thing on the weekends and isn't on the computer.

I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.

Actually, I just did see that bitcoind went to leveldb as of 0.8, so I moved us to using leveldb as well for this (I do like NoSQL a lot better than SQL...so much easier to deal with).

I agree with having a Windows and MacOSX build. However, building for Windows is hard right off to bat because of 'sx', which grazcoin's mastercoin library uses. There will need to be a decent amount of work put into creating a build script for windows that will get everything built and setup. Once that is available, then a windows version should be no problem. At first, Ubuntu is the first target because I know it well, and it's much easier to build things like this under it (compiling from source on windows --- especially in an automated fashion -- can be a big pain in the ass).

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed. 

Ah sorry I missed that you're wrapping grazcoin's implementation.
That's awesome then

+1 for repackaging one or more of the existing implementations and providing various APIs to it.
APIs should be HTTP, RPC, and more ... the more endpoints we have, the more useful an implementation gets.

I'd like to consolidate "development firepower" on one or a few implementations, not to create yet another Mastercoin client.

+1000

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

For what it's worth, I fully support this endeavour.

I too stayed up al night yesterday working on a project, nothing close to mastercoin/bitcoin related though. I am glad you are already leading this charge, i commend you for this. Any upcoming technical difficulties that you foresee on your path? and why not build for Windows and OSX first? the masses...and any particular reason that you choose relational databases? there are a lot of drivers for NoSQL based databases u know ..just curious.

I started development of this yesterday and have made a ton of progress after almost an all-nighter :p. Python using sqlite and cherrypy. Using grazcoin's mastercoin-tools for the mastercoin ops right now (which depends on sx). Eventually I'd like to move to bitcoind for better compatibility under Windows (or get sx compiling under windows -- looking into that), but for now this will work and will allow us to get off the ground much more quickly (grazcoin has done a good job).

The target at first is Ubuntu 13.10 (sx has major build issues with 12.04 LTS), and I have a full build script that will do everything from a stock system install.

At a minimum, even for Windows folks, one could set up a VM with Ubuntu 13.10, install mastercoind to that, and access it from your windows boxes across your LAN. I'll have a step by step how to set it up, once you get the base Ubuntu install on a VM.