Pages:
Author

Topic: mastercoind proposal to speed development: Please read, vote, and comment (Read 6059 times)

sr. member
Activity: 266
Merit: 250
zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???
CBALANCE = Confirmed Balance
CBALANCET = Confirmed Balance (Test)
UBALANCE = Unconfirmed Balance
UBALANCET = Unconfirmed Balance (Test)


* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.
Yep - doing full state processing on every run is useful in these early stages for hunting bugs.  This was much less of an issue for the first version of the engine as processing only took a few ms.  As we scale out this is no longer viable so temp tables will be used (already are in my wallet).  On the to-do list Smiley

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...
Will consider this.  Whilst it was just me using the engine for masterchest.info obviously it didn't matter so much, but if others are going to make use of it I can see how this would be valuable.

* have a way for me to query the masterchest-engine version
No problems here - not difficult to add a -ver switch.

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 Huh lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration Cheesy)
Haha - the magic 999998 is for my unconfirmed transaction processing.  999999 is a dummy block number for an unconfirmed transaction.  Again as I mentioned below there is some cleanup to be done - this (along with a couple other tricks) was a quick and dirty way of supporting the display of unconfirmed transactions on masterchest.info.

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it
Doesn't store a timezone at all, it stores the blocktime as recorded in the blockchain.  You should do localization after retrieving transaction times from the db, not while storing them.

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?
As needed sure - I haven't run into any performance issues yet (none of my queries take longer than a few ms) which is why I don't have them, but they can be added without to much difficulty if needed.

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?
You can tell what kind of transaction it is from the type column (send, selloffer, acceptoffer etc).  The transaction encoding class isn't stored as I haven't come across a need for it.  Simple sends are the only supported Class A transaction type.  If you throw a simple send to mlib.getmastercointransaction it'll return the correct tx details regardless of whether it used Class A or Class B encoding.

It's not hugely disruptive to add it, but could I ask what you'd be using it for?


* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?
Assume you mean for the 'generate' transaction type - and yes, 'generate' transaction types include the early adopter bonus.

Hey rbdrbd,

Sorry for the delayed response, I was away for the weekend & when I got back the forums were down.  Yep the code is quite scrappy but functional and will be cleaned up as time allows.  Please see inline for responses to your Qs.

Thanks! Smiley
sr. member
Activity: 462
Merit: 250
zathras,

Got a few Qs/todo items for you:

* with balances, what is the difference between CBALANCE, CBALANCET, ubalance and ubalancet???

* it appears you clear out the transactions_processed table totally every run. If I have an API query going on during this time when it's cleared and being regenerated, I would return bad data. Can you regenerate into a temp table and rename, or come up with another way? I can pause transactions for now when this is being regenerated, but that's kind of ghetto.

* masterchest-engine sql versioning - have a version table or a way for me to deduct the sql schema version....auto schema upgrades in masterchest-engine would be best... worst case, if blow away the DB on schema upgrades, if necessary, and regenerate...

* have a way for me to query the masterchest-engine version

* an abundance of magic numbers -- makes the code confusing to read. e.g. what does "If .Item(6) < 999998 Then" mean??? What is that 999998 Huh lack of comments around these kinds of things. Also, things like "curtype = 1" -- maybe use enums instead? A future task I'm sure...I know you're busy getting Dex working (and my sqlite integration Cheesy)

* what timezone does masterchest-engine store -- we should probably force UTC time. Or, the timezone must be stored with it

* you have no indexes on any rows, columns of your database. This greatly can impact performance. Do you have plans to add indexes?

* from transactions_processed, how do I tell what kind of transaction it is (multi-sig, class A, etc)?? can you add this into the schema?

* also, for transactions_processed, does "VALUE" include the bonus amount for txns in the exodus period, or not?
Ola
sr. member
Activity: 311
Merit: 250
rbdrbd this is NXT's api..this should motivate you:    https://bitcointalksearch.org/topic/nxt-api-of-nxt-313082

They will probably have hundreds of apps at their disposal when the launch their exchange..I hope you are trucking on I am 100% behind you.
sr. member
Activity: 266
Merit: 250
Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.

I'll create a MasterchestCLI repo and push the source up to git after work.  FYI you can use either MSC or TMSC (test MSC) for the currency ID for now.  Perhaps I'll drop this straight to an int and whatever currency ID number is specified on the command line is passed straight to the encoding functions (rather than if ucase(cu)=MSC then curtype=1 etc).

I'll take a look at your SQLite schema when I add the support (again sorry but queued behind DEx bug squishing).

FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.


You're not kidding - driving me to tears (I run avast as well).  It seems a good portion of the binaries I spit out get flagged with a 'generic' virus match and deleted.  "Build failed" (because Avast deletes the output file) is a daily occurrence for me.

They're safe as houses - Avast have basically said because the EXEs I'm creating have no 'reputation' they are assumed to be bad.  It's an ongoing debate (I think they're well off track with this).  Just google Visual Studio & Avast and you'll see this is a fairly common issue.

In terms of end users, apparently I'm supposed to submit the binaries to Avast first so they an clear them and avoid false positives, though I'm really quite unhappy with that assertion.

To be continued...

Thanks! Smiley

sr. member
Activity: 462
Merit: 250
FYI, the MasterchestCLI.exe is showing up in avast! as infected with "Win32: Evo-gen [Susp]". Ran it through Virustotal.com and it looks like a false positive, but it could cause issues for folks with avast! on their computers.
sr. member
Activity: 462
Merit: 250
Zathras, great! Thank you! I will be integrating this and testing today and tomorrow.

By chance, do you have the source code on github for those CLI utils (perhaps as a part of masterchest-engine repo)? (wanted to check a few things, e.g. like if I could specify a currency ID instead of "MSC", for instance.)

Looking forward to SQLite support being added!

I've converted your schema to SQLite, it's at pastebin here: http://pastebin.com/39juANE1
^ please check and let me know if I missed anything...was pretty straight forward but I'm not super experienced with SQLite itself.

I'm thinking I'm going to build this thing to push through support for both Sqlserver and SQLite...so people can scale up or not depending on their needs.
sr. member
Activity: 266
Merit: 250
DEVELOPERS ONLY.  THIS IS NOT READY FOR END USERS.  THANKS!

Hey rbdrbd,

Here's the cli wrapper you asked for.  Just provides a binary to expose the library functions for encoding transactions.  Note there is no state or validation here, we're just creating the transaction.  The result can be signed & broadcast if you're happy with the output.

Please use with care, it's completely untested (there could be bad numeric conversions or other bugs for example) so only test against a wallet with fractional amounts Smiley

Let me know how you get on with it under Mono.  I'll take a look at putting sqlite support into the engine over the weekend, but for the next few days I really need to hunt down these bugs that are messing up my DEx states.

Thanks! Smiley

Built against .NET4.  Pick it up from here.  Note, use the -type=type switch to select between send, sell & accept.

Simple send:
Use the bitcoinrpc switches from the engine and -am (amount) -to (to address) -fr (from address) and -cu (currency type) switches.  For example:
Code:
MasterchestCLI.exe -bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=send -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -to=1MCHESTptvd2LnNp7wmr2sGTpRomteAkq8 -cu=MSC -am=1.234



Sell offer:
Use the bitcoinrpc switches from the engine and -am (sale amount) -fr (from address) -cu (currency type) -of (offer amount) -tl (timelimit) -mf (minfee) switches.  For example:
Code:
-bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=sell -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -cu=MSC -am=1.234 -of=0.5 -tl=6 -mf=0.0001



Accept offer:
Use the bitcoinrpc switches from the engine and -am (purchase amount) -fr (from address) -to (to address) -cu (currency type) switches.  For example:
Code:
-bitcoinrpcserv=127.0.0.1 -bitcoinrpcport=8332 -bitcoinrpcuser=ooo -bitcoinrpcpass=ooo -type=accept -fr=1KUu56RMsafvgwxkiGHmptrLLmBgQ16uQG -to=1MCHESTptvd2LnNp7wmr2sGTpRomteAkq8 -cu=MSC -am=1.234


sr. member
Activity: 266
Merit: 250
Great, thanks zathras! I'll start fiddling with masterchest-engine this evening.

One other thing: I looked through your library code and didn't see a way for it to handle simple sends (or DEx user-initiated operations).

In your wallet, I do see some of this functionality though: https://github.com/zathras-crypto/masterchest-wallet/blob/master/Form1.vb#L1142

Optimally, might it be possible for you to abstract this stuff (i.e. functions for simple send and similar DEx user-initiated functions) from the GUI logic and move it to your library or engine?

Then your wallet code could call this directly still. For masterdaemon, this functionality would just need to be wrapped up into a command-line utility, like grazcoin has for simple sends:

https://github.com/grazcoin/mastercoin-tools/blob/master/msc_send.py

If you abstract the functions over from your wallet to the library, I may be able to write this command line wrapper in .NET if you don't have the time (I'm rusty with it).

Let me know what you think about this.

Always happy to help Smiley

The functions you speak of (creating send & DEx transactions) are already in the library (encodetx, encodeselltx, encodeaccepttx etc).  They're just not used in the engine as its original design intent was just parsing transactions and processing them to arrive at a known state rather than creating new transactions (that's left up to my wallet).

It should be quite easy to spin up a CLI binary to expose these functions though.  I'll throw you something together (it'll be quick & dirty, my primary focus is fixing DEx state processing in the engine & wallet at the mo).

This may also help for some background here
sr. member
Activity: 462
Merit: 250
Great, thanks zathras! I'll start fiddling with masterchest-engine this evening.

One other thing: I looked through your library code and didn't see a way for it to handle simple sends (or DEx user-initiated operations).

In your wallet, I do see some of this functionality though: https://github.com/zathras-crypto/masterchest-wallet/blob/master/Form1.vb#L1142

Optimally, might it be possible for you to abstract this stuff (i.e. functions for simple send and similar DEx user-initiated functions) from the GUI logic and move it to your library or engine?

Then your wallet code could call this directly still. For masterdaemon, this functionality would just need to be wrapped up into a command-line utility, like grazcoin has for simple sends:

https://github.com/grazcoin/mastercoin-tools/blob/master/msc_send.py

If you abstract the functions over from your wallet to the library, I may be able to write this command line wrapper in .NET if you don't have the time (I'm rusty with it).

Let me know what you think about this.
sr. member
Activity: 266
Merit: 250
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs Smiley  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks Smiley

A sql schema dump would be great.... I just want to make sure I'm creating the schema to be like you are (a backup is fine too...just throw it up on dropbox or something and PM the link to me).

I'm testing on Ubuntu 13.10 x64, so Mono looks like 2.10, not 3.2.4:

"mono -V" output: "Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-5ubuntu2)"
"uname -a" output: "Linux d01b 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux"

Mono-wise, I've installed the "mono-runtime" package. If I need to install any additional packages (e.g. mono-complete, or other specific mono libs) just let me know.

I'm ready to test around sqlite as soon as you can add it, mono too. Smiley Thanks for your help on this!!
No probs Smiley Here you go:

Code:
USE [master]
GO
/****** Object:  Database [chestmaster]    Script Date: 11/26/2013 04:27:36 ******/
CREATE DATABASE [chestmaster] ON  PRIMARY
( NAME = N'chestmaster', FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\chestmaster.mdf' , SIZE = 5120KB , MAXSIZE = UNLIMITED, FILEGROWTH = 1024KB )
 LOG ON
( NAME = N'chestmaster_log', FILENAME = N'c:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\chestmaster.ldf' , SIZE = 10176KB , MAXSIZE = 2048GB , FILEGROWTH = 10%)
GO
ALTER DATABASE [chestmaster] SET COMPATIBILITY_LEVEL = 100
GO
IF (1 = FULLTEXTSERVICEPROPERTY('IsFullTextInstalled'))
begin
EXEC [chestmaster].[dbo].[sp_fulltext_database] @action = 'enable'
end
GO
ALTER DATABASE [chestmaster] SET ANSI_NULL_DEFAULT OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_NULLS OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_PADDING OFF
GO
ALTER DATABASE [chestmaster] SET ANSI_WARNINGS OFF
GO
ALTER DATABASE [chestmaster] SET ARITHABORT OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_CLOSE OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_CREATE_STATISTICS ON
GO
ALTER DATABASE [chestmaster] SET AUTO_SHRINK OFF
GO
ALTER DATABASE [chestmaster] SET AUTO_UPDATE_STATISTICS ON
GO
ALTER DATABASE [chestmaster] SET CURSOR_CLOSE_ON_COMMIT OFF
GO
ALTER DATABASE [chestmaster] SET CURSOR_DEFAULT  GLOBAL
GO
ALTER DATABASE [chestmaster] SET CONCAT_NULL_YIELDS_NULL OFF
GO
ALTER DATABASE [chestmaster] SET NUMERIC_ROUNDABORT OFF
GO
ALTER DATABASE [chestmaster] SET QUOTED_IDENTIFIER OFF
GO
ALTER DATABASE [chestmaster] SET RECURSIVE_TRIGGERS OFF
GO
ALTER DATABASE [chestmaster] SET  DISABLE_BROKER
GO
ALTER DATABASE [chestmaster] SET AUTO_UPDATE_STATISTICS_ASYNC OFF
GO
ALTER DATABASE [chestmaster] SET DATE_CORRELATION_OPTIMIZATION OFF
GO
ALTER DATABASE [chestmaster] SET TRUSTWORTHY OFF
GO
ALTER DATABASE [chestmaster] SET ALLOW_SNAPSHOT_ISOLATION OFF
GO
ALTER DATABASE [chestmaster] SET PARAMETERIZATION SIMPLE
GO
ALTER DATABASE [chestmaster] SET READ_COMMITTED_SNAPSHOT OFF
GO
ALTER DATABASE [chestmaster] SET HONOR_BROKER_PRIORITY OFF
GO
ALTER DATABASE [chestmaster] SET  READ_WRITE
GO
ALTER DATABASE [chestmaster] SET RECOVERY SIMPLE
GO
ALTER DATABASE [chestmaster] SET  MULTI_USER
GO
ALTER DATABASE [chestmaster] SET PAGE_VERIFY CHECKSUM
GO
ALTER DATABASE [chestmaster] SET DB_CHAINING OFF
GO
USE [chestmaster]
GO
/****** Object:  Table [dbo].[transactions_processed]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[transactions_processed](
[TXID] [nvarchar](100) NOT NULL,
[FROMADD] [nvarchar](100) NOT NULL,
[TOADD] [nvarchar](100) NULL,
[VALUE] [bigint] NULL,
[TYPE] [nvarchar](100) NOT NULL,
[BLOCKTIME] [nvarchar](100) NOT NULL,
[BLOCKNUM] [nvarchar](100) NOT NULL,
[VALID] [bit] NOT NULL,
[CURTYPE] [int] NOT NULL,
[ID] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL,
[matchingtx] [nvarchar](100) NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[transactions]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[transactions](
[TXID] [nvarchar](100) NOT NULL,
[FROMADD] [nvarchar](100) NOT NULL,
[TOADD] [nvarchar](100) NULL,
[VALUE] [bigint] NULL,
[TYPE] [nvarchar](100) NOT NULL,
[BLOCKTIME] [bigint] NOT NULL,
[BLOCKNUM] [int] NOT NULL,
[VALID] [bit] NOT NULL,
[CURTYPE] [int] NOT NULL,
[ID] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[processedblocks]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[processedblocks](
[BLOCKNUM] [int] NOT NULL,
[BLOCKTIME] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[exotransactions]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[exotransactions](
[txid] [nvarchar](100) NOT NULL,
[blocktime] [bigint] NOT NULL,
[blocknum] [int] NOT NULL,
[vouts] [nvarchar](2048) NOT NULL,
[id] [int] IDENTITY(1,1) NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[exchange]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[exchange](
[txid] [nvarchar](100) NOT NULL,
[fromadd] [nvarchar](100) NOT NULL,
[type] [nvarchar](100) NOT NULL,
[blocktime] [bigint] NOT NULL,
[blocknum] [int] NOT NULL,
[valid] [bit] NOT NULL,
[curtype] [int] NOT NULL,
[id] [int] IDENTITY(1,1) NOT NULL,
[saleamount] [bigint] NULL,
[offeramount] [bigint] NULL,
[minfee] [bigint] NULL,
[timelimit] [int] NULL,
[purchaseamount] [bigint] NULL,
[unitprice] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Table [dbo].[balances]    Script Date: 11/26/2013 04:27:40 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[balances](
[ADDRESS] [nvarchar](100) NOT NULL,
[CBALANCE] [bigint] NOT NULL,
[CBALANCET] [bigint] NOT NULL,
[ubalance] [bigint] NOT NULL,
[ubalancet] [bigint] NOT NULL
) ON [PRIMARY]
GO
/****** Object:  Default [DF__transacti__VALID__0519C6AF]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[transactions] ADD  DEFAULT ('0') FOR [VALID]
GO
/****** Object:  Default [DF__transacti__CURTY__060DEAE8]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[transactions] ADD  DEFAULT ('0') FOR [CURTYPE]
GO
/****** Object:  Default [DF__balances__CBALAN__08EA5793]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [CBALANCET]
GO
/****** Object:  Default [DF__balances__ubalan__09DE7BCC]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [ubalance]
GO
/****** Object:  Default [DF__balances__ubalan__0AD2A005]    Script Date: 11/26/2013 04:27:40 ******/
ALTER TABLE [dbo].[balances] ADD  DEFAULT ((0)) FOR [ubalancet]
GO
sr. member
Activity: 462
Merit: 250
Thanks JR. I'll rename the project to be "masterdaemon" at your advice.
sr. member
Activity: 462
Merit: 250
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs Smiley  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks Smiley

A sql schema dump would be great.... I just want to make sure I'm creating the schema to be like you are (a backup is fine too...just throw it up on dropbox or something and PM the link to me).

I'm testing on Ubuntu 13.10 x64, so Mono looks like 2.10, not 3.2.4:

"mono -V" output: "Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-5ubuntu2)"
"uname -a" output: "Linux d01b 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux"

Mono-wise, I've installed the "mono-runtime" package. If I need to install any additional packages (e.g. mono-complete, or other specific mono libs) just let me know.

I'm ready to test around sqlite as soon as you can add it, mono too. Smiley Thanks for your help on this!!
sr. member
Activity: 266
Merit: 250
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs Smiley  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?

It's on the to do list (auto creating tables) just hasn't been prioritized.

I'm just using express for masterchest.info - if you like I'm happy to shoot you a backup of its DB (the engine runs that site) which you can then just restore and test Mono off the bat without needing to manually do the tables?

From a cursory glance SQLite looks an easy enough addition.  Let me know how you get on with Mono (& what distro/kernel/mono versions you're testing with so I can duplicate).

Thanks Smiley
legendary
Activity: 1260
Merit: 1031
Rational Exuberance
I think it would be cool to have a mastercoin daemon (probably just name it "MasterDaemon" or something -mastercoind sounds too official).

We seem to be doing pretty well with no reference implementation so far. The various devs implement a piece of the spec, find differences and corner cases, discuss them, resolve them, and then repeat.  If I were to contribute code, it would probably just be pull requests against one or more of the existing code-bases.
sr. member
Activity: 462
Merit: 250
Zathras, thanks for your reply.

I gave some thought to things...to support both Windows and Linux (and even Mac OS X in the future) in as unified manner as possible, I'm thinking the best approach may be go to with your masterchest-engine/masterchest-library. I could use mono (under Linux and Mac OS X) to run it.

You mentioned testing under wine, but have you ever tested the engine and library under Mono? Beyond testing under mono (which I can do if you haven't), one enhancement I was wondering if you would be interested in making to masterchest-engine in order to make it possible for mastercoind (or whatever we end up calling it) to use your stuff cross-platform would be to allow it to write to a Sqllite database (instead of just SQLServer).

You could use System.Data.SQLite for this (http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki) ...probably just have a command line option to specify the engine to use (SQLServer or SQLite) and the actual SQL API commands wouldn't hopefully have to change much... you have very basic SQL statements and logic in there, thankfully.

My app could then periodically launch masterchest-engine.exe (either natively or via mono) to update the SQLite DB file, and my app would then just be reading the data from that DB as-needed. This would allow a single approach across all platforms, without requiring SQLServer on Windows...or any external database server. Sqlite supports a concurrent writer and reader no problem, with a bit of locking during the write itself.

This would make "mastercoind" be able to run well on Windows, Linux, and Mac OS X in the future....and I could come out with a build for Windows and Linux right off the bat. And as an advantage, folks on Windows that wanted to use Masterchest-engine directly could do so without having to install and run SqlServer, which is often overkill for smaller/test installations.

What do you think?

Firstly I'll just quickly say it doesn't have to be my library - you won't be stepping on any toes if you run with one of the other guys libs Smiley  With that said, if you think masterchest library & engine would be a good fit for you, have at it!

Just to clarify, I've only tested & fixed up the wallet for wine, not the engine.  Since they share a common codebase (and the same library) though I don't see too much of an issue.  I removed roadblock incompatibilities with the wallet & wine in about half a day so even if there are a few quirks with the engine, it's not masses of work to get it running.  Not tried Mono with either wallet or engine, but I'm happy to spin up a quick test if you like (will just connect to SQL over network for now).

I don't see any major headaches offering an SQLite option, I'd have to check it out but should be fine - let me come back to you on this.

FYI state processing for DEx is still very alpha in the engine & the matching isn't quite there yet.  State processing is fine for things like simple sends.



Yeah, I'm thinking your stuff is the best fit considering the multiplatform component. Tachikoma's stuff may work as a backup, but it looks like his whole data storage layer is inside the mastercoin-explorer project (as ruby on rails). Also, I did research calling over to a ruby library from python... didn't find anything conclusive with the short amount of time I spent on it.

masterchest-engine I should be able to us as-is (once it has the sqlite backend support and any necessary mono fixes), without having to touch/fork any of your code.

I'm installing SQL Server Express to be able to test masterchest-engine. The plan is to test it under windows first, then under mono (pushing to a SQL Server instance in Windows still). Once you're able to add the sqlite integration, I can test it all under mono in Linux.

Do you have any .sql files to create the tables and indexes for your schema or should I write my own?
sr. member
Activity: 462
Merit: 250
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.

What I was trying to say is that mastercoind should call the API of bitcoind to make the actual low-level bitcoin/mastercoin transactions, this way we reduce the security risk even more.
This is because we can assume that bitcoind is very secure.

Also, good to hear that security is one of your main concerns. Programmers often ignore security issues. In such application that deals with money security is a top priority concern.

Regarding your first point, I think that eventually, I'd like to see a fully integrated mastercoind. For now, however, I'm planning on probably going with Zathras masterchest-engine to reduce code duplication and greatly speed development time. My hope is that a "reference library" emerges from the Mastercoin community that is written in python (or has python bindings) and can be used by mastercoind directly. Even in that event, I'd still leave in the masterchest-engine backend, and allow switching between them for the reasons I've raised earlier.

Grabbing of masterchest-engine will be done directly from github in the automated setup script, to reduce the chance of error/security. It will also be possible for people to create their own mastercoind windows binaries, if they so desired, or didn't trust the pre-made binaries offered.
newbie
Activity: 12
Merit: 0
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.

What I was trying to say is that mastercoind should call the API of bitcoind to make the actual low-level bitcoin/mastercoin transactions, this way we reduce the security risk even more.
This is because we can assume that bitcoind is very secure.

Also, good to hear that security is one of your main concerns. Programmers often ignore security issues. In such application that deals with money security is a top priority concern.
sr. member
Activity: 462
Merit: 250
A few more things:

* This thing is going to be slick, I have plans for packaging the windows version as an exe and using a windows installer to install (using pyinstaller and NSIS or something similar), similar to bitcoind on windows.

* I'm open to naming this "masterdaemon" or something that's not "mastercoind" for the reasons zathras and Tachikoma brought up (mastercoind maybe being a "reserved" name). If JR/Ron would like this, just let me know and I'll change the name.

* Regarding the library, after spending an extensive amount of time looking and working with sx (and even improving its build process, see https://bitcointalksearch.org/topic/m.3700259), it's clear to me that Windows support will be a big effort...since it totally lacks support for Windows now, and its not only about compiling sx on Windows, but about getting sx, libbitcoin, obliesk, and leveldb working on windows, along with all other dependencies!

A big goal for mastercoind is to be as multi-platform as possible. If I had to do Linux only, I'd go with grazcoin's stuff (which is written to use sx/libbitcoin for all transaction processing). However, given our multiplatform goal with this, I'm thinking at this point, zathras library sounds like a more straightforward implementation to be the primary/initial library that mastercoind uses. The gotchas are getting it working under mono, and getting it working with sqlite, both of which I think can get done without too much hassle. I'll get into working with it some and have a better idea....(hey, at least we have choices here! Smiley)
Ola
sr. member
Activity: 311
Merit: 250
Forgive my ignorance but what does this differently compared to the current implementations? Are you not just creating Yet An Other Mastercoin implementation?

I'm actually not creating another Mastercoin implementation at all, as I'm utilizing grazcoin's mastercoin-tools package for the heavy-lifting (the integration will initially be a bit ghetto, but hopefully that will smooth out with time).

As I wrote in the initial post, the goal is to lower the knowledge requirements for mastercoin application development by making a program similar to bitcoind, where future mastercoin developers can just query using a simple API, instead of having to figure out blockchain parsing. This is much like how bitcoin developers today don't need to know about the guts of bitcoin's workings to create a simple bitcoin-related app, they just need to know how to interact with bitcoind's JSON RPC API.

Ok thanks for clarifying you are not actually creating a client but a RPC wrapper on top of a client.

I still think J.R. should be the one who writes the reference client since he can make decisions on how the spec should be interpreted. No offence intended, but who gave you the right to decide your implementation should be the reference one. It's not so much how the clients handles, you can strap an RPC client on top of any implementation, but the decisions you make in the underlying layers.

I think an RPC accessible client on top of any library is a excellent idea, I just think that the word reference implementation should be removed.  


I agree with that. My initial post was a bit confusing up top, sorry about that, but I wasn't thinking that I was making the "reference library" here.....only that mastercoin needed one.

In my mind, mastercoind is simply a gateway/frontend to whatever the reference library is. At some point, once/if it makes sense, I would like mastercoind to possibly become the "reference client"/"reference front-end" that uses a separate backend reference library, but I have no say in that decision itself. As I said, it can be done if it makes sense, and people start building apps around this thing. Plus, I have no desire or intention of writing any mastercoin parsing library, or having to deal with the low-level details of one (this was the purpose I wanted a mastercoind so badly :p).

As I said earlier, right now I'm using grazcoin's MSC library, which is more utilities for his mastercoin site than a formal library ready for 3rd party use. This requires me to hack it a bit to get it to "fit in" to mastercoind. However, this is 110% better than me writing something from scratch. I see the backend library as taking care of those difficult details, and mastercoind wraps it up real nice and makes it very easy to use (as well as allowing it to be used by a client written in any language, or on a different server, etc ... list the list of advantages I have in my 2nd or 3rd post in this thread).

Maybe a future goal for mastercoind is to allow it to plug into 2 or more backend libraries...i.e. the "reference" one (whatever that is) and the main alternative. That way, you can get the similar front-end API functionality, but swap out the backend library if you need/want (e.g. for easy compatibility testing, or maybe one doesn't compile under windows and another one does, etc). The swap should be as easy as modifying a config file in the program, and making sure the library is installed on the system.

The end goal with this thing is to have something that's as easy to use, or easier, than bitcoind is today.

+1000 If you could design the backend to plug in into more than one of the main libraries that will be super awesome...That way all code from the main and new mastercoin devs will keep being used. A modifiable config file is also definitely the way to go in determining what library to choose...

I urge you to keep on the development path, there is noting stopping you from dong this. More-so it is all open source and WILL DEFINITELY open mastercoin up to main stream developers.  I will be available to aid in testing out the features if you need too because I am seriously waiting on this, I am sure a lot of regular web devs are waiting on this also. I thought this feature would be a year or more out, but you changed all that by taking on this initiative and this could be a reality within a couple of months.


I hope JR/Foundation blesses this initiative so that it is looked upon as a credible point to get involved with mastercoin apps development. There  should be nothing political about this..If the name "Mastercoind" is a problem, which is simple a convention, maybe that can be fixed later by JR or any of the executives of the foundation..Please push on rbdrbd

sr. member
Activity: 462
Merit: 250
Hi,
I think that writing mastercoind  is an excellent idea, and I have an idea for improvement:

I think that the best way to implement this is by making mastercoind use bitcoind as a backend.
Just like HTTP implementations are using Berkeley sockets and not rewriting them.

The benefits we get by choosing this architecture are simplicity and more important security.
We can now assume that bitcoind is a very secured piece of software.

Few days ago I saw a lecture by Gavin Anderssen where he said that the bitcoin-qt had a lot of security holes in it's early days.

We can't afford security holes in mastercoind, it will not be under the radar like bitcoin-qt in it's early days.





Good point, I'm aiming for the level of security to at least match bitcoind. From a remote exploit prevention, mastercoind is written in python using cherrypy. This fact makes certain types of exploits less likely (e.g. buffer overflows, and other things that could plague writers in lower level languages like C/C++ if they aren't careful with their string length checking). Other vulnerabilities, such as SQL injection, are still possible, and from there it's good coding practices, and doing extensive cleanup/sanity checking on user-provided input (the lack of which is always what leads to these kinds of attacks, essentially).

Then you have things like stored data security. For now at least, RPC username and password are stored in a plaintext settings file, similar to bitcoind. We can look at changing this in the future, however. If something very sensitive is required (such as private wallet key for signing requests), then I plan to collect that as part of the request and not store it, if at all possible.
Pages:
Jump to: