Topic: 300 BTC Coding Contest: Distributed Exchange (MasterCoin Developer Thread) - page 2. (Read 129207 times)

I propose that if the specification is accurate then only transactions which are not invalidated (or otherwise specifically validated) by the specification are valid.

I'm going to use the same example as my previous post to demonstrate the effect of this. The specification states, "A single transaction must be constructed satisfying the following requirements:"


Excuse me for quoting myself, but this is what follows:


If the specification is accurate it also follows that, because there is no implementation that recognizes Class B transactions with P2SH outputs to uncompressed public keys as invalid, no implementation can be trusted to display the correct mastercoin balance of an address.

Further, a significant sum of bitcoin has been unknowingly traded with parties which may not even know how much mastercoin they have. Given the possibility that this party has a mastercoin balance to cover the trade, there's no guarantee that it was actually sent to the buyer.

It's really tragic for mastercoin if the spec is accurate, because the loss of bitcoin in invalid mastercoin transactions cannot be recovered, and every single wallet/implementation is useless as a method of sending, receiving, trading, or otherwise interacting with the mastercoin network.

I would suggest against maintaining that the specification is accurate.

EDIT: Concerning the invitation to submit a pull request, it's not a matter of the spec simply being outdated. If it's reasonable to consider that the specification can be out of date, the public release of an updated specification can change the validity of mastercoin transactions confirmed in the blockchain prior to the time at which the revision is published. Bitcoin sets a very high standard on the ability to verify the validity of a transaction, and there's no reason that any entity should be able to retroactively change mastercoin balances.

I recognize the invitation as a suggestion to get involved and address items I may have issues with in a structured and accepted manner, however this isn't about redefining a valid Class B transaction. It's about the ability to know an addresses mastercoin balance, to trust that an addresses mastercoin are secure, to feel safe moving or trading mastercoin, and finally to minimize the impact of data storage on the blockchain.

But for the sake of argument, if I did submit a pull request and it was merged, this would just exacerbate the issue I have with my ability to trust the specification as a document that defines mastercoin, and my ability to trust mastercoin as valuable. If the specification does not or cannot define mastercoin (and consequently the security of mastercoin that one owns), I believe it follows that mastercoin is not valuable.

EDIT 2: I expanded my first edit, and clarified a poorly worded sentence.

Easy.

First, define the purpose of the mastercoin specification. Revisions as necessary.

Second, add a section to the specification that details a system of revision control, and how a specification revision affects past and future mastercoin transactions. I'd start with the following bulletpoints:

A specification revision...

• Does not change the validity of any mastercoin transaction prior to the public publication.
• Will not impose changes that cannot or do not take place immediately upon public publication. e.g. define Class B transactions as only valid if OP_RETURN is used for data storage, because no implementation is prepared to make this transition.
• Is considered to define mastercoin, the mastercoin protocol (etc.) until the point in time at which another valid revision is publicly published.
• Is defined as valid when cryptographically signed by the entity that is granted the right to revise the specification.
• Is defined as publicly published when the following has taken place: 1.) The revision and signature is announced and made publicly available, and 2.) The hash of the revision is stored in the blockchain and is signed by the same entity that signed the revision.

And thus established is a method by which an individual can independently prove the validity of a mastercoin transaction at any point in time. The only tricky revision is the first one included in the blockchain, which must establish which transactions up to that point are valid. I say, publicly publish the revision, and then take a "snapshot" of the balances of all mastercoin address greater than 0 as of the block height at which the revision hash transaction was first confirmed. Take a hash of that dataset, store the hash in the blockchain, and define that as proof of a transactions validity up to the point at which the first revision was confirmed.

I'll let one of the devs weigh in on the class B stuff. We do plan to have better options for non-windows users soon. Omniwallet should be a good solution, and for those who want to run a full node, we want to have a full desktop client too (possibly a port of Masterchest using mono). We hope to greatly improve usability on all wallets which we sponsor.

I think the spec is accurate right now, but if you've found something that is out of date, we welcome pull requests

Thanks for weighing in.

Say the protocol would adapt a more strict output policy (e.g. only a valid pub key of the sender + data-packages are allowed), how would you deal with historical transactions that may be affected by such a change?

Wow.  That was a mouthful.  You probably have a fair point on some of that stuff.  For sure the PR is lacking.  No offense to Dom who works hard.  But I think the priority has been to work diligently on the technical aspects.  Once those are working, a good PR scheme will introduce them properly to the community. 

On the other hand, it seems like a knowledgeable longtime mastercoiner like you has loads to contribute.  I sure hope you find your deal with the team and are welcomed into the effort to perfect that which will soon become a very powerful Mastercoin protocol. 

JR.

My Mastercoin Address:
1KSim7sKQ9bjsMd2RyUbe9HxU73yX6CwuJ

will drop an email too.

regards
Colbert Lau

I started a discussion with dexX7 over in the mastercoin announcement thread. He invited me over here due to the technical nature.

I've seen a bit of mastercoin history - I participated in the fundraiser, I attempted a few decentralized trades back in mid November here on this forum, and I was one of the first to complete a trade on the 15th. I consider myself pretty gung ho about mastercoin, however a few items have left me a bit disenfranchised recently. So, if you'll excuse some potentially blunt sentences I'd like to open up discussion on some issues that I feel fairly strongly about.

I'm going to preface this with a quick distinction. As demonstrated below, a private key produces two distinct key pairs. Both key pairs have the same private key, however as they are implemented in a wallet, the private key WIF of key pair #1 cannot spend outputs sent to address #2, and the compressed private key WIF of key pair #2 cannot spend outputs to address #1.


Keep that in mind while I continue.

• Currently different mastercoin implementations generate Class B transactions differently. One includes the sender's uncompressed public key in the P2SH, while another includes the sender's compressed public key in the P2SH.
• I acknowledge that the outputs produced by both implementations are spendable, however I pose that it's not currently reasonable to expect a user to spend them.
• There is not a mastercoin implementation that will always generate a P2SH output that can be spent by key pair associated with the sending address.
• There is a likelihood that the P2SH output will not be able to be spent by a key pair in the sending addresses wallet.
• P2SH outputs from Class B transactions are not recognized, and cannot be spent by any bitcoin or mastercoin wallet currently available(please correct if wrong)
• If mastercoin clients send P2SH outputs to both compressed and uncompressed public keys originating from a single private key, the implication is that mastercoin recognizes a "unit of identity" as being defined by a private key, and thus two key pairs. This isn't compatible with any other client that I'm aware of, can be a privacy leak, and security risk if one of the WIF private keys is compromised. Further, I'd be inclined to think that mastercoin implementations would always need to display and identify both key pairs associated with a private key.
• Class B transactions technically do not create unspendable outputs, however it's a cop out to say that Class B transactions address the concern of blockchain bloat due data storage. If outputs of a transaction are not recognized, cannot be spent, or are otherwise not used in transactions when reasonable, they will not be spent and are subject to loss due to perceived lack of value or understanding.

What I'm trying to point out is that the absence of standardization between mastercoin implementations creates a weird scenario in which it's significantly harder to spend P2SH outputs than it has to be. If a wallet could recognize and spend those P2SH outputs, another weird situation occurs in which the P2SH outputs cannot all be spent without the wallet generating the other key pair associated with a private key. IMO, that's funky and there's no graceful or secure way to handle it.

The next thing I want to bring up is the mastercoin specification. It's not my intent to be a stickler, however I believe it's critical to meticulously maintain the specification, because without the specification there is no mastercoin and there is no value. Yes, the disappearance of the mastercoin specification is an extreme case. So let me provide a lesser example. The mastercoin specification states that a Class B transaction must satisfy the following:


Like I said previously, I've been enthusiastic about mastercoin. However, recently it feels esoteric, loose, and out of touch. I recognize that decentralized trading is groundbreaking, and I recognize that trades are being made.

But I do not think that it is by any means reasonable to think that a client is close to a "high bar for usability" if it doesn't recognize the outputs of a transaction it created, signed and broadcast, cannot spend unspent outputs which it is (cryptographically) capable, and creates multisig outputs that cannot be spent by a key pair in the wallet. Further, there are only native Windows clients, and other OSes have to use a web client and manually handle WIF private keys.

Also, the depth of knowledge required to spend P2SH outputs as they are currently generated is unreasonable to expect in a user. Like I said previously,


If a mastercoin client can't accommodate and expect this perceived lack of value and understanding in a user, it is irresponsible to taut the client as one that has a "high bar for usability."

Mastercoin,

• uses irresponsible implementations of Class B transactions.
• creates outputs that aren't spendable by any wallet.
• greatly privileges Windows users.
• and doesn't have a specification that describes current implementations.

I feel strongly enough about these items that I'd rather see some sort of resolution than sleep like a normal person. I don't feel like it's my place to offer unsolicited solutions to the problems I'm posing, however would be happy to discuss my ideas on the subjects.

And if consensus is that the items I bring up are not problems, take this as feedback from a "long time mastercoin user" that the project feels out of touch, lacks on the PR front, and has direction at the expense of overall quality.

Guys,

Here are the addresses I plan to use for February, based on what I have from each of you:

Who	BTC Address	MSC Address
Faiz	1JJ6e9iWjGgEnCQRivKvdfNjRTgYVaKtWy	1MMF42zk8f4TqoV1QFeia18bb14rBHXzAU
Colbert Lau	1MnPcdUFDYnTz3jz9n3erfWbd7JZfPunxL	??
Grimentz	128MGmQKtKUPf66w2w9eaGc7YBqLrcnLx5	1rM1oMEFMfhPBo4kRaf2CMXf9XQCHyYWi
Adam Chamely	18hFjy7CWvX7FgZEXEW3bgz35KpYAzFR1c	18hFjy7CWvX7FgZEXEW3bgz35KpYAzFR1c
Bitoy	14cn7BiyzUeMcddC3RUFk77Z2k8NcP66yf	14cn7BiyzUeMcddC3RUFk77Z2k8NcP66yf
jeroenn13	15EMewc1nH8Cqzsx8hrFNNdbQBjVTEWahp	15EMewc1nH8Cqzsx8hrFNNdbQBjVTEWahp
Marvshneider	1ACsaXeqiPwZVzEuWXa2vwe9etxaet6QWg	1PKKRfHgvfZi2ToZDKkKrCUFpvuVKHz9WV
Ryan Keenan	1PyQbHr3ywRdbkcq3sZKpkkJNBJUXbGXJH	13NzZ7nwzEzLfuCEtfC1iQv5mJzs4PsFoW
grazcoin	1CZ5WRYj48vWV7cYRRdheyhiodMVP3EMi3	1CZ5WRYj48vWV7cYRRdheyhiodMVP3EMi3
Tesca	16BurP5f7n5RqjWGQ44Lj5NX6GzxaoAaqU	16BurP5f7n5RqjWGQ44Lj5NX6GzxaoAaqU
Craig	17xr7sbehYY4YSZX9yuJe6gK9rrdRrZx26	??
MasterXchange	1zobh1SwsVdk3uMjEHeGV86hjNZeJ5CwC	1zobh1SwsVdk3uMjEHeGV86hjNZeJ5CwC
Curtis	19TwBziBeCb4VnrZgJasJukwRKNSVvDp7Y	13pm7cmA5vVpKkDLJCvqh26kcp6V6PJ1Aq
dexX	11NiSNoGYomEA3ZHEN4R9ne4kwDjECsAy	11NiSNoGYomEA3ZHEN4R9ne4kwDjECsAy
Marin(Bebopzzz)	15AnM1gg5HAAnRPPBUXFX1N6mbTt4cLyZJ	1GmsjoERUACUwAsjKSLhSekH8EeXLRWKku
Shannon Code / Genecyber	1N6vMzy46RNGny7u5BYMFGpPin7uW3cTKK	1DMH2jezmNha2ShBhq3ogsXGviPBRppmwX
jakecnn	1HXsrKJHbJCdWThFQL14tQ2ThuUhKUzmsf	1HXsrKJHbJCdWThFQL14tQ2ThuUhKUzmsf
Patrick	1EdtK7EkErgeU17qRuifxASA2yMoxibxYv	1AQBeWsRfbHst7NojYdt42MZThcR4jy8fk
Zathras	1DrYXscsbMxhQMvvCmK7aVppbJGhr3RRSN	1Px5ruEuWdtvkG3AJ1sJ8Q7ckV8B1f3us5

Please check your address to make sure it is correct. Speak now, or forever hold your peace!!

Yup. February numbers are nearly ready. They are my top priority once I catch up on urgent emails.

edit: I think we are close on usability. If we spend the next couple weeks fixing things that make the various wallets hard to use and trade with, I don't see any barrier to finishing the payout at the end of March.

J.R can comment his perspective on this. Here is mine:

We have a bit of a mixup here. On the one hand, I do believe that the "High bar for usability" goal has not been met, and doubt it will be met this month.
On the other hand, we moved to paying out monthly bounties in February, and .

I hope it is clear that the DEx bounties paid in Feb, and the March monthly bounty, count towards the 300 BTC bounty (out of which 150 BTC has been previously paid).
So after March I don't think that a lot of money will remain in the original 300 BTC money pool (J.R needs to reply with some numbers).
I expect the bulk of the money remaining from the DEx to be divided in March's monthly payout.

Perhaps if there is only a tiny amount left after that, we will decide to increase March's payout and pay out the full remainder of the 300 BTC bounty this month, in order to avoid the accounting/judging headache. We will wait for J.R's number regarding February's payout (due any day now), and see what makes sense.

As far as I see it - on 2014-03-25 04:54:22 GMT it will be exactly 10 days and the DEx prize can be distributed. Right?
It is still not fully clear how the Feb bounty and the DEx bounty are coordinated.
A short explanation would be great.

Well the thing is bitcoin uses heuristics to intelligently select coins to use intentionally to minimize the transaction weight on the network etc.  Masterchest does none of this and just follows the rules whether that results in an optimized transaction or not.  

Like a car from 50 years ago, it still gets from A to B but it could be more efficient.

I think the best way to come at it is to just go straight for proper adaptive transaction crafting and make Masterchest smart enough to handle some context - eg a couple examples from one of my git comments:

Can I craft this transaction in a way that leaves unspent inputs available for further Master Protocol transactions from this address same block?
Can I craft this transaction in a way that minimizes the fees payable by the user?
Can I craft this transaction in a way that consolidates some of the small reference outputs (.00006 etc)?

And so on...  After that we move from MP to Bitcoin considerations:

Can I craft this transaction in a way that minimizes transaction size on the network?

Etc Etc

Appreciate the offer but I think I'll be OK - after looking at this a bit more I actually don't think it's that cumbersome to look at increasing fee after initial transaction creation & signing - even if the fee does need to be increased (and the total input doesn't cover it), that's at worst two extra >dust inputs which would be about ~300 bytes so not enough to trigger another fee increase - that nullifies my concerns that we'd get into a loop adding inputs to cover fees which grows the tx to require more inputs thus more fees, which grows the tx and rinse repeat.  Worst case is we have to go back and change things once and resign, which isn't so bad.

Thanks
Zathras

I think there are two ways:

You can calculate the transaction size by taking each subitem into consideration (e.g. public key, signature, outputs etc.) and determine the fee directly. All those numbers should be fixed, but I think this is probably a pain. Or you create the transaction, sign it, check the length and adjust the fee, if necessary, and sign again.

I could create a lookup-table for all included parts, if that would help? E.g. one uncompressed public key = 65 byte etc.

It's an interesting topic - receive lots of MP messages and you'll have lots of small ~0.00006 inputs - redeemimg them at the same time means a bigger transaction, thus requires more fees.

I actually think it's mostly about us devs being smart & creative about the way we craft our transactions to make sure we're spending these 0.00006 inputs where possible together with inputs from other transactions but at the same time not throwing around silly transactions with 50 tiny inputs.

That's actually a really interesting problem - especially in relation to these fractional transactions.  We build the transaction before we can see its total size, but by then we've already selected the inputs.  As we're talking such tiny amounts (often inputs of 0.00006) increasing the fee by 0.0001 likely means we have to add another/more inputs to cover the increased fee, but that has now changed the size of our transaction again so we may need to yet again go back and add more inputs & more fees and so on.  Fun game

In all seriousness though I do agree with DexX - changes to payments make me nervous but it should be safe.  I've pushed up a shortcut fix which grows the fee during transaction build using simple (very conservative 200 bytes per input) guestimation based on total number of inputs at that point in the transaction build.  Should take care of any potential risk for now until I can be a bit more classy with input selection


Note this only applies to DEx payment transactions, all other transaction types Masterchest selects a single input only to cover fees.

Thanks
Zathras

+1

and thanks Zathras for the explanation

what should be the process for this?  someone going to file an issue? i guess technically its not a master protocol thing, but it would need to be at least discussed in a special considerations area of the spec?

This is the most important part related to this transaction. Over 1 KB size = 0.0002 fee required to be treated as standard transaction. If this was issued via one of the wallets, the wallet creator needs to adjust to take transaction size into consideration asap. Delayed transactions due to insufficient fees can be a game breaker, if MSC are bought via DEX and the time window closes before the transaction confirms.

This is not so much a problem of the Master Protocol, but more one of how bitcoin works.  If I may take a moment to explain

Miners 'prioritize' transactions.  This means when your transaction is relayed to a given mining client it adds the transaction to its memory pool and assigns a priority and fee.  If the priority/fee is high enough, hopefully the transaction will be included in the next block.

Now - the interesting bit is in how this transaction priority is calculated by bitcoin core, as follows:

priority = sum(input_value_in_base_units * input_age)/size_in_bytes

Thus we can see the smaller and younger the inputs, the lower the transaction priority becomes.  

If we specifically relate this to your transaction, we can see that the inputs were all very small.  Additionally because there are a number of them, the size is actually over 1KB.  Thus we have a small input value and a (relatively) large byte size for the transaction.  If you pop those numbers into the formula above you'll see this provides for a very low priority transaction.

The good news (and this is bitcoin-wide) is that in laymans terms - the more you send, the higher your transaction priority.  Thus this may be an issue for low value fractional transactions and I'll do some further investigation and discuss with the team - but essentially as the value of the DEx purchase increases, the risk of the issue you describe occurring decreases.

I have also been doing some thinking on input selection, but that would apply only to the Masterchest implementation when I get there.  Currently I select smallest input for risk minimization (avoid losing a big input if something goes wrong creating the transaction).  On the todo list is adaptive transaction encoding in the library, so rather than 'dumb' input selection we look at input values, ages and eventual transaction size to determine priority before it's broadcast & if this would result in a LP transaction that is likely to take a while to get mined, re-encode it if possible for a higher priority, else send it with a higher fee.  That's a complex undertaking though so will take a little time (we always have to be careful when coding input selection and amounts as that's where the biggest potential to lose bitcoins sits IMO).

Thanks!
Zathras

EDIT: Also realized I wasn't explicit in stating it's of course also feasible to simply discourage (or disallow) fractional purchases of for example 0.000X BTC after a specific block height in the protocol itself too - the Master Protocol DEx has never been intended for high frequency or micro trading (and these micro trades cost more in fees than the trade itself).

The payment in question is https://blockchain.info/tx/b9151c3371d46853e59ee6c9124c2a534f41363b29f372d914a19a83a5917665
and according to blockchain.info a tx-fee was paid.

the tx had no mining fee right?