Topic: 300 BTC Coding Contest: Distributed Exchange (MasterCoin Developer Thread) - page 18. (Read 129207 times)

Testing work:

Installed Bitoy's wallet on 2 systems, but couldn't get those to work. Currently communicating with Bitoy to find and resolve the issues.

Extensive testing of DEx and giving feedback over the last few days using grazcoin's HybridWebWallet and testing plan.
Addresses used:
1G3P5bws8wRVrVfKWxv8F85pRjs9qXyyA6
14Q2NNiF5YzDZCjo7vrBuHdbZcmHRJyW3j
1NVNkPqSLo4N2seaZCzAkVFPjcGLPJSZAZ

Wow, so many terrific updates, this really shines a light to the wonderful work you guys are doing.
Thank you thank you thank you!


I summarized a brief TL;DR of the current status on the wiki.
Feel free to add more stuff/edit away any mistakes.

I am the developer of Masterchains's web site. In the previous round, I received the payment together with Grazcoin. In the current round, Grazcoin and I feel that I have made significant contributions that may be relevant to other related implementations testing and projects. These contributions include the integration of BitcoinJS and Javascript implementation of DEx transactions signing and sending. Grazcoin and I really like the visual language I developed to visualize the different types and status of the transactions (please find the link to a detailed documentation of this visual language below.) This helps greatly especially when it comes to testing DEx on masterchain.info. Although Grazcoin is the one that is responsible for most of the communications, please consider processing our payments separately.

Detailed list of achievements:

Development:

* Design icons for all the transaction types and backgrounds to indicate each transaction DEx's status.
Document describing all possible transactions / status combinations:
https://masterchain.info/downloads/masterchain_icons_list.pdf

Implemented the icon / background scheme with transparent icons and CSS classes and integrated in masterchain.info site.

* Implemented HTML form + signing (in javascript) + sending of wallet skeleton required for DEx testing.

* Implemented an offline version of the signing form in javascript (in final stages of testing, will go live tomorrow).

* Responded to users and testers feedback (on mailing list and the forum) and improved usability of the transactions sending form.

* Implemented HTML client side wallet for masterchain.info: https://masterchain.info/wallet.html (users add addresses to wallet from address pages.)

* Many small html5 features and improvements to masterchain.info

Testing:
 
* I tested the status of individual transaction types on masterchain.info using the icons mentioned above and hover HTML message that I also implemented.
 
* Test signing of all possible DEx transactions within masterchain.info.
 
* Suggest structured testing format doc based on the transaction type / status icons and backgrounds I designed:
https://masterchain.info/downloads/DEx_test_plan.pdf
 
* Followed the testing document to test transactions on masterchain.info

* Testing (+ fixing) of masterchain.info DEx forms on >20 different combinations of platforms, browsers and screen resolutions including 3 tablets (iOS, Android and windows 8.1) and 2 smart phones (iOS and Android.)

just an FYI for folks, most of you are already on the dev@ email list but for people just getting onboard:

Master Protocol Dev Email List ([email protected]) and Google Group https://groups.google.com/a/mastercoin.org/forum/#!forum/dev

Update of work progress of Grazcoin since last contest.
The contribution that Grimentz has done is covered on:
https://bitcointalksearch.org/topic/m.4664542

Code:
178 out of 224 commits of masterchain.info (pushed in "real time") belong to the this 300 BTC contest.
https://github.com/grazcoin/mastercoin-tools/commits/master
These include cleanups, infrastructure upgrade for transaction validation (e.g. https://github.com/grazcoin/mastercoin-tools/commit/ca669d885ffbe22daedce234961c72c9ef0d81d6 ), the whole DEx parsing as well as encoding, and DEx wallet skeleton.
Some code contributions needed for masterchain are on sx project repository:
https://github.com/spesmilo/sx/commits?author=grazcoin

Major part of the implementation is the set of publicly available web apps (POST/json):
1. validating address and checking if pubkey is on the blockchain (/wallet/validateaddr/)
2. generating a bitcoin/mastercoin/bitcoin payment raw send transaction to be signed using the hybrid web wallet (/wallet/send/)
3. generating a raw sell offer transaction to be signed by the hybrid web wallet (/wallet/sell/)
4. generating a raw accept offer transaction to be signed by the hybrid web wallet (/wallet/accept/)

Time:
I have changed to 100% time on the project after the first week of Jan 2014. I am working only for bounties.

DEx Development:
Tutorial showing DEx in action can download at:
https://masterchain.info/downloads/masterchain_tutorial.pdf
e.g. payment stage for accept offer:


e.g. using the hybrid wallet for paying the accept offer:


Collected the major contributions from the forum:

Parse buy/sell offers on matercoin-tools
https://bitcointalksearch.org/topic/m.3483170
https://bitcointalksearch.org/topic/m.3487202

Discuss purchase offer of more than available amount.
https://bitcointalksearch.org/topic/m.3507904

Discuss Giving priority to purchase offer over selling offer in the same block
https://bitcointalksearch.org/topic/m.3551859

Compressed/uncompressed pubkey in multisig (partly relater to sell/accept offers):
https://bitcointalksearch.org/topic/m.3407464
https://bitcointalksearch.org/topic/m.3408040
https://bitcointalksearch.org/topic/m.3420952

Discuss future compatibility of mastercoin to other currencies (other outputs)
https://bitcointalksearch.org/topic/m.3433105


Discussion and commenting on pull requests:

Added reserve clause to Selling Order description
https://bitcointalksearch.org/topic/m.3553595
https://github.com/mastercoin-MSC/spec/pull/1#issuecomment-28246689
https://github.com/mastercoin-MSC/spec/pull/1#issuecomment-28302612

Updated verification API for DEx (comments)
https://github.com/mastercoin-MSC/spec/pull/35#commitcomment-5050239

Suggesting "Remove the ability to change a sell order without canceling it first"
https://github.com/mastercoin-MSC/spec/pull/39
https://github.com/mastercoin-MSC/spec/pull/40#issuecomment-32792661

Integrate new icons for DEx:
https://bitcointalksearch.org/topic/m.3579227

Discuss respecting parsing of BIP11->BIP11
https://bitcointalksearch.org/topic/m.3636439

Keep DEx rules simple discussion
https://bitcointalksearch.org/topic/m.3636439

Differences between bitcoin payment and a send/sell/accept
https://bitcointalksearch.org/topic/m.3742159

Announce mastercoin_verify API compatibility
https://bitcointalksearch.org/topic/m.3742428

Discuss consensus conflicts:
https://bitcointalksearch.org/topic/m.3976570
https://bitcointalksearch.org/topic/m.3976769
https://bitcointalksearch.org/topic/m.3990170
https://bitcointalksearch.org/topic/m.4106402
https://bitcointalksearch.org/topic/m.4119458

Discuss multiple change on tx:
https://bitcointalksearch.org/topic/m.3983975

Different values of outputs in class A:
https://bitcointalksearch.org/topic/m.3990014

Consensus is nearing - close gaps:
https://bitcointalksearch.org/topic/m.4449664

suggest standard 31556926 seconds per year
https://bitcointalksearch.org/topic/m.4450947

Describe rounding error discussed on hackathon
https://bitcointalksearch.org/topic/m.4472218

Announce working Hybrid wallet + DEx update:
https://bitcointalksearch.org/topic/m.4542980

Discuss race condition with cancel sell offer
https://bitcointalksearch.org/topic/m.4578796


DEx Testing
A parallel masterchain.info setup for testing was installed on https://dev.masterchain.info (it is not always in sync with the net).
It is used to reproduce reported bugs, test new features and verify bugfixes.

Skeleton wallet was developed to ease testing for casual testers (it will develop to a full project during Feb).

I was searching actively for parsing problems on other implementations:

Start a thread for comparing implementations:
https://bitcointalksearch.org/topic/m.3742428
https://bitcointalksearch.org/topic/m.3882308

Live consensus checker between explorer and masterchain
https://masterchain.info/general/difference.json

Negative balance on explorer
https://bitcointalksearch.org/topic/m.4542980

zero inputs error
https://bitcointalksearch.org/topic/m.4595679

Announce test plan docs:
https://bitcointalksearch.org/topic/m.4608073
https://masterchain.info/downloads/DEx_test_plan.pdf


Addresses without MSC/TMSC issue:
https://bitcointalksearch.org/topic/m.4614733
https://bitcointalksearch.org/topic/m.4614767


DEx Testing Support

It is important to be responsive on testers inputs:

UI issues on masterchain - .003 vs. 0.003
https://bitcointalksearch.org/topic/m.4571532

Refresh required (after server side update)
https://bitcointalksearch.org/topic/m.4617418

Debug missing tx:
https://bitcointalksearch.org/topic/m.4626273
https://bitcointalksearch.org/topic/m.4646002

comment on found issues:
https://bitcointalksearch.org/topic/m.4654669


Looking forward to achieving consensus also on the DEx

Edit: never mind

Keep up the excellent work Devs!

Summary of Work

MyMastercoins website
http://www.mymastercoins.com/Default.aspx
Source Code
https://github.com/Bitoy88/MyMastercoinsWebsite/

Create multisig parsing of Simple Send and Dex transactions from blockexplorer.com and blockchain.org.
(Simple send, sell offer, purchase offer, payment)

Test MSC DEx order book
http://www.mymastercoins.com/Orders.aspx?CurrencyID=2

Charts for Simple Send, test msc
http://www.mymastercoins.com/About.aspx

Transaction module.
http://www.mymastercoins.com/Transactions.aspx

MyMastercoins thin client wallet MMW (windows based)
http://www.mymastercoins.com/MyMSCWallet.aspx

Send MSC (simple send)
Send TMSC (simple send)
Sell TMSC Distributed Exchange
Buy TMSC Distributed Exchange

Gets data from mymastercoins.com SQL server
Used Zathras library to send multisig transaction

Help synchronize MSC with tachikoma,grazcoin, and Zathras.  Synchoronization of TestMsc ongoing.

Sent test transactions using the MMW wallet from address
1BKpa19m5Xy9SvSzC5djPWtCfbuynSDwmb
1QFWodNASZv8KRHnhypKaMys5CGRg7GrFQ

Hi ATriz,

You have 107.12380845   tmsc.  You offered to sell 1.  Your available balance is 106.12380845 tmsc.  

Hi all,

Update as requested by JR on my progress to date.  I have some code cleanup to do and then I'll get these new versions pushed up to git.  FYI following discussions with Ron, these new versions will form the first main branch and I'll be moving development to an experimental branch with increased sync frequency to increase visibility.

Development

• Masterchest Wallet
  My wallet software is coming along nicely, this now has full DEx support.  Still some work needed, the UI needs a good cleanup and there are still a lot of bugs on my outstanding list.  Lots of user-orientated stuff (for example bitcoin config autodetection).  Solid progress though, pretty use-able - but as I say still a lot of bugs and I need to do more testing on my validation for the DEx payment on buys.  I don't want to put out unfinished stuff so it'll be a little longer for release.
  
  I'd at least like to communicate my progress though, so despite best efforts to stop me by this recording software - I've put up a demo video.  You can catch it at the following link:
  https://masterchest.info/walletdemo
  
  
  
  
• Masterchest Library
  The biggest change in v2 of the Masterchest library is of course DEx support.  All DEx messages are supported, and the library handles both encoding and decoding functions.  Also some nice performance gains achieved through tuning (recursive inputs etc) which speeds up blockchain scanning in the respective softwares that use it.  Lots and lots of bugfixes and new Class A parsing using the simplified model.
  
  
• Masterchest Engine
  v2 of the Engine has also been rewritten to support the new DEx messages and incorporate DEx state processing.   Also runs the consensus system - v2 includes test MSC.  DEx state (as with addresses and sends etc) is maintained in easily accessible tables to facilitate ease of use for web devs who may not be coders (as long as someone knows how to run a select * from exchange where type='pendingoffer' and valid=true for example) can build a web service on top of the engine.  For the results of the engine please see the masterchest.info website - the engine is what populates the databases for it.
  
  
  
• Masterchest.info
  v2 of the website brought with it - you guessed it - DEx support (a theme is appearing ) and a significant redesign.  The original v1 site was going to be a web wallet before I thoroughly thought through the security implications and decided security & limited time were not two concepts that mixed well!  As such all the text related to web wallet functionality is gone, as are the unnecessary two-step processes for looking up transactions/addresses etc.  Charting is real-time & once there are more trades taking place as we ramp this up I'll open up the view options other than 60-day.
  https://masterchest.info
  
  
  
  
• Masterchest CLI
  Masterchest CLI is your bread & butter 'does what it says on the tin' utility.  It simply wraps the Masterchest library in a CLI wrapper so you can call the functions from the command line.  Originally built for rbdrbd to help him get a jump start but his project didn't progress for various reasons so this hasn't been developed past an initial alpha.  Works fine for creating say for example DEx messages manually on the command line etc, if there is a need for it I can polish up a bit.
  https://bitcointalksearch.org/topic/m.3748546
  
  
  
  

Testing & Protocol

• Appendix - Storing data in the blockchain
  Consolidated all the various discussions on transaction storage and drafted an amendment to the spec which would explicitly define how this is done.  Onboarded feedback from the other devs until the approaches detailed in the amendment could be agreed by all.  Subsequently submitted and this now forms part of the spec.  I considered this a fundamental necessity as going into the DEx contest we still had a bunch of ambiguity around transaction storage.  This includes adding obfuscation to Class B & introducing the last byte rotation method for ECDSA point validation.  Significant transaction storage testing prior to submitting the appendix (everything from testing a couple million pubkeys against byte rotation to simulating various transaction storage methods for their efficacy).
  https://github.com/mastercoin-MSC/spec#appendix-a--storing-mastercoin-data-in-the-blockchain
  
  
• Masterchest Consensus System
  Built and introduced the Masterchest consensus system to put the verification API (credit Tachikoma) to use in automated testing.  This monitors each address and evaluates differences across the various implementations.  I believe this has been a very useful tool allowing us to test consensus quickly and identify areas of ambiguity/differences in spec interpretation and lock these up.  Now moving into specific transactions so the consensus system can identify at a glance not only addresses with differences, but highlight which transaction(s) caused said differences.
  https://masterchest.info/consensus.aspx
  
  
• Class A Simplification
  Rewrote the rules on Class A to allow for a variety of edge cases that had appeared on the network but were technically invalid per spec.  These transactions could be decoded via 'Peek & Decode' and it was hurting consensus having ambiguity on these transactions.  Wrote a test version of the library with the simplified logic and performed significant testing against a bunch of edge case transactions to prove the approach was viable before requesting a pull.  I (think) there are now no edge cases at all that aren't catered for in the rules and we're now talking about a bounty & inviting people to try and come up with a bug in the modified Class A spec.
  https://github.com/mastercoin-MSC/spec/pull/36
  https://bitcointalksearch.org/topic/m.4499342
  
  
• Ad-hoc testing as needed (examples simultation testing for multiple change outputs in Class A, testing my own implementation against the other guys etc)
  
  

Thanks
Zathras

Ah, I see. Thanks.

Checking to see If I got the rule on Selling Offers.
A Sell Offer must be cancelled before sending another sell offer is accepted.
(To cancel a sell offer AmountofCoin for Sale to 0 )

The Tx below will be invalid
1b8572009435e554a68fc640f43efa50c9117f6881813ae56eab5b10f9b3fd53     
(Invalid Offer to Sell: 0.01 TMSC.  )

because Offer to Sell     
8b2deeb03c2590bbc6698ab3d663eb920e8bb7e89437a7984a54b8d0154d6741
still open and not yet cancelled?

 

As far as I know, TMSC are only exchanged for BTC. Since the test BTC don't interact with the actual BTC blockchain there's also no way of getting TMSC for testnet bitcoins.
if you try buying TMSC on the masterchain.info DEx with your testnet bitcoins it will just tell you that your testnet address can't be found on the blockchain ( but I just tried it to make sure )

Did we ever fix the balance issue with my account (1LjT88X7Zu8BdbqJw8vfRa83NJuzYL9kqm) Don't really care about the TSMC but just wanted to make sure there wasn't something that could cause a issue later on down the road...

I see 106.12380845 TMSC on Masterchest

108.12380845 TSMC on Masterchain

107.12380845 TSMC on mastercoin-explorer

It seems 107.12380845 is the correct amount (1 Reserved for current sale offer, for a Total of 108.12380845)

Is there anywhere I can buy test mastercoins for testnet bitcoins?
Ideally this would be part of the test plan for the DEx.
I was able to acquire some testnet bitcoins and they work with bitcoin-qt.
When you run 'bitcoin-qt -testnet' it syncs with the blockchain near instantly.
Voila, you've got testnet bitcoins but nowhere to spend them. 

one more thing about available coins:
should it be possible, that a sell offer has more available coins, than are for sale?
i'm talking about this offer: https://masterchain.info/selloffer.html?tx=b9e6b1f6f428441d7e91e7715ed4524de942b8d755367828e1d85ca850a9dc43¤cy=TMSC

it has 0.01 TMSC offered and 0.065 available. screenshot is taken in case that's a temporary issue.

I just bought some TMSC from there, let's see what happens


edit:

ok, after I did the payment and everything was confirmed, the available coins changed to 0, so it seems that was temporary and doesn't cause any issues.

apart from that everything appears to be fine on my side, accepting and paying for the offer was no problem.
at the moment my address doesn't have consensus across all implementations though


edit 2:
@grazcoin: i can now also see addresses that I added to the wallet. I don't know if you changed something, but if you did, it worked

Thanks.
Test 2 + test 3 are known issues (at least to me ;-))
There are also more minor issues with icon color updates - all should be fixed soon.

As for multiple sell offers per address - this is a bug. Currently the logic that checks older offers on the same address is not yet implemented. I wait for deciding which of the pull request is chosen (39,40,41), and then it will be implemented.

Newly created offer should match the amount to sell. This I didn't see. Will be checked.

@grazcoin
Test 1: Theres no partially done/ wrong symbol showing up.
Test 2: Sale offer status does not change to done after the full amount has been bought and paid.
Test 3: After expiration of the bid, the symbol of the sale offer stays half blue/green.
Test 4: The partially done/fully accepted doesnt show up either.
Test 5: Cancelled offers still show up. Someone mentioned it before I think.
Test 7: Sell accept goes through, even though offer was cancelled before.
Test 8: I bid 0.02 from 1NVNkPqSLo4N2seaZCzAkVFPjcGLPJSZAZ, and the amount accepted is listed as 0.02, even though only 0.01 was offered. Someone else bid on it aswell, and both offers show up as accepted.
https://masterchain.info/selloffer.html?tx=b9e6b1f6f428441d7e91e7715ed4524de942b8d755367828e1d85ca850a9dc43¤cy=TMSC

Are multiple sell offers per address a feature or a bug? I've got 3 on one address, 2 of them have the same price.
There's also something off with amounts for sell/available. Take a look at https://masterchain.info/selloffer.html?tx=8b2deeb03c2590bbc6698ab3d663eb920e8bb7e89437a7984a54b8d0154d6741¤cy=TMSC . 0.01 were initially available, 0.002 + 0.007 were bought, so only 0.001 should remain, but amount available shows 0.003.

Also, newly created offers seem to always show 0 as amount available, shouldn't it match the amount for sell?

150 BTC TO BE AWARDED TOMORROW - PLEASE POST A SUMMARY OF YOUR WORK TOWARD THE DISTRIBUTED EXCHANGE

Everyone participating in the current coding contest bounty: please post a summary of all your work to date here in this thread. Go through your old emails, commits, and forum posts. Do not assume that everyone remembers what you did. Please be as complete as possible - you will be reading each others' summaries and using that information to determine payout sizes for each other. Any work which contributed to the distributed exchange should be included, including prerequisite work such as fixing our consensus issues.

If you did both development work and testing work, post separate summaries for each, since those bounties are awarded independently of each other.

As before, I'll ask each of you to tell me how you would divvy up the money (percentage-wise) if you weren't participating in the contest. I realize this is a difficult task, but it is very important, and I have seen first-hand how well you guys work together as a team when you know that you'll be evaluating each other's work at payout time.

Thanks to each of you for your hard work. I hope each of you enjoys the fruit of your labor.

You could send the (signed) tx directly to the bitcoin network via:

It will not make any difference, as the queue position that you see, is not internal to blockchain.info, but an estimation of the status of the tx in the bitcoin network (taking into account its inputs complexity and amount of fee).

In future version the send would use masterchain.info directly.

Transactions before and after worked well, just wanted to mention that transactions can still fail, despite the notification "Transaction sent".
Also, I didn't know that the TX is being sent client-wise.

Btw, is there a way to send the signed transaction using bitcoin-qt? I've tried sendrawtransaction with the signed raw transaction, but I get 'TX rejected (code -22)'.
Would it speed up confirming the TX, since blockchain.info shows a queue position? Or do they broadcast the transaction straight away?