Topic: Isis ATP [Automated Trading Platform] - Discussion - page 3. (Read 45225 times)

I'm thinking this product is going to need it's own website in the near future.  The OpenPay version is going to be called just ATP, not to cut Aido out but to distinguish it so people don't get confused.  I plan to take it to the point it works reliably and then open it up to the broader community.  Fortunately I'm now at a spot in life again where I can do that.

Now as to the cert.  If your cert came down as www.mtgox.com you grabbed the wrong cert.  You need to grab the cert from the TOP of the certificate authority chain.  The exact directions I followed myself are here... http://codebistro.com/2010/03/25/adding-cacert-to-the-java-trusted-store

For firefox you click on the lock icon, go to "view certificate", click "details".  Then you will see the ca cert at the top of the list that's the one you want it won't be MtGox it will be StartCom or whatever.  Select that one then click export.  Then follow the directions for your platform for installing a new Certificate Authority cert.  In windows remember that you need to do this from a command line and you need to do it with a command shell elevated to administrator (type cmd, right click select "run as administrator").  For linux you will need su or sudo and I'm guessing it's similar for OSX.

You'll know you made a mistake if you get a "file access error" from the keytool command.

Sorry it's so complicated.  We may be able to package the CA cert as a resource and just have a "first run" state that must be run as root for setting this part up.  Again I'm only doing that if we decide to roll a proper installer.

BTW Aido, that section that was messed up is not your fault, it's mine.  I recognize my own code style there, I'm just shaking my head in disbelief that I didn't notice it earlier.  I know better and I would have caught something like that in a code review.
It would be a very good idea as we refactor to create a series of unit tests to validate the complete state of the application.  Also things like trade blocking should be set at the application level from here on out and the application should be refactored to be a bit more like a state machine.  I'm working out in my head the changes I plan to make to this, but it will happen in phases.  There won't be any instances where we throw the baby out with the bathwater anymore.

Ok, so back to the new design.  I think the idea of programmable logic is an admirable idea, but when someone is just setting up for the first time, there needs to be a logical set of defaults included.  In my fork, I've set the default logic to be VWAPCross, not because I believe it is in any way inherently better, but because it's the simplest and easiest to understand.  Last trade was below vwap we buy, above vwap we sell.  As it stands now if you don't put anything into the bid/ask logic fields it spits out a nasty stack trace and does nothing.  I've fixed that in my branch, but it's refusing to let me commit at the moment.  I'm sort of at a loss on that one and will try to massage it a bit tomorrow.

After we refactor to use the order book and limit orders, the next step will be to add in a proper AI.  Right now, all we are using are algorithmic traders.  The AI will be optional and until it's trained with a few weeks of data it runs a very real risk of making very expensive mistakes.  However it's "learning profile" is a stored data set that can be downloaded and updated regularly with new market information.

Because it will attempt to predict the market direction, it will necessarily be significantly different than the algo traders we have now.  

The AI is a predictive DSP with a genetic algorithm based on a project I wrote a few years back for a company that couldn't afford downtime (downtime meant lives lost not just dollars).  The AI would watch all systems globally and learn about which systems were failing.  At first it just watches everything and reports failures, then it begins to make predictions and give each prediction a weight based on if it comes true or not.  Eventually the strongest predictions survive and the weakest ones die out.
The goal here was to compile an alert of probabilities and send it as far in advance as possible.  
The longer it runs the smarter it gets.  With a failure rate of 1 machine per day it could predict up to 4 hours in advance with 90% certainty.

I've been running it against MtGox since I first created the Isis-ATP project and it's a bit novel in that it can deduce the lump sum direction of the algorithmic traders at play (after all that's what a DSP does it isolates signals from noise).

Since January it's been showing that the pricing structure is actually being supported by VWAPCross and someone somewhere appears to be running a stochastic oscillator and that someone has some serious dollars.  That's pretty much all the bots and it accounts for over 80% of the trading activity.  The remaining is human traders (random noise tends to be from human activity).  

This is MtGox data only.  I haven't tried it against any other exchange and the DDOS attacks a few weeks ago gave the poor thing brain damage for lack of a better word. I'm not sure how to reconcile the dataset so I'll start a new profile when I integrate it.

I will also be adding an internal wallet and a configurable "sweep" to allow you to get your profits out daily and still keep enough on hand for expected volume.

Finally I will also be adding peer to peer mode to allow nodes to specialize in exchanges and share resources.

If that doesn't make a lot of sense imagine this...
There is a significant arbitrage opportunity between btc-e & mtgox on BTC right now.  You can buy bitcoins cheaper on BTC-e and sell them for more on MtGox (feel free to check this).

Each node can be configured to specialize in a particular exchange and this is where the bulk of it's money is at.  

If we can predict the size and length of an arbitrage opportunity we can do the buying and selling individually and settle up the profits later.  This obviously only works if you are working with close friends and people you know.  But the fact is this can work in the right circumstances.

Imagine if I had $5000 USD on BTC-e and Aido (sorry you're volunteered), has a few hundred BTC on MtGox.  I can authorize my machine to spend the $5000 USD into BTC on BTC-e and Aido can authorize his machine to spend the BTC on MtGox into USD.  I am effectively lending him my USD while he in turn is loaning me his BTC.

Once the arbitrage window is over (or we run out of cash), then we settle and split the profits.

Obviously this system would require trust which is scare in the bitcoin community, but if the transactions were small enough and there were enough people running it and we added a "confidence" ranking of some sort based on previous interactions, this thing has real potential to make serious money for anyone who cares to participate while not requiring everyone to have a hundred thousand accounts everywhere to try and keep track of.  

It's optional and will be default disabled, but just be aware this is one really easy way to run arbitrage without so much risk from window expiry.
Hope you guys find it exciting.  There is quite a bit of work to do here, but I think it will all be worth it in the end.

wooo! wish i tried btc-e sooner on it. I'd still like to get gox going so I can take advantage of arbitrage (guess i could use bitstamp or something but gox generally seems to be highest). I'm sure i'll figure out the certs sooner or later, i think I have most of it figured out---to anyone else stuck on this, I found this link to be the most useful so far: http://www.opentox.org/tutorials/q-edit/how-to-install-ssl-certificates
Not sure if I'll have to install a cert for each error message I got or if I just need a single one from gox but I think I'll figure it out. Thanks to those who've offered help.


PS
Perhaps this thread should be restarted as AidoATP? I've gone through all 30 pages here way too many times for bits and pieces of information. Just a thought since the platform now seems like its getting off the ground all and all and now discussions seem to be more regarding the fine tuning seems more to be logic oriented than structural.

I will take some "credit" for this cock-up. I messed around with Nova's arbitrage code and how it was disabling the trend trading. The reason the problem wasn't caught earlier was because, like Nova, I didn't fully trust the arbitrage engine so mostly disabled it when running live.

I had given the arbitrage code a total overhaul and implemented a Bellman-Ford algorithm to calculate opportunities. From my brief testing, it found the same opportunities as the original code so I re-implemented the original again.

I'm also stuck at this point:
"Impl.checkServerTrusted(Unknown Sour......................................
..........Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to
 find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown
 Source)
        at java.security.cert.CertPathBuilder.build(Unknown Source)
        ... 27 more"


I've exported the cert from mtgox but I dunno what to do with it or how to install it.
I used keytool as outlined here (http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html?) to try and load it but to no avail...When I exported it it was saved as "www.mtgox.com" which i renamed as mtgox.cer and used "java keytool -import -alias gox -file mtgox.cer" but i get "Error: could not find or load main class". I've been doing everything from a command prompt in win7, should I be using some java distro or something instead?

can anybody detail exactly what cert(s) are needed? daybyter was nice enough to link me some guides that explained the installation process but I'm still not clear on the process. If anybody could share an example of the code needed w/ one of the certs it'd be greatly appreciated. I'm really not at all familiar with java but if I have the codes/certs i think i can follow easily enough. I tried saving: http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java as a .jar and running it from command prompt but clearly I don't know what the hell i'm doing

I haven't tried using btc-e yet but I imagine I'll run into the same problem right? Are there any other specific certs I need for btc-e?


I know i've asked alot of questions but if anybody can even answer one of them i'd really appreciate it, thanks.

I noticed that once the arbitrage engine fired and ran into a condition where I didn't have money in the from-currency wallet, that trend trading would continue on as normal but trend-trades would no longer show up in my account history, even though I could see the little engine vigorously trying to make me some money.

I've solved the problem.  There is an exception thrown in the ArbitrageEngine that is being caught, but there is also a boolean set/unset that wraps the call to execute.  This boolean disables trend trading temporarily, but with the exception being thrown before it has a chance to re-enable, the trend trading was being shut off and never turned back on.

The solution was to move the code to re-enable trendtrading to a finally block, since those are called whether or not an exception is thrown.

We should probably make sure that if we are doing a try / catch wrapping functionality that will disable something temporarily, that we either move the whole thing outside the try catch block, or at least place the re-enabling functionality into a finally block.  This should be our coding standard for the future.  

Most likely the cock-up was on my end.  I'm just surprised it went this long without being found out or noticed.  That probably translates to a lot of money that was left on the table and it makes me sad to know everyone would have missed out on it because of such a simple mistake.

I'm committing all of these changes to the OpenPay repo under the ATP directory once I can see this thing run for 24hrs.

There seems to be a huge amount of opportunity in JPY/GBP according to the arbitrage engine right now.
It's been alerting me for 5 hours about a chance for 150% profits if only I had some yen in MtGox.
Just to test it I sold 1BTC into Yen and restarted the engine, then waited. 
10 seconds later I had a bunch of GBP which amounted to $150 or so.(pulled the USD/GBP rate from my forex account to do the math)

Have to be careful with the arbitrage engine though. 
I wrote the dang thing and I still don't trust it. 
Among other things, it doesn't care about market depth and just tries to buy the whole dang market out, immediately.
It's the reason for a solid month I owned all the Rubles on MtGox and couldn't get rid of them, I had to sell my rubles at a rather significant loss in the end.

Anyways, if anyone has some Yen in MtGox, the arbitrage engine will turn that into a nice profit right now assuming you have the ability to do something with GBP later.

Talking to a friend of mine who performs security audits on enterprise systems, I've found that this can also be set by the encryption policy.  There is a java setting somewhere to tell java to accept invalid certs (it's a VM flag or something).  So maybe that will resolve the issue.  If not you will need to install the cert and we will have to keep this problem in mind should we ever choose to roll an installer.

My code has a feature to download certs from exchanges, but cannot add it to the keystore as a regular user AFAIK.

Cross-Site trading: I've started some code to get data from btc-e to withdraw and deposit. Would be cool, if someone came up with the rest of the code to do so. Someone in this forum mentioned, that there's an API method to perform such stuff, but this method is not public and has to be enabled by the sysadmin there....
So I'd be interested to collaborate to complete my code, that just parses and posts to the website.

I've investigated this further and found that it's because mtgox cheaped out and started using a cert from a company called startcom.
Java does not recognize this company as a valid certificate authority and thus does not recognize the certificate as being valid.
You will need to visit https://data.mtgox.com/api/2 (you will get an error on the page but just ignore it).
You will need to download the cert and manually install it. You can do this in most cases by clicking the "lock" icon in your browser and going to "export".

I don't think there is a way to automate this particular process.  If someone comes up with one please let me know.
 

Wow. Some very big plans there.. ( and heavy coding)

And all very good ideas. I really look forward to seeing this in action

The cross exchange dealing would be Amazing and so would the local wallet
To protect When an exchange gets closed or hacked

Thank you for the hard work and again welcome back into the mix
All the best from me

Well reliability, but more importantly they're a HUGE single target.
That and I noticed there is a significant arbitrage opportunity across exchanges if we get the cross exchange thing functioning correctly.  First part though is converting to limit orders.  I don't think we're taking the order book into account either, might be neat to pull that data in and see what kind of discovery we can do, esp if the market depth is there.

Cross exchange arbitrage has a significant delay though.  I just manually arb'd a trade that took 2 hours.  Original expected profit was $50 on the trade, by the time I could fully execute it I was down to $3.00  

Also I'm concerned about the stability of the exchanges in general.  Even the venerable gox could one day be shut down.  While I can do nothing about fiat balances, it might be good to have a solution in place for bitcoins.

What I'm considering here would mean integrating a wallet (encrypted of course), that most of your money is swept into.  A minimum balance would be kept on the exchanges for active trading and when an arbitrage opportunity opens up it would start buying on x and selling on y until the opportunity is no longer profitable, or becomes the least profitable option.  When not arbing, the engine would keep enough in the exchanges to fund an average of 1 days trading + 10% this way significant opportunities aren't missed (as often), but balances are kept close to home.

Just a thought.

Lovely...


I'm guessing gox is using a self signed cert on their SSL connection??

Welcome back nova

Looking forward to you working with Aido

Lets get Isis / aidoATP

Working soon on multiple exchanges

Mtgox. Is far too unreliable

Thank you

Ian

This is an interesting error.

Anyone else seeing it?  (First run of a new checkout)

I've opened a BTC-e account and have been active on there.  Now I have a few bucks on the exchange, I'll see what I can do to get limit orders working.

I've realized most of the time when I suffer a loss it's because the quote and the execute prices are not the same as the system was expecting, i.e. order lag. 
It may only vary by a few cents, but those pennies add up quick and negatively impact the bottom line.

My code change would make limit orders the default. That way the system doesn't accidentally suffer a loss when the market turns against it in the time it takes it to decide an order needs to be created and the time that order finally executes at market pricing. 

Back in the day this was my original plan, but I got lazy because market orders were just so much easier (no need to track & cancel if needed).

I've managed to recover my account key at github, I'll grab Aido's code, do a merge, then push to the Isis-ATP repo (possibly with a new project designator), then change things over to the limit system and issue a pull request to Aido.

This would be a HUGE breaking change, so please for the love of God and everything holy, run the dang thing in simulation mode or only with money you can afford to lose.  We know what problems this change will fix, but we do not know yet what problems this change will cause.

BTW my non-compete officially expired today, so let's see what the knowledge I gained working for a major financial services firm building automated trading systems can do for a little ol' open source project like this

So just to clarify, AidoATP is or is not working with MtGox at the moment?  Sorry having trouble following the thread.

Also meowmeowbrowncow

Did you have any luck with the other exchanges ?

Do you have them working on your version of AidoATP
Can they trade?

Thanks

Thank you for doing that

I've opened an issue with the XChange team and Mazi to investigate.  Two things happening:


1.  MtGox changed, slightly, their URI paths for their v1 API.  A slight change is a major f'up.

2.  XChange 1.7 using the MtGox v2 API should work, but it isn't.  There is a problem with Mazi.


So, as soon as I hear back I'll provide any response that gets my copy working.