Ok so the time has come to discuss the peer to peer networking aspects of this application.
I was originally going to use a system similar to bitmessage where an anonymous encrypted message is passed around from machine to machine, each machine making an attempt to decrypt the message. Obviously if you can decrypt the message then it must be meant for you.
While that's a nice idea, the system load at scale is horrific. Bitmessage drains my laptop from a fullcharge to 0 in less than 30 mins. I normally get 4 hours of regular use on a charge. Also I simulated it and realized that all it would take is someone flooding the system with junk messages and the entire network could be denied. Thus routing in this method is not optimal for our particular situation. I still love bitmessage, it's just not the right tool for what we're accomplishing here.
So here is the solution I've worked up and what will be in the v1 when it's released.
The basic trading principle is one of aggregation and diffusion or put another way, scatter & gather.
There is a common channel where broadcast messages (trades) are sent to all highly connected peers by other highly connected peers.
A trade request enters the broadcast channel and is diffused into a series of smaller trade requests through a collection of closed loop tightly connected peers (called a peernet).
The peers in a peernet quickly work amongst themselves to attempt to facilitate as much of the trade as possible by matching their own internal resources against the trade. If the trade matches some threshold that is set on a per peer basis (by it's owner) then the peer produces a partial match.
These smaller partial trades are aggregated by the primary node and presented to the channel as a match.
The two primary match nodes then communicate directly to find a settlement option that works for both of them and then they perform settlement according to the agreed upon rules.
Here is an example.
I am connected to MtGox & BTC-e, my bot notices a $10 descrepancy and wants to arbitrage that for as long as the window remains open.
I have 100BTC and $500USD. My bot directly arbitrages the trade within it's own darknet but sees that the window is still open.
My trading window has netted me 150BTC and I now have $0 USD to purchase more.
My bot would make an announcement (to it's peer controller, more on that later) of 150BTC@USD:BO:SEPA|OKPAY|PAYPAL|DWOLLA|ACH (BO is best offer, the others are proposed settlement methods in order of preference).
This is carried onto the primary channel where it is picked up by 4 primary nodes and diffused/scattered to 40 secondary nodes.
The first node is connected to SEPA, the second through OKPAY etc.(They could be multiconnected, but for now let's assume a single node/settlement method).
Rather than any single node taking the full risk of the transaction, each secondary node comes back to it's primary node with a partial match according to it's own internal rules.
Node1, subnode 1 matches 1BTC@110USD:SEPA, subnode 2 matches
[email protected]:SEPA etc. For a total of 40BTC@105USD:SEPA
Node2, subnode 1 matches 1BTC@117USD:OKPAY, subnode 2 matches
[email protected]:OKPAY etc for a total of 90BTC@110USD:OKPAY
...
This goes on until 150BTC have been matched.
My node then presents settlement instructions to the primary nodes directly. My node then sends the offered BTC to each primary node. The primary nodes perform the fulfillment and distribute the BTC to their secondary nodes and settle according to their own negotiated rules.
Obviously there is a bit of trust involved here. However there are rules in place to prevent a node from becoming a primary node if it has trouble with fulfillment, furthermore each primary node is in charge of rating it's secondary nodes to the network. I call this trustnet and it's something I'll get into in another post, but I sort of touched upon it before.
Now as to the network itself...
When the application is launched for the very first time it will generate a UUID and an encryption certificate.
It will join 4 or 5 IRC servers on a specific channel, let's say #expressnet
Unless you have selected the darknet option and put in your friends UUIDs, once joined, it will poll a list of all nodes (clients) sitting on that channel and perform a DCC offer to each client currently sitting there.
For our purposes a peernet and a darknet are essentially the same thing.
The difference is that a darknet is a "pre-authorized" peernet, for instance you and a group of close friends would constitute a darknet.
It's a self agreed network operating to the exclusion of others, it is "dark" to the outside world.
In the trustnet setting, a darknet node is automatically given a trustscore that you assign. If you fully control it you may as well assign it a 10.
A good example for using a darknet might be someone with nodes located close to each major exchange who just wants to create a network directly between those nodes for the purposes of direct arbitrage. Personally I would see this option being used as a way of becoming a "major player" on the expressnet, due to the highspeed interconnects and direct connection to sources of liquidity, but I digress.
If you've selected the darknet option, it will only poll for pre-authorized UUIDs and only communicate with those pre-authed IDs.
(There is a mixed dark/open option too, but it's waiting on the trustnet implementation which will be a few more days before I can talk about in depth).
Back to the implementation details...
A DCC roundtrip time will be calculated between every node.
A new client will then sort the nodes into an array ordered by roundtrip time.
The default nodelist size is currently 12, but that number is configurable.
On a quad core that's probably the max you want, with about +2 nodes per additional core if your machine is going to be fully dedicated to this task.
If this machine won't be fully dedicated to trading, I recommend cutting that size in half. Eventually I'll put in a self optimizing algorithm, but for now this is what we've got.
DCC is a direct connection and it is setup between two peers, bypassing the IRC server once the connection has been established.
It is beneficial to use this method rather than rolling our own because most firewalls and routers understand it and will co-operate, there is therefore no need to try and explain ports and configuration which frankly is more of a headache than it's worth.
However it's unencrypted and none of the libraries I could find knew how to use transport layer encryption without puking in the event of a self signed cert.
Therefore we will need to implement a TLS style, Diffie Helman key-exchange and secure the message body itself while in transit.
This will produce a series of loosely connected closed loop peernets (or darknets) that are great for high speed clearing between themselves, but will suck for long range message traversal. Obviously messages under this scheme will come in, but they won't come out.
For most purposes this is what we want, we want to trade frequently with people in close physical proximity to ourselves.
However there are two classes of message, standard and broadcast.
Standard messages are internal control and prep signaling, essentially the system is going to try and aggregate multiple smaller trades in the closed loop peernets.
It will also work in the reverse, that is to say if a big offer is detected, it's going to try and fulfill it by diffusing it through the local darknet and presenting it as a single completed trade to the network. (darknet & peernet are interchangeable here).
Broadcast messages are reserved for the big trades.
To correct for this, one peernet node will remain in channel and when it receives a broadcast message it hasn't seen before and cannot be completed internally, it will relay the unfulfilled portion back on to IRC, exactly once (ideally aggregated with other internal offers that cannot be fulfilled internally).
Only one node per peernet will be given voice or authority to speak on behalf of the peernet. This is to limit inchannel traffic to only large trades that cannot be diffused or aggregated through a local peernet.
It will be enforced at least in the beginning by a bot operator that will set the voice attribute only on peers that have been particularly long lived on the channel.
Note that everything I've described above is extremely similar to how bitcoin does peer discovery, except that it doesn't do the voice exit to IRC option.
I hope this is exciting and I'm posting it here to invite feedback. It is mostly code complete, however if anyone sees a significant problem with it, I would be glad to take it into consideration, I'm open to suggestions here.
Enjoy!
