Topic: Mempool synchronization (Read 2381 times)
@Mike Hearn, I understand it's not currently an issue, or will be for a while. I should have been clear about that. This thought came out of an investigation of the technical limits of the system, and a repeated claim that mempool synchronization was a theoretically impossible problem (clearly it's not).
Firstly, I don't think "anonymous mining" requiring very low bandwidth has really been proven with data. But let's put that to one side.
We don't even sync blocks as full txids today. Doing so isn't particularly hard though:
1) Set a mem pool limiting policy. I think there are pull requests floating around to do that.
2) Send the mempool message at node connect time.
3) Set a full match Bloom filter on every connection so blocks get synced as partial merkle trees.
Done. Now if for some reason miners are still suffering from too high orphan rates, it might make sense to push things even further. But I bet that's not a real issue right now.
We don't even sync blocks as full txids today. Doing so isn't particularly hard though:
1) Set a mem pool limiting policy. I think there are pull requests floating around to do that.
2) Send the mempool message at node connect time.
3) Set a full match Bloom filter on every connection so blocks get synced as partial merkle trees.
Done. Now if for some reason miners are still suffering from too high orphan rates, it might make sense to push things even further. But I bet that's not a real issue right now.
There is a fundamental problem of policy. Namely, why do you get to set my mempool policy instead of me?
Because you recognize that we're all much better off if I do to some reasonable extent. Nash equilibrium and all. Note that this doesn't mean I get to choose what you include, only what you don't exclude, and only if I can prove that there's a reasonable chance that it'll soon make its way into a block. Of course you're the final arbiter of what's reasonable, so I wouldn't want to overstep my bounds. But at the same time, if your boundaries are unreasonable, then I'm free to ignore you. 
There is a fundamental problem of policy. Namely, why do you get to set my mempool policy instead of me?
It's well known that if mempools are well-synchronized, then only a few bytes per tx hash (the txid) are required to uniquely identify txs in mempools during the propagation of new blocks. This has the potential to significantly lower orphan rates and bandwidth requirements of miners, and keep the anonymous mining of large blocks feasible.
I've heard some devs describe the problem of synchronizing mempools as the same one that Bitcoin itself solves - namely, agreeing upon a single consistent transaction ordering - and so a solution isn't theoretically possible. I can see this would be true if mempools didn't permit conflicting spends and transaction orderings; is this the case?
If mempools are designed to accommodate potentially conflicting spends and transaction orderings, then they can be inclusive to all potentially valid transaction sets waiting to be included in blocks, making the synchronization problem simple. The only potential problem then would be spam, which could be dealt with by nodes requiring periodic proof of work for persistent mempool inclusion - something miners would be producing anyway, and would have ample incentive to widely distribute. Furthermore, because having transactions downloaded and verified in advance of receiving them in new blocks lightens the load on a node, it's in one's interest to make sure they've got the txs miners are working into future blocks. It's also in their interest to kick nodes that don't maintain good synchronization in order to lower overall orphan rates for the good of the network - less wasted hashing power means more secure transactions for all, including them.
Essentially the mempool would encode a set of directed acyclic graphs, each of which describe valid tx orderings, and upon receiving the txids in a new block, a node would check whether the DAG they describe is in the aforementioned set.
I'm admittedly not familiar with how the mempool is currently designed, but the above characterization of the synchronization problem suggested to me that this might be the way to think about a solution.
I've heard some devs describe the problem of synchronizing mempools as the same one that Bitcoin itself solves - namely, agreeing upon a single consistent transaction ordering - and so a solution isn't theoretically possible. I can see this would be true if mempools didn't permit conflicting spends and transaction orderings; is this the case?
If mempools are designed to accommodate potentially conflicting spends and transaction orderings, then they can be inclusive to all potentially valid transaction sets waiting to be included in blocks, making the synchronization problem simple. The only potential problem then would be spam, which could be dealt with by nodes requiring periodic proof of work for persistent mempool inclusion - something miners would be producing anyway, and would have ample incentive to widely distribute. Furthermore, because having transactions downloaded and verified in advance of receiving them in new blocks lightens the load on a node, it's in one's interest to make sure they've got the txs miners are working into future blocks. It's also in their interest to kick nodes that don't maintain good synchronization in order to lower overall orphan rates for the good of the network - less wasted hashing power means more secure transactions for all, including them.
Essentially the mempool would encode a set of directed acyclic graphs, each of which describe valid tx orderings, and upon receiving the txids in a new block, a node would check whether the DAG they describe is in the aforementioned set.
I'm admittedly not familiar with how the mempool is currently designed, but the above characterization of the synchronization problem suggested to me that this might be the way to think about a solution.
Jump to: