Pages:
Author

Topic: A World of Trust – eMunie Consensus Primer (Read 7372 times)

member
Activity: 96
Merit: 10
September 04, 2015, 05:49:42 PM
1) It would have to be a really bad day for the internet globally if all of the eligible nodes at any time are seeing latencies approaching 60 seconds.

2) Eventually yeah, but then that will likely also mean the network is dead and there aren't any client nodes coming online to make transactions either.  Same as if all the miners went offline never to return.

1) Of course, steady state you don't expect to see latencies like that, but connection interruptions and crashes do happen. How does a crashed node, which has been disconnected for, say 1 day know how to resync and who the correct nodes are to listen to?

2) What I mean is that if you have set of N nodes, who propose future nodes M from the subset of N, nodes going offline will lead to 0 eligible nodes if M must always be a subset of N.
Well, any eMunie client, new or offline for several days, can fully sync even when only non-service nodes such as a client only.  Some of the client only also have the full ledgers. I proved this by syncing a new client with just the client only faucet as the only active node.
legendary
Activity: 1176
Merit: 1134
there is no way to predict which SN will get the next transaction as it is based on human behavior

Ahem. You need to prove mathematically and holistically that is a random process and not subject to game theory. That is essentially the fundamental problem with PoS, its entropy is bounded unlike PoW where the entropy is external.
How does http://blog.cr.yp.to/20140205-entropy.html relate to the assertion that bounded entropy is bad? Bitcoin's reused R value attacks arise since it requires new entropy for each transaction, while with the curve25519 this does not seem to be the case.

James
legendary
Activity: 1008
Merit: 1007
1) It would have to be a really bad day for the internet globally if all of the eligible nodes at any time are seeing latencies approaching 60 seconds.

2) Eventually yeah, but then that will likely also mean the network is dead and there aren't any client nodes coming online to make transactions either.  Same as if all the miners went offline never to return.

1) Of course, steady state you don't expect to see latencies like that, but connection interruptions and crashes do happen. How does a crashed node, which has been disconnected for, say 1 day know how to resync and who the correct nodes are to listen to?

2) What I mean is that if you have set of N nodes, who propose future nodes M from the subset of N, nodes going offline will lead to 0 eligible nodes if M must always be a subset of N.
legendary
Activity: 1050
Merit: 1016
1) No, there is a set interval of 60 seconds (which is also the transaction failure duration) where eligible nodes are valid.  So say that the time now is 00:01:00, a set of nodes are valid, at 00:02:00 a different set of nodes are valid based on the endorsements made in the interval 00:00:00 -> 00:01:00.  This should allow even the most latent of nodes to keep pace.

2) Hmm I'm not sure how you have come to that conclusion.  If nodes go offline, then come online again later, they will receive endorsements again, which will increase the eligible set of nodes again.

Yes I'm aware of that, the FBA algorithm is the replacement for that broken one.

1) I suppose it depend on what bounds the latency is under. I agree that 60 latency is hilariously long for nominal operations, but its always the edge cases which cause the problems

2) If nodes go off-line never to return, doesn't that reduce the set eligible set to 0 on the limit?

Regarding FBA - I thought that was the broken design, not the new one?

1) It would have to be a really bad day for the internet globally if all of the eligible nodes at any time are seeing latencies approaching 60 seconds.

2) Eventually yeah, but then that will likely also mean the network is dead and there aren't any client nodes coming online to make transactions either.  Same as if all the miners went offline never to return.
legendary
Activity: 1008
Merit: 1007
1) No, there is a set interval of 60 seconds (which is also the transaction failure duration) where eligible nodes are valid.  So say that the time now is 00:01:00, a set of nodes are valid, at 00:02:00 a different set of nodes are valid based on the endorsements made in the interval 00:00:00 -> 00:01:00.  This should allow even the most latent of nodes to keep pace.

2) Hmm I'm not sure how you have come to that conclusion.  If nodes go offline, then come online again later, they will receive endorsements again, which will increase the eligible set of nodes again.

Yes I'm aware of that, the FBA algorithm is the replacement for that broken one.

1) I suppose it depend on what bounds the latency is under. I agree that 60 latency is hilariously long for nominal operations, but its always the edge cases which cause the problems

2) If nodes go off-line never to return, doesn't that reduce the set eligible set to 0 on the limit?

Regarding FBA - I thought that was the broken design, not the new one?
legendary
Activity: 1050
Merit: 1016
Endorsements are public, so anyone can see which nodes have been endorsed and when.   You simply use a set of the most recent endorsements to determine who the current consensus nodes are for a transaction at the time it was created (and if the time is forged, it doesn't matter).

This works, is reliable and robust for a number of reasons:

1  Only endorsed nodes that were included in final transactions, voted for by a previous set of eligible nodes are selected, so you have a reliable set that everyone can agree.  Consensus on the transactions results in consensus of a future agreed set.

2  Providing that the selection set is recent, then you have a very high chance that those nodes are still online and that they will be around to take part in the consensus process.  Thus you have a reliable majority available.  Only in rare edge cases would a majority of those nodes have all suddenly gone offline.

Two questions:

1) Is is possible that the eligible set of nodes changes so rapidly, that nodes with high latency don't have time to pick up on what the agreed set should be?

2) Doesn't this lead to an ever decreasing set of eligible nodes, since nodes will go offline and only nodes that are eligible are from the previous round?

Quote
The exception is FBA (Stellar), but I'm not sure if I 100% align with the philosophy of it, as having nodes pick their own consensus nodes raises a few flags for me.

You are aware of the fact that stellar's consensus was so completely broken that they had to resort to running only one validating node? The reason being that they were unable to deal with forks, which the consensus didn't expect to be possible.

1) No, there is a set interval of 60 seconds (which is also the transaction failure duration) where eligible nodes are valid.  So say that the time now is 00:01:00, a set of nodes are valid, at 00:02:00 a different set of nodes are valid based on the endorsements made in the interval 00:00:00 -> 00:01:00.  This should allow even the most latent of nodes to keep pace.

2) Hmm I'm not sure how you have come to that conclusion.  If nodes go offline, then come online again later, they will receive endorsements again, which will increase the eligible set of nodes again.

Yes I'm aware of that, the FBA algorithm is the replacement for that broken one.
legendary
Activity: 1008
Merit: 1007
Endorsements are public, so anyone can see which nodes have been endorsed and when.   You simply use a set of the most recent endorsements to determine who the current consensus nodes are for a transaction at the time it was created (and if the time is forged, it doesn't matter).

This works, is reliable and robust for a number of reasons:

1  Only endorsed nodes that were included in final transactions, voted for by a previous set of eligible nodes are selected, so you have a reliable set that everyone can agree.  Consensus on the transactions results in consensus of a future agreed set.

2  Providing that the selection set is recent, then you have a very high chance that those nodes are still online and that they will be around to take part in the consensus process.  Thus you have a reliable majority available.  Only in rare edge cases would a majority of those nodes have all suddenly gone offline.

Two questions:

1) Is is possible that the eligible set of nodes changes so rapidly, that nodes with high latency don't have time to pick up on what the agreed set should be?

2) Doesn't this lead to an ever decreasing set of eligible nodes, since nodes will go offline and only nodes that are eligible are from the previous round?

Quote
The exception is FBA (Stellar), but I'm not sure if I 100% align with the philosophy of it, as having nodes pick their own consensus nodes raises a few flags for me.

You are aware of the fact that stellar's consensus was so completely broken that they had to resort to running only one validating node? The reason being that they were unable to deal with forks, which the consensus didn't expect to be possible.
newbie
Activity: 54
Merit: 0
Which do you want?

Sorry not meaning to be a pain. I think as you said, this is evidence that you can't discuss this piecemeal. There needs to be a white paper.

Hi TPTB/AnonyMint,

I bet some of us are actually happy with Fuserleer sharing/discussing "chapters" as he writes them rather than him holding them back until he finished writing the whole book.  Time may be limited once the book is finished - so any head start we can get to wrap our heads around the apparently radical solution to crypto that is "eMunie", the sooner we can support the nascent ecosystem when/if it launches.

One option for you and those wanting answers ASAP is to hold off reading the early deliveries until after the last delivery (when the details arrive in full).   That way both groups are happy: the read/discuss-it-in-stages types, and the read/discuss-it-in-one-fell-swoop types.

cheers,
- Wingspan.

p.s.  I've been a beta tester for 2 years almost, and while it is taking a long time, I think we're in for a wonderful treat - a gift from Fuserleer - a crypto that has a shot at replacing fiat for billions.
sr. member
Activity: 420
Merit: 262
Which do you want?

Sorry not meaning to be a pain. I think as you said, this is evidence that you can't discuss this piecemeal. There needs to be a white paper.
legendary
Activity: 1050
Merit: 1016
there is no way to predict which SN will get the next transaction as it is based on human behavior

Ahem. You need to prove mathematically and holistically that is a random process and not subject to game theory. That is essentially the fundamental problem with PoS, its entropy is bounded unlike PoW where the entropy is external.

*sigh* please, for once, just read it for what it is and accept the general concept of what I am explaining.

In normal operating conditions it is random.  I am fully aware that this will not always be the case, but every time I try to explain something clearly, someone jumps in and complicates it.  If I was to explain something taking into account all the attack vectors, 99% of people wouldn't understand it.

Which do you want?
sr. member
Activity: 420
Merit: 262
there is no way to predict which SN will get the next transaction as it is based on human behavior

Ahem. You need to prove mathematically and holistically that is a random process and not subject to game theory. That is essentially the fundamental problem with PoS, its entropy is bounded unlike PoW where the entropy is external.
legendary
Activity: 1050
Merit: 1016
The hardest to answer Fuserleer topic so far IMO, let's see what the answer is:

This is one of the critical flaws in existing crypto currency. If you know of an exception, please enlighten me.

I remember you having many problems with reputation systems for a variety of reasons, but the #1 problem I see with them is that the worse case scenario, government co-opting of cryptocurrency to create "surveillance coin", would be done by linking biometric data to a single wallet user address instead of having many pseudo anon addresses like Bitcoin.  The easiest way for them to accomplish this is with a reputation system derived currency.

What is the question here? :|  Or are you commenting on monsterer's question?

A service node would run a dedicated wallet for services, and any wallets the user might use on it for day to day activities are completely decoupled and separate.  There is no way to tie a SNID to a regular wallet on the same machine, as the addresses of regular actions are never public.

Only the address of the service wallet is revealed, and even then it is masked as a 12 byte ID
legendary
Activity: 1050
Merit: 1016
There has to be a set of selected nodes that everyone agrees on, otherwise you run into some problems if a majority of those nodes are offline at the time of a transaction, a majority vote can never be reached.

This is the key to understanding how this can work in practice... How do you arrive at a consensus for who your consensus nodes are?

This is quite easy to explain, and it is essentially a random set.

Remember endorsements, they are the key.  These are acquired through TXNs that produce transactions, and there is no way to predict which SN will get the next transaction as it is based on human behavior, and also depends on which nodes are connected to a TXN at the time of creation.

Endorsements are public, so anyone can see which nodes have been endorsed and when.   You simply use a set of the most recent endorsements to determine who the current consensus nodes are for a transaction at the time it was created (and if the time is forged, it doesn't matter).

This works, is reliable and robust for a number of reasons:

1  Only endorsed nodes that were included in final transactions, voted for by a previous set of eligible nodes are selected, so you have a reliable set that everyone can agree.  Consensus on the transactions results in consensus of a future agreed set.

2  Providing that the selection set is recent, then you have a very high chance that those nodes are still online and that they will be around to take part in the consensus process.  Thus you have a reliable majority available.  Only in rare edge cases would a majority of those nodes have all suddenly gone offline.

3  You are still able to calculate what the majority trust value should be, as you'll be able to easily calculate what the portion of total network trust is that those nodes control.

4  The same probabilities and information theory limits apply no matter set size, whether it be all the nodes in the network, or a handful.

The trust consensus is similar to other attempts in a few ways, but this is (as far as I know) the only one so far that has a agreeable, globally verifiable set of ever changing consensus nodes.  Most consensus algorithms have a fixed static set that is easy to attack (as you know who the nodes are going to be at all times), or it is a leader-follower set up which is not suitable for P2P systems.

The exception is FBA (Stellar), but I'm not sure if I 100% align with the philosophy of it, as having nodes pick their own consensus nodes raises a few flags for me.
legendary
Activity: 1260
Merit: 1000
The hardest to answer Fuserleer topic so far IMO, let's see what the answer is:

This is one of the critical flaws in existing crypto currency. If you know of an exception, please enlighten me.

I remember you having many problems with reputation systems for a variety of reasons, but the #1 problem I see with them is that the worse case scenario, government co-opting of cryptocurrency to create "surveillance coin", would be done by linking biometric data to a single wallet user address instead of having many pseudo anon addresses like Bitcoin.  The easiest way for them to accomplish this is with a reputation system derived currency.
legendary
Activity: 1008
Merit: 1007
There has to be a set of selected nodes that everyone agrees on, otherwise you run into some problems if a majority of those nodes are offline at the time of a transaction, a majority vote can never be reached.

This is the key to understanding how this can work in practice... How do you arrive at a consensus for who your consensus nodes are?
legendary
Activity: 2142
Merit: 1009
Newbie
After the formal white paper, you can work on how to explain what is in that white paper to layman. Don't put the cart before the horse.

I know a cryptocoin that was launched without a whitepaper but still became pretty popular...
legendary
Activity: 1050
Merit: 1016
Fuserleer, honestly I would hire some academics to make a formal paper, or do it yourself if you think you are qualified.

The more you waste time trying to piecemeal the design to layman, the more confusion and the less certain you will be that you've caught every flaw.

Before coding, I was very focused on making sure the white papers were precise, so that coding can proceed from that which is well specified.

I really can't make a final determination on your design until it is fully specified with a mathematical model. That intended to be a strong statement of my fairness.

After the formal white paper, you can work on how to explain what is in that white paper to layman. Don't put the cart before the horse.

Yeah, I (mistakenly) thought that a higher level overview would assist those that are not as technical to have a general understanding and provide an initial look for more technical people until the paper is completed soon.  

Perhaps that is a good idea in a different arena, but here the audience is mainly technical to some degree, so all its done is raise questions, even if the general understanding is appreciated.

I have some academics in my immediate council, they aren't professors or anything of similar caliber, but they know how to write a paper so that is ongoing at the moment with guidance from myself.
sr. member
Activity: 420
Merit: 262
Fuserleer, honestly I would hire some academics to make a formal paper, or do it yourself if you think you are qualified.

The more you waste time trying to piecemeal the design to layman, the more confusion and the less certain you will be that you've caught every flaw.

Before coding, I was very focused on making sure the white papers were precise, so that coding can proceed from that which is well specified.

I really can't make a final determination on your design until it is fully specified with a mathematical model. That intended to be a strong statement of my fairness.

After the formal white paper, you can work on how to explain what is in that white paper to layman. Don't put the cart before the horse.
legendary
Activity: 1050
Merit: 1016
Please excuse me if this is ignorant but if there are no blocks how does one know when a tx is "confirmed" ?

How do I know there is not another tx around the corner that has an even highter consensus value that some of the trusted nodes just haven't seen yet ?

Is there a specific time that a tx needs to be on the ledger for so it can be considered confirmed ?
Is there a certain threshhold of consensus value that a tx needs before it is actually added to the ledger at which point every future tx will be disregarded ?

Transactions are not committed to the ledger until there is a majority vote on them by the currently selected and agreed group of voters.  Votes are final, so by effect consensus is too.

The ledger is append-only, so once a majority consensus between the set of voters has been reached, any conflicting transactions are simply ignored.

If a transaction is not agreed upon within 4 vote rounds (30 seconds each) from its declared timestamp it is discarded.  If it is presented after 4 vote rounds have passed since its declared timestamp, it is automatically discarded and to send those assets a new transaction would have to be recreated with a current timestamp and resulting hash.

Thanks for explaining. I must not read anymore primers before my first coffee. I hadn't even realized that there are agreed upon voters. I though trusted nodes just sign and then at some point all nodes in the network will have gotten it signed by one of their trusted nodes.

Hehe no problem, perhaps I shouldn't write anymore high level primers either.  It seems to have caused more confusion than it was worth! Smiley

There has to be a set of selected nodes that everyone agrees on, otherwise you run into some problems if a majority of those nodes are offline at the time of a transaction, a majority vote can never be reached.

Perhaps I should amend the article to make that, and some other points more clear.
hero member
Activity: 980
Merit: 1001
Please excuse me if this is ignorant but if there are no blocks how does one know when a tx is "confirmed" ?

How do I know there is not another tx around the corner that has an even highter consensus value that some of the trusted nodes just haven't seen yet ?

Is there a specific time that a tx needs to be on the ledger for so it can be considered confirmed ?
Is there a certain threshhold of consensus value that a tx needs before it is actually added to the ledger at which point every future tx will be disregarded ?

Transactions are not committed to the ledger until there is a majority vote on them by the currently selected and agreed group of voters.  Votes are final, so by effect consensus is too.

The ledger is append-only, so once a majority consensus between the set of voters has been reached, any conflicting transactions are simply ignored.

If a transaction is not agreed upon within 4 vote rounds (30 seconds each) from its declared timestamp it is discarded.  If it is presented after 4 vote rounds have passed since its declared timestamp, it is automatically discarded and to send those assets a new transaction would have to be recreated with a current timestamp and resulting hash.

Thanks for explaining. I must not read anymore primers before my first coffee. I hadn't even realized that there are agreed upon voters. I though trusted nodes just sign and then at some point all nodes in the network will have gotten it signed by one of their trusted nodes.
Pages:
Jump to: