Topic: What is sharding, and will sharding scale Bitcoin? (Read 787 times)

Hello Ali!


This is already done using altcoins. And for POS chains the security is stronger the wider the distribution of coins. I see no Security benefit to pooling POS coins. (POW chains are a different matter)

If you can't communicate cross chain, I would say it is preferable to have them separate - they can already communicate using smart contracts / HTLC / Layer2 etc.

Increasing the TPS means can be a security threat and we see sharding can be a solution but will sharding also be a security threat to a blockchain?

I think it was vitalik that said blockchain is meant to be slow like 300K transaction. If a blockchain can process more than 10K per second, it may have some security issues.

I don't think so. On the contrary cross shard transaction is exactly what you should prohibit to avoid complexities.
In my scheme, we keep any segment of the state self-contained and to resist state take-over attacks we have higher level shards in charge.
Keeping coins on same shard and avoiding cross-shard transactions would be easy by applying the two rules I've proposed before:
1- Transactions should use inputs from same shard.
2- Using a light nonce, wallets should guarantee the transactions they generate falling in the same shard with their inputs (which follow rule #1 any way).

Belonging to different shards do not make two utxos any different because they all belong to and are maintained by the root shard eventually. You can think of the root shard as a very slow, yet heavy blockchain with gigabyte blocks (like 4 GB) and long blocktimes (like 1-2 hours). Bitcoin using such a scheme scales up to 1500 tps and more.

Although the name hierarchical resembles such thing, it is not what it looks like in the first glance:
It is absolutely possible to make it hard enough for upper shards (maintained by fewer full nodes because of the costs) to do anything bad against the network by applying a new technique: collaborative proof of work. In my model, upper level shards are free to accumulate works done in their subshards as part of their proof.

To commit any sort of misbehavior ( intentional chain rewrites for double-spending, censorship attempts, ... ) they need to collide with their subshards or ignore the work they have encapsulated in their blocks.
Any miner in any shard has to provide a block with a total T work. To do this, the miner needs a nonce for the block header with T/2 difficulty and the Merkle root of the header should refer to two blocks from its left and right subshards with a total difficulty of at least T/2.
It is a bit more complicated and yes, it is really, really going down the rabbit hole. For instance it is not just about a subshard's single block difficulty, it is about a chain of blocks with unknown length that are being committed to an upper level block, and it is obviously recursive process, ...

Sharding is not simple problem but the solution should be kept as simple as possible it is why I don't like what they are doing in Ethereum, for now I'm becoming convinced that tree chain and hierarchical sharding is the most go direction.

The way Sharding is being 'attempted' on ETH only works for POS.  It does not work for POW.

The Reason :

They use sub-sampling of the POS stake holders to limit which accounts can mine which shard. It is impossible for more than the the allocated POS miners to mine a shard than the percentage allocated.

So 10% of the pos holders can mine each 10% shard. This changes every block. So you are randomly allocated a different shard to work on.

You can't do this in POW as there is no way of preventing a POW miner from adding his hash-rate to the shard of his choice. Subjective vs Objective miners.

I think this is a clever trick! But it does not fix the numerous other outstanding issues.

Clearly you MUST be allowed to perform transactions cross shard - or you destroy fungibilty. I have looked at the cross sharding communication algo.. err.. LOL! It's very complicated and messy. I still need to be shown _proof_  that this will in fact reduce traffic and not increase it.

The Security of the Shard-POS chain running the original ETH algo can be copied _exactly_. (As with ANY POS chain) In fact I can increase the security by simply increasing the distribution of stake. I can also charge less for gas.  Why would a new user choose a more expensive less secure chain ?  (You can't do this with POW)

Anyone who thinks this Hierarchical Tree Structure doesn't centralise faster than Usain Bolt - Is missing something.

I wish them Luck! I love seeing cool teck _attempted_

..

(I'm way way down the rabbit hole myself at the moment..)

Thank you for sharing your invaluable opinion. But what's your opinion on aliashraf's proposal of a low-security shard for Bitcoin micro-payments?

Plus in the context of that, wouldn't an off-chain layer for micro-payments be a more feasible solution?

I think the nice thing about LN is though that it improves scalability without making any sacrifices on the lower layer.

That is, it does leave room for alternative scalability approaches as well. So while there's a lot of focus on LN right now I don't think that in practice it detracts that much attention from possible alternatives. There will be dev teams dedicated to other approaches just like dev teams are dedicated to LN right now. Maybe these approaches will end up as part of Bitcoin, maybe they won't. Point being, while many people may see LN as the scaling solution to end all scaling problems, many people don't. And those will still keep working on other ways to improve crypto.

Stop. Why is it "not Bitcoin", in your opinion? Because Bitcoins in Lightning channels are Bitcoins under multi-sig escrow, with transactions stored locally, that can be broadcasted on-chain at a later date, or time.

Because it is not bitcoin, and bitcoin is the only monetary system that is truly decentralized and censorship resistant. It is very easy to have 2nd layer applications that do excellent job on top of bitcoin but they involve either trust-->centralization or liquidity-->centralization just like LN.

It is more than obvious that an onchain solution is superior to any 2nd layer one no matter how fancy it would be. It is no magic, LN, you deposit some coins and bitcoin acts as an escrow thanks to its HTLC feature, the LN node does the same and you both now have a channel. When it comes to arbitrary trades the LN node needs to deposit even more with other nodes to act as a router and it escalates liquidity problem and puts people with more deposits in a central hub position eventually, it resembles traditional banking systems rather than cryptocurrency.

I understand LN now looks like an adventure but it is not bitcoin and it is very important to keep investigating onchain scalability solutions instead of saying:
"hey, LN solves everything why should we bother even thinking about improvements? Isn't it just more sensible to keep bitcoin safe and uncompromised and focus on LN?"
It is what they say, by the way, and it is where they are diverging from bitcoin without even being aware of their divergence! And it is sad. A decade of experiments and discussions and debates is just going to be denied because some guys are not good in strategy in spite of their expertise in math or coding.

But instead of "micro-payment shards", why not a micro-payment, off-chain layer developed, and built on top of the Bitcoin blockchain?

I believe there is already one, called the Lightning Network.

َAlthough I do agree with you and @d5000 to have such factors considered, it is worth mentioning that basics come first in the context of discussing fundamental issues like sharding.

For instance, in my hierarchical sharding proposal, we deliberately "widen the surface" to let both utilities (micropayments) and attacks become reasonably cheap, users enjoy instant micropayments as long as the shard is not attacked and when an attacker with like 1/1000 hashpower (which is still huge) for political reasons takes over a shard, users start to wait for upper level confirmations now the attacker need to undertake multiple times of more costs to dominate upper level shard and forcing users to wait even more (by committing to higher level shards). The safety measure is always the same: wait more to be safe more.

The interesting point in this scenario is built-in damage control: to cause a serious damage attackers need to undertake orders of magnitude higher costs. From a game theoretic  point of view such a system is considered secure. Actually in this regard my proposal is much safer than what bitcoin is now because with current topology a reckless trader may commit to a high stake transaction with less than secure confirmation numbers and remain exposed to a short-range chain rewrite attack because unlike what most people think you don't need  50%+1 hashrate for double-spending in bitcoin:
an attacker with 10% hashrate has a rough 20% chance to commit a one block rewrite attack! i.e. If an attacker with 10% power could convince traders to release funds with 1 confirmation after few trades he would be able defraud some of them by making the last block orphan and double-spending the funds.

Crypto-politics might also come into play. Because sharding has widened Bitcoin's attack surface, then it will be attacked, maybe by Bitcoin Cash supporters, Core haters, or state-sponsored attackers. It would also be profitable to do it because the attackers can create doubt, and short sell Bitcoin.

@mda,
I'm strongly against the necessity of any kind of intra-shard transactions and have proposed a solution to avoid such transactions:
Rule #1 : Any transaction should use inputs from same shard.
Rule  #2: Any transaction Id should fall in the same shard as of its inputs.

For a network consisted of 1024 leaf shards, a wallet could satisfy second condition by flipping a nonce in few microseconds for commodity mobile devices.

As of grouping transactions in blocks according to their shards, it is what exactly happens when a parent commits to child blocks by merging their hashes. In my proposed model,  leaf shards have their own chain which is usable for micropayments with low security requirements, upper level shards only commit to this chains (no ultra-shard transactions) for  consolidation and security purposes. The root chain consists of blocks that have the same structure as you are talking about: each block has a Merkle path partitioned recursively. Any node in any level can verify the subset of Merkle path it maintains necessary data for.

We are so close in our thoughts, mine is more complicated because I offer a gradual security level and give users freedom to wait for any level of security they wish. I can Imagine a root chain with few hours block time and leafs with 10 seconds. A multi million dollars transaction should wait to find its way to root chain and get 6 confirmations (like in 12 hours) but a 5 bucks payment for a cup of coffee is secured in few seconds.

I don't see the point of this complexity. Why not just squeeze these two leaves in the same block. User downloads and verifies his leaf up to Merkle root because in Merkle tree transactions are arranged by shard. There is some overhead because Merkle paths are required to verify atomic swaps between shards but this can be reduced by grouping atomic swap transactions by shard as well. This overhead is actually the only restraint otherwise the scaling would be limitless.

Edit:
There is an additional small problem to consider. When we split a shard in two by transaction id to scale further, amounts of coins in two children will be different. Atomic swap rate between them will be nowhere near 1:1 leading to rampant speculation. Therefore users will have to anticipate these splits and divide their coins in advance to hedge value.

1- In my model there is no intra-shard transaction ever:
State is partitioned by means of  transaction id and new transactions are supposed to use utxns from a same shard and (by adding a nonce) should stay on the same shard as their inputs.

2- There is almost nothing an attacker would be able to do other than double-spending because higher level shards commit to the lower shards eventually.

What I'm thinking of is a top-down commitment (the parent finalizes childs, childs light-check the parent) from root to sub-chains. that makes any low cost attack totally void. What I'm working on is a good incentive mechanism for distributing hashpower in all depths of hierarchy. It wouldn't be enough to distribute rewards according to the ratio of difficulty of each shard against total network difficulty because fees are collected by miners in the leaf level but higher levels consume more resources to maintain larger state shards and validate bigger blocks.