do miners zip blocks when trying to propagate them? - page 3.

adamstgBit

legendary

Activity: 1904

Merit: 1037

Trusted Bitcoiner

Quote from: brg444 on September 09, 2015, 10:08:24 PM

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

bitcoin does not currently compress blocks for propagation in anyway.

do you think we should explore this idea, or should we stick to 1MB limit because fees?

"most basic innerworkings"

is that an oxymoron?

brg444

hero member

Activity: 644

Merit: 504

Bitcoin replaces central, not commercial, banks

I mean.. we're really trying to have rational discussions about scaling and security yet some of you are ignorant of some of the most basic innerworkings of Bitcoin.

Then you wonder why no one cares or value your opinion...

Peter R

legendary

Activity: 1162

Merit: 1007

Quote from: achow101 on September 09, 2015, 09:50:06 PM

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

This would be true if the receiver (another node or miner) didn't already know a lot about the contents of the block already. However, since most nodes have fairly similar mempools, and since block are built from transactions held in a miners mempool, this redundancy can be exploited to achieve compression. Examples of this type of compression include the Relay Network, the IBLT proposal, or Danny's Merkle Tree idea (which I like, BTW).

adamstgBit

legendary

Activity: 1904

Merit: 1037

Trusted Bitcoiner

Quote from: achow101 on September 09, 2015, 09:50:06 PM

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

your probably right.

Quote from: Peter R on September 09, 2015, 09:41:47 PM

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.

However, the Corallo Relay Network does support a sort of compression. Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice.

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.

Compression of solved blocks has the effect of reducing the network propagation impedance. It results in lower fees to users (positive) but also lower fees to spammers (negative). This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):

Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

this "Corallo Relay Network" sounds like what bitcoin needs to scale properly. 250 coding gain, that's great!

achow101

staff

Activity: 3458

Merit: 6793

Just writing some code

I don't think blocks are easily compressed. The data is rather random since hashes are random and in order to preserve all of the data so it can be verified, it must be losslessly compressed. However, since there isn't that much of a pattern to the data, it becomes rather difficult to losslessly compress blocks with a good compression ratio.

Peter R

legendary

Activity: 1162

Merit: 1007

The p2p protocol presently only supports propagation of solved blocks in full; i.e., blocks are not compressed.

However, the Corallo Relay Network does support a sort of compression. Rather than transmitting all the transactions in a solved blocks, since most the other miners know about them already, it just transmits indices that refer to each transaction (sort of like a map for how the TXs fit in the block). Greg Maxwell claims that the Corallo Relay Network attains a coding gain of about 250 (1 MB is compressed to about 4 kilobytes); however, I believe it is less in practice.

Techniques like invertible bloom lookup tables (IBLTs) could also be used to compress solved blocks in the future; Rusty Russell is presently researching this possibility.

Compression of solved blocks has the effect of reducing the network propagation impedance. It results in lower fees to users (positive) but also lower fees to spammers (negative). This chart shows how fees decline with improvements to network propagation impedance (x-axis in the chart):

Source: https://dl.dropboxusercontent.com/u/43331625/feemarket.pdf

adamstgBit

legendary

Activity: 1904

Merit: 1037

Trusted Bitcoiner

Quote from: DannyHamilton on September 09, 2015, 09:17:32 PM

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block. Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise). Then peers can just request any transactions they are missing from their immediate peers.

zipping a few MBs has got to be a less then 1 second operation no?

but just sending the header, and merkle tree, would require even less bandwidth than sending the the hole block zipped.

so there you go, bitcoin can have GB blocks without requiring miners to use a huge amount of bandwidth propagate these huge blocks

miners would simply be required to keep up with the TPS.

DannyHamilton

legendary

Activity: 3472

Merit: 4801

Seems like a bad idea.

Some other miner could finish solving and broadcast a block during the time a miner wastes compressing the block. Then the miner is sitting on a worthless compressed orphan block.

Seems it would be better to have a protocol that allows the miner to just broadcast the 80 byte header, then the merkle tree.

Since most peers will already have most transactions, there's no need to re-broadcast the entire megabyte worth of them (compressed or otherwise). Then peers can just request any transactions they are missing from their immediate peers.

monthlyemployee1

newbie

Activity: 2

Merit: 0

I don't think this would be very effective. It would only cause extra steps to need to be taken to verify them

adamstgBit

legendary

Activity: 1904

Merit: 1037

Trusted Bitcoiner

anyone know?

what kind of compression do they get on a 1MB block?

Topic: do miners zip blocks when trying to propagate them? - page 3. (Read 1854 times)