Topic: Bitcoin based Blockchain compression algorithm (Read 3690 times)

Well if you want this to be adopted by Bitcoin Core, you will have to put it into the master branch. As I said earlier, 0.8.6 is old and outdated. It is 5 major releases behind, during which time the code has been refactored, changed, and shuffled around. Furthermore 0.8.6 has some vulnerabilities in it that were solved in later releases.

If you want to be taken seriously, make it for the latest master. No one cares about what you can do, clearly you can program and read code, it shouldn't be hard to figure out where to put it in the master branch. Bitcoin Core is very well documented and commented.

Almost everyone who contributes to Bitcoin Core does it for free. Sure the compression algo is possible on 0.8.6, but will it still work on master? Why don't you try it out.

I do agree though that gmaxwell shouldn't have moved this as it is about Bitcoin Core, albeit an old version.

Don't be put off, everyone in crypto has their own bias & agenda depending on what coins they own, what projects they're following, and who pays their wages. Keep doing your own stuff!

You guy hurt my heart, 
I offer my ideas for free, and provide the source code that can be compiled,
It is used to test and prove that this compression algorithm is possible,
I'm doing this for free, I just hope to help bitcoin, make it better.

As you know, a new code need more time to study,
I am familiar with the 0.8.6 version, it is easy to transplant the compression algorithm code to 0.8.6 version.

You got served

0.8.6 is what most altcoins are based on, it is an old codebase with many vulnerabilities. Also look at the screenshots. This is in the wrong subforum.

Why don't you try integrating this into the master branch of Bitcoin? 0.8.6 is incredibly old and outdated software and should not be used anymore.

Bitcoin version 0.8.6 integrated blockchain compression algorithm source code has been upload:
https://github.com/Bit-Net/Bitcoin-0.8.6

These source code just for test, it can compile and run in ubuntu and windows,
And it is compatible with the bitcoin, does not fork bitcoin.

is the new wallet ready yet?

Bitcoin version 0.8.6 integrated blockchain compression algorithm source code has been upload:
https://github.com/Bit-Net/Bitcoin-0.8.6

Recently I will upload the complete source code to GitHub,
It is integrated in the bitcoin version 0.8.6   .

Thanks

Fast compression algorithms can trade very slow I/O time, for much faster cpu time - thus speeding things up in general.

For example, if 100 mb can go down to 80, reducing read, write, seek operations, the overall time of a process involving disk I/O on these 80mb can be sped up significantly.

Now if you have a much better compression algorithm that takes 100mb to 60mb but takes a couple seconds instead of msecs to do the job in order to gain the extra -20mb of compression - that doesn't work to speed things up in local operations. It could be still useful though for network transmission because those 20mbs could take more than a few seconds to get transmitted.

Another reason is that if you are marginal with the blockchain in a particular storage medium, especially SSDs, you can then fit the blockchain after compressing it. That will allow you to exploit the faster SSD instead of going with the slower mechanical.

The blockchain won't fit in a 64gb SSD right now, but it can fit compressed (say a BTRFS partition with compression) - although it doesn't have much headroom for SSD operations that require some free space. In some time, it won't fit in a properly configured 120gb ssd running an OS (assuming it has like 40gb for linux, swap, and free space that SSDs require) but it will fit if the data is compressed. The gains there are exclusively from using the faster SSD over the mechanical disk.

In any case, from searching the dev mailing list in older discussions I think some devs are reserved in using popular compression algorithms due to security concerns and exploits.

Very interesting stuff. There are cases where hdd space for node is not a problem as someone mentioned above, but there are cases this is problem. Same with traffic, some have problem with it, some don't.
It would be great to allow to choose in configuration to use compression on your node (if hdd space is problem) or not (if you have a lot of free hdd space). Similiar to GZIP in web/browser traffic.

Like knightdk said, the best way to get it into Bitcoin Core is to do a pull request.  A few other suggestions:
1. If you create unit-tests and/or other tests that will also increase the speed at which it is evaluated and potentially merged.

2. If you were to create an option to enable it conditionally for both disk compression and network compression, so that it could be running on some nodes that were testing it vs an all-or-nothing approach to test it, I also think that would increase the likelihood of adoption.

3. Similarly, for the network bandwidth there might need to be a way for nodes running this compression code to identify each other so that it could serve compressed data when possible and uncompressed data (e.g. for non-upgraded nodes) at other times (this may already be implemented, I didn't check the code).  Or alternatively be able to send either compressed or uncompressed dynamically through some type of negotiation.

4. You might also consider that some people (e.g. miners) might choose to serve uncompressed blocks when they have found a new block so as to minimize latency.  e.g.  I would serve compressed blocks for old blocks, but for one I just found, I might just send it uncompressed to avoid the time taken to compress it.  It could be a trade off - is it faster to send an uncompressed block or to compress and then send.  Much would depend on the network speed of connected nodes and I am not sure if anyone has tested which is faster to do.

In short, the more groundwork that is done, the better.

In general, on disk and sending old blocks that are compressed is a nice feature.  

My thoughts.

Our compression algorithm is also a lossless compression,
We tested on the bitcoin, compression and decompression about 200,000 blocks without any problem,
And Blockchain compression rates may be the highest,
And it's not just compression.

Best thing I read all day lol

I bet Hooli did an under the table deal with Lempel–Ziv on this lossless algo we're seeing here 

  Not sure he got it

Our compression algorithm is binary compression,
The core algorithm is LZMA, it is better than LZ4.

Hashes and encryption are extremely resistive to ASCII based compression algorithms so it is likely you are at best compressing the script and small transactions may actually get bigger. Have you reached out the the UPX developers for a look at binary compression?