Topic: [Fundraising] Finish “Ultimate blockchain compression” - page 4. (Read 25512 times)

As far as I understand it, it might not be possible to do complete block chain analysis after implementing this change, so there might be some reservations from bitcoin-qt developers to this (as you delete parts of history).

Maybe consider a full fork of Bitcoin-qt ("Bitcoinquick-qt"?) in case this doesn't get accepted to the main bitcoin-qt client. Other than that congratulations on reaching your funding goal so quickly and looking forward to first updates regarding implementation!

Great maaku! I've sent 1 btc.

This is much necessary. Specially for certain use cases such as having colors in the chain.

About the funding alternative to bounties, I want to tell the concrete story about the freicoin initial development one, because I think it's an interesting example of how things can change even when all seems clearly and simply defined (at least it was in my head if you disagree it was well defined).
The bounty asked for the implementation of the new currency based on my simple design draft plus adding merged mining from namecoin's code.
First, my solution to the "exact payment problem" didn't guaranteed that a transaction would survive a reorg. This is an important problem he identified, and after some discussion, he came up with the elegant solution of using a referece Height set by the payer to calculate demurrage. Although the solution was reasonably simple in design terms, what I initially thought would be an "easy task" was starting to get substantially more complicated.
There was some rounding potential problems I missed too. He also implemented some gitian building scripts and adapted p2pool to interact with demurrage. He implemented the modification to send the budgets to the foundation that I didn't wanted at first but the community was asking for.
He didn't took the code for merged mining because it wasn't (and still isn't) a priority.

Well, I was more than happy with his work and the other donor didn't complained when I asked so I just gave the bounty to him (after the fact, so not as useful for him as the crowdfunding campaign). But if there were many more donors, maybe some of them would have blocked the bounty release.

Thank you maaku. My 0.5 BTC because we all want to see it happen.

Wow... that escalated quickly!
Sending few cents anyway

Maaku, instead of whining and complaining about the size of the blockchain, you have stepped up like a man to solve the situation.

You've just been sent a grant of 150 Bitcoins by myself, on behalf of SatoshiDICE, for the ongoing growth and development of this amazing technology we call Bitcoin. (Note to SD shareholders, this is a personal contribution, not coming out of SD earnings)

Further, if after three months, Alan (whom I respect very highly) recommends further development, I will commit to funding another three months (we'll revisit the btc amount based on the exchange rate then).

Please contact me on skype (evoorhees) and we can discuss further. And I agree with Alan that this needs a better name

just a little 0.1 to show my support.

good luck 

@maaku

Thank you.  I think you are a good match for this development activity, as you clearly have the self-motivation and technical expertise to pull it off.  And you have the proper attitude as well -- sober but encouraging. 

One thing we might consider is renaming this proposal.  The thread was started a year ago with the phrase "Ultimate Blockchain Compression" because of a really silly oversight on my part:  that the address-indexed DB would have to be in addition to the txHash-indexed DB.  I originally thought it would be a replacement for the tx-indexed DB, but you clearly need both.  Therefore, this necessarily adds resource usage on top of an optimally-pruning miner.  I think the recent discussion where I realized that you can eliminate all the pointers, makes the overhead much more manageable, but it's still non-negligible.    I don't know your real name, but if you make this work, perhaps it would we could just name it the Reiner-Maaku tree   But I think the "UBC" part should be adjusted since it's not "ultimate compression" (though it does fit on top of an ultimately-compressed blockchain).

I'm happy to contribute to this design where I can.  As you pointed out, I'm quite busy, but I also think this is quite a valuable addition to the protocol.  Even if it somehow turns out to be infeasible, we'll learn a lot from the exercise that will help us do it right on the next try.  However, I'm confident that this will bear fruit with someone capable devoting themselves to it. 

You might consider/research using something other than LevelDB for the full nodes.  LevelDB is very fast and efficient for some things, but I fear it may not be reliable enough for the heavy lifting that will be done by the mining nodes in the [far] future.  For partial/lite nodes, LevelDB makes perfect sense, especially because it's easily bundle-able into end-user software.  But the miners of the future will probably be running dedicated hardware anyway, so they won't be terribly inconvenienced having to have a hardc0re DB engine available to run something like this, with all the robustness/ACID properties that are desired.   Therefore, it would be really nice to have a generic interface (or two interfaces, one for SQL and one for noSQL-key-value-store), so that something like LevelDB can be bundled with the software for "regular" users, and something more concrete can be swapped in for full-mining nodes. 

Also, I should mention that socrates/amiller had some interesting ideas that would be worth discussing.  He had an idea for lite-storage-but-full-verification nodes, which I'm still not confident it is possible, but he insists it is and we agreed to debate it in the future when I had time.  Maybe you can discuss it here before embarking on this journey, to make sure the design is extensible enough to accommodate it if it is feasible.

Update: Please read the latest news here:

https://bitcointalksearch.org/topic/m.3117454

I have approx two weeks left on the first round of funding the members of this forum have so generously provided. I will use that time finish and push the pruning and proof generation code, and time permitting expand the utxosync.py script into a small Django web API for querying the UTXO indicies. This could be used for prototyping wallet software that uses the new proofs. However this will provide no increase in security until bitcoin is soft-forked to include UTXO index commitments, or a merged-mined meta-chain is implemented.

This brings me to the next round funding. I am requesting a final 3 months of time to implement the final revision of the index structure in C++ and using LevelDB, optimize it to achieve sub-second update times (or as close to that performance goal as possible), and either (1) submit a documented pull request adding coinbase UTXO hash commitments to the Satoshi client, or (2) write a proxy server for miners to do their own commitments on a merged-mined meta chain. Which outcome depends on whether acceptance of a coinbase commitment pull request is likely to happen, which in turn depends on the achievable performance metrics which we do not know for certainty at this time.

Because of unexpected costs (insurance) and volatility, I'm forced to increase my monthly cost to 65btc, and will fully cash it out on receipt. I am therefore requesting a total of 195btc to finish the implementation as described above. If this is received by the end of the month, then I can promise the deliverable by the end of the year. Otherwise I will continue work, but at a rate in proportion to funds received, as funds are received ("billing" my time hourly). Please send donations to the same address:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Donation address for implementing phase 2
of "Ultimate blockchain compression"

13snZ4ZyCzaL7358SmgvHGC9AxskqumNxP
-----BEGIN PGP SIGNATURE-----

iQIcBAEBAgAGBQJSLkT7AAoJECsa1YSJj/UDJ0IP/26oyR1COsLs/f+rEvz33kFP
0HtGvsjbEoF+7cetJpIV0eyFomngOWpafr0uhy+uO6mGguPHbXPJlmcgywFdKDwB
pQrRVYcT2DQx+Hfwnhn51QNIJoB6LhnykUi9KrDar8FwbNfYOgLaSUDGqKShtDOC
lc/qVkP56cCvalcqs6a6Q8O0D69BLO+TwifMPJmtdzdnn/2Fs9ONdgXpnnNLGwpJ
g/LWPy9Zdjspq7qoH/p3kFWo2S8TmX5EShsMDM8C4oUTnMjXbBvodJQwm6AzC0KY
XWdg+/W82YpMpmAmhSxqw43/VzUrODw9WAn7cXrCA86/IwhihZnNhLsELYP7Cd77
kgBWR9HE+NILWTRn+x8CONfi6+gk8ZqYsKmcj+PcYbf1u2BAknKb1EVpTwNp2ehD
8y6oNFj99vkDfZz8hSmt8HLn7YbU9jnmJcFqXwCwDFZD+nvWi1dHFeqnJppH3lWX
HaIF3V+psYQuMpglk+fFVrbmF+omCzsd7dArCXk4n+txA8rGIVBD2nWz4MUUxB9e
TLIeVr4EkH+yjvEK00VzjryfINE6mG58hetm1/y4Y/1EvoDATOyZhR91UFB6x/21
+pCagBDSVquc7DVYk7e275PnKSxjM4miAcf88jkO6TvcdiUaiYnYGxZQRCBY89/6
WgWf1x6CQvknPrYT6sZv
=hg1Y
-----END PGP SIGNATURE-----

As always, I welcome questions and comments regarding this proposal and my work towards implementing it, particularly from funders or developers of lightweight clients that might use it.

---

TL;DR: I plan to implement the Ultimate blockchain compression w/ trust-free lite nodes proposal by @etotheipi. Please help me acquire funding work full-time on it by pledging coins to 13snZ4ZyCzaL7358SmgvHGC9AxskqumNxP.

After a startup misadventure, I have recently found myself back on the job market. Rather than take a soul-crushing 8-5 job, I have taken inspiration from @etotheipi and would like to try my hand at full-time open-source development.

In my mind the most critical barrier to the widespread adoption of Bitcoin is the initial block download. Requiring the user to spend days downloading and processing gigabytes of data before getting started is unacceptable, and SPV clients do not provide enough security to be an acceptable alternative for many purposes. We can have our cake and eat it too, but it will require a heck of a lot of grunt-work.

I reject bounties as being inappropriate for a task of this scale. The likely length of the project prevents me from doing this as pay-on-delivery as I would be receiving no income for living expenses between now and then, and I have a wife and daughter to support. It would also force me to develop in secret and take on a hostile stance with possible competing developers who may “scoop” my ideas or code to claim the bounty before me. The rules of the bounty may also prevent me from having the flexibility to alter the scope or structure of the project due to unforeseen considerations and unavoidable complications. I do not believe such an outcome would be good for the community.

Rather, I promise to develop in an open fashion. I will implement the UBC proposal directly in the Satoshi bitcoind daemon/Bitcoin-Qt client, with my changes periodically pushed to a github repository. I will make weekly status reports on a blog, cross-posted to this forum. I will engage with the developer community to get peer review on design choices and address concerns as they come up. The general plan for development would be:

* Implement the PATRICIA trie index of the UTXO set, using LevelDB as the storage backend. Implement p2p messages and rpc commands for querying this index.
* Add support for tracking multiple chains to bitcoind, and implement the merged-mined UTXO meta-chain.
* Restructure bitcoind to allow retrieval of wallet transactions from a UTXO peer, and construction of the pruned block/transaction database from the UTXO set.
* Implement bittorrent distribution of blockchain data, offloading IBD and full UTXO set retrieval from the bitcoin protocol.
* Switch the default Bitcoin-Qt behavior to perform IBD in the following manner:

1. Download bitcoin block headers and verify proof-of-work.
2. Download coinbase transactions for most recent X bitcoin blocks.
3. Determine head of UTXO meta-chain, retrieve meta-chain block header, and extract UTXO root hash.
4. Fetch wallet transactions from peer, with UTXO proof.
5. Download UTXO set (fast) and construct full transaction validation database.
6. Begin operation as a validating transaction processing node (total time elapsed: minutes).
7. Download entire block chain history (slow) and verify blocks.
8. (Potentially) begin operation as a mining node (total time elapsed: hours/days).

(Update: please read this blog post for my current views regarding SPV+ support in the Satoshi client, in view of the fact that the Satoshi client is no longer the officially recommended wallet app. Implementation of SPV+ mode in lightweight clients such as MultiBit or Electrum has taken priority, although recent changes to the Satoshi client make SPV+ mode more plausible.)

I expect that full implementation of this proposal would take 4-6 months. It might take longer. I am requesting 150 BTC (or equivalent fiat) to cover rent, food, insurance, medical bills, etc. for the first 3 months in lieu of taking a job to support my family. Obviously this will not be sufficient time to complete the project, but I am expecting to make significant progress and certainly have a better idea of how much follow-on work would be required. You may then evaluate whether a further donation is a worthwhile investment for you.

I have started a blockchain.info wallet for this purpose.

13snZ4ZyCzaL7358SmgvHGC9AxskqumNxP

If it does not reach 150BTC, I will return all contributions. Please reply in this thread or PM me if you have any questions, and I will be happy to answer them. I will also be a panelist at Bitcoin 2013 and would be happy to meet with anyone in person.

CREDENTIALS

I implemented Freicoin, which adds interest/demurrage to the bitcoin blockchain. This required considerably more design and simulation than one might naïvely expect, as interest obviously depends on timing, transactions lack timestamps, and block time is untrustworthy. The solution developed for Freicoin is original and works well in practice.

More recently I have implemented an improved difficulty adjustment algorithm for bitcoin protocol currencies, replacing the 2016-block average with an optimal Parks-McClellan filter co-designed by @galambo and tweaked to give very quick response response times (adjustments every 9 blocks) while maintaining accuracy and avoiding instability.

I have also co-designed with @jtimon a colored coin proposal that adds interest-bearing assets and a distributed, peer-to-peer exchange. I will shortly be posting this as a separate bounty / work proposal.

Outside of bitcoin, I have worked for several years at NASA-Ames Research Center, doing various web development & data analysis & visualization tasks for the planetary science and astrobiology institutes.

EDIT: After less than a day, the funding goal has been achieved. Thank you! As there are still contributions coming in, I will treat it as a rolling fundraiser for continued development after the 3 months have elapsed. This is a project that will take at least 4-6 months, and possibly more. The project will officially start June 1st, after I have wrapped up some other obligations.