Topic: [IDEA] - Bitcoin-Powered Database (Read 2211 times)

With Cascasius' approach you'd still need to audit the code + trust that the people signing the file etc. are trustworthy and really who they claim to be. Yes, it might be an acceptable risk, but having the block chain yourself and doing the calculations and eventual pruning yourself is still what you need to be ultimately sure. For that you need to have all 200k blocks (pre-pruned or not).

D&T: Yes, you don't need 100% of all transactions, "only" 100% of all currently unspent transactions (something that's not always easy to find out) + block headers. Eventually it might also be nice to have then still stored/hashed in merkle trees, so you can hand at least pruned blocks to the P2P network. It can be estimated that in the future, clearly spent transactions will be pruned as fast as possible, things like this database though might start to bloat these unspent TX. On the other hand, store and delete operations would then be possible, leading to a write being a delete + a new store.

Not true.  Due to the Merkle tree structure spent tx can be pruned away and without any trust a miner can reconstruct the entire blockchain back to the geneis block.  The only missing parts are the spent tx and dead ends.

You do not need the full blockchain to be a miner.  You DO need a full set of all unpsent outputs and all block headers.  They aren't the same thing.

I geez I should have clicked the link in the OP before i wrote my reply down i was just in a hurry dont mind me, I'll just keep following closly anyways this is interesting subject

Publishing DB backups (minus sensitive information) is a step in the right direction but if we were to build a blockchain tx approach then it could provide a common protocol that could be used by different rating applications so that in the end users could pick and choose which specific ratings they care about (whilst also being certain that the data hasn't been altered after initial publication).

EDIT: Ooops I forgot which thread I was posting this in - the above is relevant to a rating system rather than a generalised DB.

I'm not sure I follow. Can you rephrase your question/s?

Almost. It still assumes all copies of the nightlies won't be destroyed by an attacker (meaning, that someone will actually download the nightlies for safe keeping). This is quite reasonable in practice.

Aren't their SQL database tables that have a "write and read" option?

This is a good point your bringing up.... Having hashing power to verify SQL databases... Publicly?

Intermediate solution: publish nightlies and store just their hashes in the blockchain.

Seems like it solves the problems with both approaches to me.

I am only proposing trust tx's to be in a/the blockchain - the measurements of them would be entirely up to software apps that read the blockchain (and different apps could have very different ways of measuring it).

They actually do not need the whole block chain. They merely need a reliable set of unspent tx outputs, and your challenge concerns the term "reliable", not the term "whole block chain".  what you are saying is that the only way to know you have the accurate set of outputs is to download a large amount of extra data so your computer can say "yes this is it", and you are saying something different than what I am saying.

If I could download a digest of the first 200000 blocks - a digest that was produced by a published open source utility and there was a wide public consensus that the digest file with hash X is what the utility produces no matter who runs it therefore is a faithful record of all unspent txouts of blocks 1-200000, with lots of verifiable GPG signatures from respected people attesting to it, I would have no problem trusting that versus spending days downloading a huge file just to know I got the right one. And I definitely would be able to mine with it.

How would your trust system be measured? Would you pin the trust system to USD?

+1

Am not quite sure why having public DB records of trust votes == elitism. The idea I had in mind was simply a system of publishing specific rating tx's but nothing to do with how they are interpreted (so different apps could take the raw data and draw very different conclusions depending on how they want to do that).

I am very much against the "web of trust" as it is only a continuation of elitism. There are other ways to apply trust to Bitcoin, but a universal database isn't one of them. A Ripple type system would provide the trust a lender might need. Ripple will work best in local economies, but still need a way to trade globally. Ripple/Bitcoin exchanges might do that.

I think that this idea is something that could be very useful for creating a web-of-trust system where all "rating" entries become a part of the "public" DB (I made a suggestion along these lines to https://bitcointalksearch.org/topic/website-for-all-lenders-and-borrowers-87012).

sounds like a job for namecoin

They definitely DO need the whole block chain, otherwise they might include an invalid/double spent transaction and loose a block, because this block won't be accepted by anyone.

This "reliable" set of unspent transactions can only be produced by someone you trust (Bitcoin = 0 trust to others!) or yourself - and this person needs to have the whole block chain to be reliable in filtering out bad transactions.

Mining is not verifying the past chain. Miners need nothing more than the same data - a reliable set of unspent tx outputs - to do their work.  They definitely do not need the full block chain.

I know "normal users" won't download the blockchain - people will migrate to lightweight/thin clients anyway pretty soon. However, miners will still need to download the chain in order to verify it. As you know, miners are not volunteers, but have an incentive to mine and verify blocks. Miners could delete ancient history, but I'm not sure they will really choose to, as 100GB isn't really that much to keep around on modern hard drives. Moore's Law will help keep the cost of storing the ever-increasing blockchain small, compared to the mining rewards / tx fees.

The minimal storage burden for being a full client in a fully optimized Bitcoin network is to store all unspent transactions (and only the unspent outputs thereof), which is a very small fraction of the total blockchain size, which grows only in proportion to coin ownership becoming "fragmented" (i.e. the total number of unique bitcoin addresses possessing coins).  Any storage beyond that is consumed merely to cryptographically vouch for the integrity of that first data set.  

The only reason to store the entire transaction history is to favor of some highly conservative security assumptions that are important for the proof of concept, but which are impractical to the point of ridicule.  If 85% of the block chain file exists solely for the benefit of the client's ability to verify the block chain isn't tampered with, and I am able to download a block chain digest 15% of the size and there is some other way for me to establish the accuracy/veracity of the unspent transaction set it represents, and my user experience is all the same, I'm likely to opt for the latter.

Imagine if online banking had two options, http and https, and using https required you to wait 8 hours each time you wanted to use online banking.  Nobody would use it.  People would just bank offline, or take their chances on getting attacked.

Nor is anyone going to ever download the block chain - EVER - when it gets to, say 100GB+, other than self-appointed historians who want to make sure the original block chain never dies.  Everyone else not using a hosted service will be downloading pruned or digested block chains.  As the block chain size tends to infinity, the number of volunteer historians will tend to zero.

If only a dwindling number of historians will download the block chain when it gets so big, you're back to the same problem as square one: you're depending on volunteers to do your backups for you.  You may as well let your volunteers back up your data directly via http, so they are only backing up data they have a stake in (something they might actually do out of prudence), rather than every gig of everybody's data who mistakenly thinks the block chain is their own personal closet, toilet, and dumping ground.

Yeah, my idea isn't original ... all I propose is to standardize and open source the protocol and client library.

You will not hold the blockchain on your personal hard drive forever. Eventually it will migrate to dedicated miner servers with strong hard drives.

If this "feature" of Bitcoin, the ability of its users to pay money in exchange for everybody to store their data forever, is a fundemental unsolvable property of the blockchain (in other words, if future-proof methods of encoding data in the chain exist), then you have no choice in the matter - to participate as a full client in the Bitcoin network, you'll have to store this data. I wonder if this is (yet another) Tragedy of the Commons situation ...

The cost of storing something in the chain is fixed - whatever the fee is at the time of encoding. The overall cost to Bitcoin miners over the future could be unbounded (depends how quickly the price of storage/network solutions drops). So this might be an inherent externality in the protocol (because the net benefit to everyone, of storing data in the chain might be negative). However, this does't mean Bitcoin is doomed because it will become everybody's favorite porn hard drive - if Bitcoin's overall net benefit to users is greater than this negative cost (and I suspect it is greater by several orders of magnitude), then Bitcoin will continue buzzing along despite being used to store "irrelevant" data.

Yeah ripper234, you are onto what I've been planning for a long time. I am planning an art registration business using the blockchain to verify authenticity of original paintings, prints, and other physical works of art.

What if I don't use the service?  Does the mere fact that the service is commandeering resources on my hard drive and bandwidth on my internet connection mean I should be happy and that I am benefiting somehow?

None of us really like downloading an ever-growing block chain.  But we tolerate it for now because we want Bitcoin to succeed and trust that solutions to cut the growth of that burden will be forthcoming.  Don't confuse that with a presumed willingness of Bitcoin users to donate their computing resources to anything anybody feels like throwing their way just because they feel they can.  Look to Wuala to crowdsource storage needs.  (they even accept BTC)

Every user of the service knows his on guid upon registration, and can monitor all records with this guid on a daily basis. How can an attacker build a false history for this guid? An attacker could append transactions to my account that weren't authorized by me, but this would be detected immediately (I get a notification every time an object with this guid is written). After the hack becomes known, the approximate attack date can usually be traced, and all transactions after this date are considered invalid. I would have a complete, authentic, verifiable record of all asset transactions up until the moment of attack.



Sure, the key might have been mismanaged, but we need a way to understand, post-mortem, which version of history is the correct one. Bitcoin is the most secure way to provide a time stamped record of authenticity / commitment. There is no other data structure that is safe from deletion, tampering, and capturing by governments.

I was suggesting storing the database on the bitcoin blockchain. No additional hash power is required.


I have never understood why people don't like it. This usage pays for itself. I haven't gotten into the details of implementing a PUT operation, but I assume there exists a method of doing this that is resistant to reasonable future changes in the Bitcoin protocol e.g. blockchain pruning. I don't know enough about the details of the existing methods of encoding messages in the blockchain to know whether they are resistant to such changes or not. For sake of discussion, let's assume there is such a method.

If such a method exists, then, nobody has to like or dislike it - it just works. Let the free market speak for itself, and price transactions accordingly. Bitcoin has always had to deal with transaction spam, this is just one form of it ... but any transaction that pays the fees is legitimate, even if it is intended as encoded data.

By definition, it's encrypted.  Plain FTP/HTTP would be more than plenty.  Their PGP key isn't secret, unless you mean their private key, which wouldn't need to be on an FTP server in the first place.  It could be on a Crypto-Stick for all we know, where it would be totally unstealable.


They would provide the signed GPG message where the service provider committed to the hash.  The block chain doesn't prove anything and is the wrong tool for the job.  The block chain proves that data X was known on or before time Y, but an attacker could just as easily be building his bogus data and putting it in the chain.  The right tool for this sort of job is message signing, something we had long before Bitcoin, and in fact something Bitcoin relies upon for its existence.


Unless all of those sockpuppets are able to sign messages with the service provider's signing key, it probably isn't going to work.  If they could, the mere existence of multiple signatures asserting the validity of differing history records for the same day would be evidence on its face that the key was mismanaged.

I know, but you would need hash power to keep the central database secure. "Confirmations" are backed by hash power, and not many users are going to leave bitcoin to hash your chain for no monetary benefit to themselves, leaving it wide open to attack resulting in blocks rolled back. Merged mining gives you the benefits of the Bitcoin network hash power with not much additional work needed to make it happen.

NB: My use of the word "central" above means universally accessible, not centralized. It would be decentralized in the same manner as Bitcoin.

I wouldn't like the idea of massive amount of unrelated information being dumped into the blockchain.  Now w/ merged mining and an alt-chain containing encrypted backups that might be interesting.

I wasn't suggesting an alternate coin.



You would still need an agreed, secure way to publish it, and later verify it. If for example they published it on their FTP server, an attacker could  steal their PGP key, and publish an alternate history of nightly backups.

Sure, some clients that actually bothered to download the backups before the attack took place would complain, but how would they prove their version is in fact the correct one? The point in my proposal is having a way to formally prove the balance without relying on anything except a small open source client program, and the blockchain.

Suppose the owners of the exchange are themselves the attackers - they could arrange for many dishonest users (real or sockpuppets) to claim that their version of history is the correct one.

How would this work fundamentally better, than, say, Bitcoinica publishing a nightly encrypted backup (a dump) that anybody can download (either via http or torrent), and also publishing a PGP signature to commit to its hash on a particular date?

This would be a whole lot simpler and less complicated and trivial for stakeholders to participate in the solution.

If there were a need to demonstrate that prior revisions aren't changing and that only thing happening is new records being added, the backups could simply be an archive file, where the backup of any night n contains identical copies of all the files that were present in the backup of night n-1, plus a new file representing that day's activity.

With the advent of merged mining, the blockchains of such parallel databases can be secured as well as Bitcoin itself, without needing another alt coin to accompany it.

https://en.bitcoin.it/wiki/Bitcoin-Powered_Database

I would imagine that the above was discussed here before, but I'm not sure ... if so, please point me towards that discussion (in fact, I vaguely recall posting about it myself in the past, but I'm not really sure).

As far I know, the Bitcoin blockchain is pretty much the only data structure that is both global and tamper-proof.

global - There is one global instance of the blockchain (up to forks ... which are then resolved).
tamper-proof - If a block enters the blockchain, after 6+ confirmations this block can't be feasible faked or altered.

A Bitcoin-powered database would be an API over the blockchain, that would expose a subset of the usual CRUD database operations. In fact, it would be an append-only data structure, because nothing can ever be really deleted from the blockchain - so only CREATE and READ operations will be implemented. Object Versioning will be used to emulate updates and deletions.

One major use case for this would an Undeletedable, Auditable Asset Ledger. It bothers me greatly that GLBSE, Mt. Gox, and other trading websites only use an internal database to record assets. As the recent Bitcoinica incident has shown, such records can be tampered with or even completely deleted. A BPD can serve as a verifiable copy of the asset records, that can't be deleted.