Topic: [ANN] Storj - Decentralized Storage - page 114. (Read 389811 times)

Nodes can advertise that they have physical redundancy, but the algorithms can not verify physical redundancy so it will not be included as a reward.

If you are running as a node I would recommend that you run them as single drives. Without doing any calculations (I'll run numbers at some point), I figure that the profit lost from using a drive for RAID redundancy will be more than profit lost from refilling a new drive with data(provided you have a fast enough connection). For slow connection I would indeed recommend a RAID setup for drive failures.

Failure is on the side of the hoster not the user. If your drives fail you will just loose out on a decent bit of profit, as you try to fill them back up. The end user is unaffected, their file redundancy will be automatically restored at the next check in.

about proof of resource, do you get more reward if you have the resources on a fail safe proof solution, instead of RAID0?

I'll have to read more on them in. We were planning to integrate Tahoe-LAFS at a point, but if they solved the duplication portion we can integrate them directly after MaidSafe.

I believe this problem is already solved in Tahoe-LAFS.

With the public file network over replication is actually a good thing. If there are 20 copies of a popular file, then you can connect to 20 nodes and download the file much faster than if you just had the standard 3(kinda like torrents). With the private files its a little bit more tricky, but I'll get into that at another time.

Overall some good points.

Hi Shawn,
I had an idea for a distributed storage system along these lines a few years ago, so I’m glad more people saw the potential and had similar ideas, and I’m excited to see it once it’s launched.

Along my research, I found a few problems and potential solutions, so thought I’d share (since I never actually started developing it).
I hope you find these useful in some manner.

First, torrent-like technology is the way to go. And I suggest you look into how the SOP (SOPcast) protocol works as well (used in Europe mostly for P2P streaming TV channels).

Availability and redundancy will be the next major things to address once you have communication in place. Ideally you’d configure distribution of the same data packets to several computers based on their distance, time of availability and frequency of access.
My thoughts were along modeling redundancy after RAID 5 (abstracted over the internet of course).
Some people keep their computers running non-stop, but not everyone, however, if you can group people with the same patterns of activity (which I’m sure you can figure out), you can achieve much higher redundancy with fewer copies of the packets. This grouping based on availability will also allow you to store only a couple copies on always-available machines, and the gross on the machines in the group. You can then store only a handful of packets from always-on machines on the less-available machines.

Next comes speed. Computers close to each other usually communicate faster, so (at least frequently accessed) packets are best stored within proximity, or at least prioritized based on distance when retrieved. A local cache of frequently accessed files would also be advisable.

To solve the problem of availability long-term (once the technology is proven and mature), I recommend you reach out to hardware vendors, lan-cards in particular, to incorporate an SSD buffer chip, and open an API similar to WOL to allow them to still communicate even when the machine is off. I’m sure you/they can solve any legal/privacy/etc concerns.

The vast number of users don’t use their HDD anywhere close to capacity, so I’m sure they wouldn’t mind sharing their hdd for only a portion of protected storage in return. Best-case scenario would be share 90GB and you get 30GB of protected redundant storage in return (realistic number would probably be more distant).

I also suggest you look into potential commercial applications, since companies usually have the majority of their workstations on, and with very large drives that go unused, while cramming everyone’s files on some sort of central storage (SAN, directly attached, network shares, etc). I’m sure at least a few would welcome a test-run.

The problem I haven’t been able to solve is privacy vs deduplication. You’ll likely end up with thousands or even millions of copies of the exact same file, so it would be efficient in terms of both speed and resource management to reduce the number of copies, and use pointers to files (or chunks in this case).

I hope some of this helps, and good luck!

You need to do some more reading abut sidechains.

Souds exciting! I haven't read much about sidechains but can sidechains be created from Datacoin?

Great article posted about us today:
http://www.infoworld.com/t/encryption/bitcoin-finally-pays-in-secure-cloud-storage-240386

To be clear:
We are releasing our own cryptocurrency with Storj, but only as a fast-to-release prototype to gather data about the network's behaviors. Eventually we will switch to a sidechain [a blockchain that runs in parallel with an existing cryptocurrency such as bitcoin and allows its use], when that technology is ready.

Really interested to see how this project turns out.  I have a few spare TB in my SAN that i'll be throwing online

The 2nd one. A blockchain is very bad at storing data. It is, however, very good at storing where a file is and its state. It is after all a ledger. We just choose to use the ledger for file locations as well rather than just coin balances.

Interesting. Might have to do some load testing with that, and see what happens. We should not expect full blocks under normal usage.

Sign up for the newletter. We will be posted more details, and some demos soon. Unless you are volunteering directly or working on the code, don't expect anything live till the end of the month. We will be mostly likely doing an invite only beta. We will open to it up to more people, as we fix bugs and the software is more stable.

Yes, a modified Datacoin will allow us to launch out prototype software. It will give an experimental network and make our software immediately useful. We can add features, and replace any part of the system at any part in time. In that way we can avoid the whole 2 weeks^tm thing. Our development process will always be "Prototype. Deploy. Fix. Deploy. Refactor. Deploy. Repeat."

Is the data stored in the blockchain ? Or is the blockchain only used to maintain/verify the integrity of the data and provide proof of resource ?

While playing with Datacoin (which you mentioned), I have found some nasty side effects on block times when all the transaction data space in a block is full, could be due to network latency, but I really did not put much effort into testing since I don't think a blockchain can handle a big amount of data.

When will be able to see something other than a screenshot ? Is the code already public ?

Good luck!

EDIT:
I have taken a (quick) look at the whitepaper. So at launch it will essentially be Datacoin, without any new features. New features will be added later, if I understood well. I guess the block data found lately in Datacoin where your own tests

The MaidSafe approach is the most interesting, although I don't quite see the need for a currency there, other than that it would have also been interesting to see an integration with Tahoe-LAFS for the data storage, IMHO it provides the best distributed technology to do that with only limited trust required.

I was also confused.

A wild Reddit appears:
http://www.reddit.com/r/Bitcoin/comments/22qivd/guys_here_is_storj_it_is_launching_soon_and_is/

Storj could be useful as a backend for a web site. For example, if wanted to build something like Imgur, you could run the entire website off a $5 20 GB web server. Storj could store and server millions of images without you actually having to store than information on your servers.

I could see some changes that could be made to Storj web node software that you could run a website off the Storj network. Honestly I think that could be a project in itself. Decentralized and autonomous web hosting with Storj as a backend. Once we get a little further in the project remind me again, and I'll see if we can put together a prototype for just that.

Can StorJ be used for webhosting? So it can store files and transfer - but can it run programs while its being stored? Such programs is Apache, PHP, MariaDB etc.

2. Or if not, can a server run these programs, and call the source codes/database where it stores on StorJ?

Thank you.

Well Dropbox, Google Drive, Box, seem to be doing it all quite fine. We aim to replace that with something cheaper, more secure, faster, and more scaleable. Of course, every technology will be replaced with time. You must understand that is not the core of the business model or value added.  Your models limits the ability of Storj to reach all markets, and be reliant on a singular platform/token. That is not true decentralization.

Since I'm getting some question about Ethereum integration comments I'll address that. Ethereum allows "Bitcoin 2.0" like contracts. This is inherently useful for many reasons. Here are my two immediate ones:
1) Cross Node Contracts - Node A is overloaded, and needs to offload bandwidth and storage space. It can autonomously and automatically enter into contracts with others nodes to help it out for a profit.
2) Compute - Our goal for Storj is always to have 100% utilization of resources. Our nodes will mostly be I/O bound, and the CPU won't be put to task initially. Would like to use the Ethereum Contracts  + Structured Metadata on the Storjcoin chain to allow for a rudimentary decentralized cloud computing service to use that CPU. Works something like this.

User stores Data 1 on Node A
Node A Replicates Data 1 on Node B and Node C (part of normal redundancy)
User Initiates a Compute Contract
Node A-C, perform computations on encrypted Data 1
Node A-C, returns the hash of the data result to the contract
Contract can achieve consensus that the resulting data is correct, and pay Nodes A-C for their work
User receives result for Data 1

Again this is a rudimentary system for Storj 2.0, but Ethereum will allows us to prototype out something useful that we can learn from.

there are two main reasons why MaidSafe will be able to offer free storage:

1) Data deduplication - since the vast majority of the data that is going to be stored on the network is going to be the same data (movies, music, games, books etc) it wont be necessary to store them more then n number of times (n depends on the amount of activity that data sees in the network)

2) Safecoin value - Safecoin's value is going to rise because of it's properties (anonymous, fast confirmation) and because it will be embedded as a payment mechanism in the network itself (escrow, timestamp and whatever else ends up being developed). Since Safecoin's value is going to rise the farming of the storage will be more valuable because you get the Safecoins and therefore the price of storage is going to be zero for as long as Safecoins are worth buying HDD and farming

To try to really hammer the point home for anyone that doesn't follow, imagine this - what if Bitcoin instead of using CPU's to solve complex math problems for mining, uses HDD to store data for mining(farming). Well that's Safecoin.

So instead of meaningless work with mining as a proof of work you get storage as a proof of resource.


Look if you think you can sell this and make money doing this then by all means go for it. But I think you are in for a uphill battle that doesn't make any sense on many levels. And also why build it if you know that you have a year at most before storage as a service is commoditized?

P.S. this is all hypothetical and I have no proof that MaidSafe will work as advertised. I sure hope so, but that remains to be seen

I guess I don't understand why you have a limited amount of vaults this a finate amount of space, and have it be able to store unlimited data for free.

Even if it is "free" then I can still turn around and sell it. If someone gave me free and unlimited Bitcoins I'd turn around and sell it on Bitstamp.

I can cut Dropbox prices in half and offer a similar service. Using the best server I can find for $30/month it can generate a monthly profit of $13,696. Sure that won't last as the market corrects, but perhaps will last longer than you think. Dropbox has gotten away with charging 100x for the past few years reselling Amazon S3.

There is nothing stopping us from also developing nodes specifically for Maidsafe with the exact same software, but Storj can continue to make a profit regardless. Gotta diversify.

It most certainly has cookies. I'm kinda insulted you asked.

You keep assuming that payment of storage is going to be in Safecoins but I have to reiterate just one more time that storage will be FREE on the SAFE network. This will significantly reduce friction for user acquisition for MaidSafe and make things a lot harder for you to pay your nodes and charge the users. Free market rulz --> C.R.E.A.M

My suggestion to you is to work your ass of on hunting down devs to use your platform to add value because user acquisition is going to be hard without a superior product.

Well I've done my job here. Now back to being depressed after listening to Adam Backs Bitcoin sidechain project which has the potential to singlehandedly destroy all alt-chains (except MaidSafe of course  )

I guess my investing days are going to be rather short. Going to be interesting to see how this is going to pan out...