It is necessary to quote this longish post from another thread to bring the context into this thread.
Note that any solutions to the problem of ISPs blocking P2P apps that involve a TURN (when STUN tunneling fails or is blocked), VPN, or other server in the middle, defeat the entire point of extracting the value of the bandwidth allocation of users provided by their ISPs, because then one is paying for the bandwidth of the server to relay the shards.
If Storj and MaidSafe max out the consumption of each user's upload bandwidth (thus leeching off users with higher allocations charging the costs to those users' ISPs), they will also be blocked by ISPs. Additionally STUN tunnelling often fails and thus a TURN relay server has to be employed (or using the other peers as relays thus leeching the upload bandwidth of those ISPs who don't block tunneling).
In short, P2P for bandwidth consumption between ISP hosted user accounts is not going to be reliable. Many users will have frustrations when trying to be a storage provider. It will not be the case that every user in the system can also be a storage provider. And it will probably end up being the case that the most efficient storage providers will be hosted on dedicated servers.
In other words, it is a fantasy to think we can get decentralized file storage without paying for it.
We can try to design decentralized, permissionless file systems that correctly incentivize the storage and bandwidth providers, and the users of the system need to pay for it somehow. Whether or not these can remain permissionless given the need to host these on servers is open to further contemplation and study. Most all hosting providers include in their Terms of Sevice a restriction on hosting illegal copyrighted content, so unless one can provide a mechanism for which illegal content is removed from the system, it seems to me that hosts will be forced to ban the protocol (system).
So where I am headed with this line of thinking is that we ought to just give up on illegal content and illegal uses of anonymity. It isn't going to work. It is a fantasy.
Continuing my analysis, the other advantage of decentralized storage is durability and availability. This is a facet of permissionless in the sense that no one entity has a monopoly on the storage. It is not permissionless in the sense of allowing illegal activity as explained upthread (because the storage will hosted on servers, even those are owned/managed by different entities, they all are regulated by the law reflected in the hosts' Terms of Service).
So I am envisioning the possibility to design a system for decentralized file storage where the users pay the storage providers, but the storage providers are decentralized entities (even though they are all high performance hosted providers and not ISP user clients).
In this case, I think microtransactions is the only way it can be done decentralized. If we instead attempt to aggregate a monthly use plan (or similarly analogous aggregations), then some centralized party will be in charge of paying the decentralized entities, so then it is not decentralized.
So therefor I have just identified a potential market for microtransactions that can't be offered by centrally owned cloud services.
Alternatively, Storj and Maidsafe are paying storage providers coins for proving they are storing data, then data is exchanged in a tit-for-tat[1]. If used with ISP user clients as storage providers, this will have performance weaknesses as well as being economically a theft paradigm in support of Net Neutrality oligarchy and taxation (for the reasons I explained upthread). But a user can't do a tit-for-tax exchange if user is not also a storage provider, thus afaics Storj and Maidsafe are forcing every user to be a storage provider. Otherwise they need to use some form of upload bandwidth theft model such akin to Bittorrent's optimistic unchoking. The only way to fix Storj and Maidsafe is for them to adopt a microtransaction payment model so users can pay for the upload and storage costs to decentralized providers.
So therefor I have explained why Storj and Maidsafe are fundamentally flawed. And I have explained why decentralized file storage can ONLY be done with microtransactions.
Next we need to reason about the viability of the markets for decentralized file storage and also the technical viability/tradeoffs. We need to not only think about ability to prove the data has been retained by some provider, but also about how to enforce against the storage of illegal content (otherwise I have argued upthread that the entire plan is flawed since hosts' Terms of Service will likely block protocols/systems which can allow copyrighted material to be stored without recourse by injured parties).
[1] Note
Storj also alludes to microtransactions, so perhaps the tit-for-tat exchange only applies to Maidsafe. I will study this more.
Page 14 of the Storj white paper makes it clear they are relying on a propagation full node security model, which we have entirely discredited in public forum discussions I have had with monsterer and smooth over the past month or so. Storj, Maidsafe, and
Sia are incompletely specified systems, and
frankly Storj, Maidsafe, Sia, and Permacoin look like fantasies trying to sell speculators a bridge to no where.
Before exploring the potential markets for decentralized file storage, I would first like to analyze more deeply the technological viability.
AvailabilityFirst if we are not employing ISP user clients as storage providers, thus not stealing upload bandwidth from ISP user accounts (as I had explained in the prior thread where the quote above originates), and thus not able to use the unused storage space of user computers (i.e. we are using then decentralized/duplicated hosted server storage providers), then any redundancy of data storage will incur the increased cost of duplicated storage. However, note that duplicated storage would be required for a centralized storage provider as well (since it would require redundancy for durability of data) and with decentralized copies of data, each provider need not build in internal redundancy as it can require the data (after loss, e.g. due to storage drive failure) from the other providers in the decentralized system. Also with bandwidth costing roughly $1 - $2 per TB (e.g. hivelocity.net) and data center HDDs roughly $0.03 per GB ($30 per TB), a file accessed more than say 100 times is going to be dominated by bandwidth (and CPU) costs. Btw, this is why Google has created a
pseudo-offline storage class for infrequently accessed files which reduces costs to
$0.01 per GB probably by using
recently invented higher latency drive technologies.
DurabilityThe goal is to insure a copy of the data is always stored and thus the data is never lost. A centralized storage provider maintains redundancy of storage to provide very low probability of loss, and might even back up files in cold (tape or optical disk) storage.
Storj, Maidsafe, Sia, and Permacoin are proposing to use Proof-of-storage to insure there are multiple copies of data stored, but I explained already that this is fundamentally flawed and can't be sound:
Warning I as AnonyMint proposed this in mid-2013 and referred to it as Proof-of-Storage. I also discovered it was fundementally flawed. If you continue, you will be wasting your time. Eventually I will come back and explain to you and Andrew Miller PhD why this is flawed.
Details are definitely needed here. I know about Proofs-of-Space and Proofs of Space-Time(
http://eprint.iacr.org/2016/035.pdf ), Miller's Proof-of-Retrievability(Permacoin), and White's Proof-of-Storage in Qeditas. What's Anonymint's proposal about?
We found some possible drawbacks and attack vectors in Permacoin, but no fundamental flaws.
In my original analysis in 2013, I went down the same rat hole of flaws as in section "4.2 Local-POR Lottery" of the white paper. They assume so many things (including for example that Amazon bandwidth is 10 - 100X more expensive than dedicated host), and when you work through all the analysis, then the scheme does not work to prevent centralization of the mining, and thus the permissionless (uncensored) and robustness/durability of file storage attribute will not be sustained.
P.S. I also read these:
https://bitcointalksearch.org/topic/proof-of-storage-186601 (this was an offshoot I think of my idea which was in
my March 2013 thread)
https://bitcointalksearch.org/topic/m.6536798For example,
the storage nodes (providers) can be Sybil attacked such that all the attackers nodes share the same data store. Certainly this will be the outcome since the attacker can earn more coins for each Sybil node added to the system. IP addresses aren't that expensive. This is yet another reason I think
Storj, Maidsafe, Sia, and Permacoin are bullshit and probably scams.
So now that I've explained why paying each node (for the copy of the data it holds) is not immune to Sybil attacks and thus not economically viable, I assert that the only ways to insure durability in the decentralized provider context are either of:
- keep a backup copy in a trusted centralized provider (this can be encrypted if not providing public access so that the centralized provider can't censor based on data content)
- trust a statistical incentive that at least one decentralized provider will be incentivized to store your data
In the first case, the decentralized copies serve the high availability function and the backup copy the durability guarantee.
The second case would be perhaps the incentive providers have to retain the file if it is frequently accessed (assuming they are being paid each time they serve a file) and then setting up a cron job to frequently request and pay for a serving of your files when they have not been otherwise served to the public interim. In this case the decentralized copies serve both the high availability function and the backup copy the durability guarantee.
Depending on the market use case, one of the other might be more economic.
LegalityWith hosted content, the host provider holds the person on the account responsible for violating the Terms of Service which includes hosting illegal content. Hosts will likely ban a person for habitually violating the Terms of Service and for not having a streamlined policy for illegal content complaints and legal actions.
In the decentralized providers context there is suppose to be no centralized party to blame or send a legal action to. Storage providers can't know who to believe about whether a file must be removed. The legal authorities can't go serve action on all the decentralized providers, because the storage providers come and go (and are too numerous and distributed). So the only action that can be taken is to require the hosting providers do not allow the system to be run from their servers. Thus the decentralized system becomes essentially banned every where.
The only solution I can think of is to require each file submitted to the system to be signed by public key authorized by a trusted authority that verifies the identity of each signature authority it authorizes. Then the authorities can serve the legal action directly to the trusted authority to revoke the public keys (and all decentralized storage providers comply by removing those files).
So in conclusion, decentralized file systems could potentially provide some higher availability and scaling, but durability is more problemmatic (but probably can be acceptable) and authority over which files can be stored will remain centralized.
The higher availability and scaling is interesting. Not sure if it is compelling though. Users will still have to pay file storage, there is no way to reduce the costs.
And since they don't offer any centralized mechanism for legal actions,
Storj, MaidSafe, Sia, and Permacoin will end up entirely illegal or run only from ISP user clients in which case they will be slow (low availability and/or high latency), blocked by many ISPs thus not reliable for users, in addition to the other fatal flaw I mentioned above about Sybil attacks.
Storj, Maidsafe, Sia, and Permacoin (and any other decentralized file system that pays to store a file instead of only paying to serve the file) are all
provably scams.
Include
Ethereum in the list of scams (although they are still trying to fix the technology) and probably most every other shitcoin on this forum.
Here is the information about the flaw I had found in Bittorrent in 2008:
I think social media can possibly be taken over by cryptocurrency/decentralized/blockchain technology. Think about it... Facebook has a market capitalization of 266.3 billion. What if a portion of their net profit was distributed to its users instead? Which service would you use... one that makes money off of you providing you nothing in return, or one that pays you to use its service? There are likely a few projects attempting to capitalize on this space. The only one off the top of my head I can name is Synereo and I am on the fence as to whether it is is a legit project or a P&D... I am waiting on the sidelines for now.
http://www.synereo.com/One of the foundational technical challenges is decentralized, permissionless file storage (and databases); otherwise if a corporation is providing centralized file storage then they control the content and can monopolize.
Afaik, the current attempts such as Storj and Maidsafe have a fundamental economic flaw. That is they are selling for free that which is not free— the bandwidth (and most saliently the asymmetrically more expensive upload bandwidth) of the ISPs. I had warned Bittorrent about this flaw in their economic algorithm and had suggested a fix in 2008:
Did Bittorrent become popular without MSM coverage?
I'm not really sure.
Yes, it did. The Bittorrent whitepaper was a breakthrough in p2p not matched until Satoshi came along.
All the cruft of Gnutella (anti-leech arms race kludges, supernodes, etc) was swept away by Bram's brilliantly elegant tit-for-tat algorithm.
Well someone did come along before Satoshi in 2008 and that was me (Shelby), but I was apparently ignored. I basically predicted the Net Neutrality shit we have now and was trying to improve Bram's concept:
https://web.archive.org/web/20130401040049/http://forum.bittorrent.org/viewtopic.php?pid=178#p178Did Bittorrent implement my proposal? I never followed up (my life went on a tangent).
You can detect some more coherence in my writing back then because that was before I became so ill. I am amazed in hindsight that I understood the concepts of Bittorrent so well having absolutely no experience whatsoever as a developer in P2P.
Now what is really fucking amazing is that the link quoted above worked correctly a few days ago when I issued it. Apparently someone in the Monero thread communicated to Bittorrent folks and had the entire archive of the Bittorrent forum removed from the archive.org. I am not joking and I am not hallucinating.
What I had written there in 2008 (which luckily I reread a few days ago so my memory is refreshed) was I explained to the Bittorrent developers that their tit-for-tat algorithm was orthogonal to their
optimistic unchoking algorithm, and that they could improve the tit-for-tat algorithm by have the two peers that exchange a shard of data to encrypt those shards. Then after the shards had been received by both peers, the decryption keys could be exchanged. The economic benefit is that the bandwidth has already been exchanged before each peer can use the data. Thus neither peer has any bandwidth cost reason to cheat. The reason this was important is because typically download bandwidth is much greater than upload bandwidth, so by forcing all peers to trade equally, it would mean that peers could only download as much as they could upload. Bittorrent didn't like this suggestion because they preferred to leech the upload bandwidth of those who have higher allocations with their ISPs thus forcing those ISPs to pay for the upload bandwidth that the other peers at the ISPs with lower upload bandwidth allocations do not incur.
I warned Bittorrent that without my suggested fix, then the ISPs would end up blocking and rate limiting Bittorrent, which is exactly what has happened as I predicted:
http://www.pcworld.com/article/145786/isp.htmlhttp://guides.wmlcloud.com/windows/how-to-bypass-torrent-connection-blocking-by-your-isp.aspxhttps://www.quora.com/My-ISP-has-blocked-all-the-P2P-downloads-Is-there-any-way-I-can-bypass-themNote that any solutions to the problem of ISPs blocking P2P apps that involve a TURN (when STUN tunneling fails or is blocked), VPN, or other server in the middle, defeat the entire point of extracting the value of the bandwidth allocation of users provided by their ISPs, because then one is paying for the bandwidth of the server to relay the shards.
If Storj and MaidSafe max out the consumption of each user's upload bandwidth (thus leeching off users with higher allocations charging the costs to those users' ISPs), they will also be blocked by ISPs. Additionally STUN tunnelling often fails and thus a TURN relay server has to be employed (or using the other peers as relays thus leeching the upload bandwidth of those ISPs who don't block tunneling).
In short, P2P for bandwidth consumption between ISP hosted user accounts is not going to be reliable. Many users will have frustrations when trying to be a storage provider. It will not be the case that every user in the system can also be a storage provider. And it will probably end up being the case that the most efficient storage providers will be hosted on dedicated servers.
In other words, it is a fantasy to think we can get decentralized file storage without paying for it.
We can try to design decentralized, permissionless file systems that correctly incentivize the storage and bandwidth providers, and the users of the system need to pay for it somehow. Whether or not these can remain permissionless given the need to host these on servers is open to further contemplation and study. Most all hosting providers include in their Terms of Sevice a restriction on hosting illegal copyrighted content, so unless one can provide a mechanism for which illegal content is removed from the system, it seems to me that hosts will be forced to ban the protocol (system).
So where I am headed with this line of thinking is that we ought to just give up on illegal content and illegal uses of anonymity. It isn't going to work. It is a fantasy.