Topic: [IDEA] Dirt cheap online storage - page 7. (Read 14494 times)

(...You didn't say how many separately-sited copies of your data dropbox's offer is/contains/represents...)

Okay, I have spent the night digging back into the details of various actually currently working software that I could run, but as you did not indicate what operating system you plan to use to provide the storage you are at least hypothetically offering I do not know if you could run them.

I am finding so far only two systems I actually have managed to get installed and working that seem like one or more of them of them might suffice for an offer such as is quoted above.

That is, putting aside all the theory and vapourware-wishlists and philosophy and looking at that specific offer as a potentially-tangible offer.

Those two systems are GNUnet, which currently requires a unix-type operating system to run, and Tahoe-LAFS, which is python and seems to think it is usable even on Windows.

Part of the differences between them is that GNUnet is not only about storage, so it has the "overhead" of trying to provide some anonymity. Requests sent out do not have to originate at the node your node receives them from, and nodes deliberately keep packets moving as background noise all the time so that timing attacks cannot notice aha look that node talked to that one, without having heard from another one recently, so I bet it originated that request itself.

So if you do not want anonymous routing going on,  Tahoe-LAFS seems cheaper in resources to use as provider, provided your customers don't want the kind of anonymity GNUnet adds into the service-offering.

However, Tahoe-LAFS is organised as "grids", each "grid" being basically the collection of nodes a particular "introducer node" is able to introduce clients to. So in order to offer a Tahoe-LAFS based service you'd either have to have a few machines, probably best not all being in one site; or a number of us would have to team up to make one or more "grids" we would between however many of us be able to populate with enough machines, at enough sites, to be able to present the services of our "grid" as a credible service-offering.

I suggest that if you wish to sell your $5/month offering we should maybe consider teaming up with a third person so we can have at least a three-sites grid running, and consider whether to each use the price shown in the quote, totalling $15/month for someone wanting all three of us to store that much data for them so they have three copies out on the grid; or consider charging slightly less per redundant copy located at separated locations.

Did you happen to notice how many separate sites how many copies of your data the $20/month commercial offer was offering?

Despite the "overhead" of the (possibly extraneous to many people) anonymity that GNUnet provides, I think we would be better advised to go with a GNUnet solution, partly because using Tahoe-LAFS we would be trying to compete very directly against commercial services.

I am thus thinking lets start small, using GNUnet, then as we get customers for that "value (of being anonymous) added" service, revisit the price per gigabyte per separately-sited copy we could offer and whether it would be worth our while at the rates trying to compete with the commercial services would force us to drop our prices to.

WIth GNUnet we not only get to have this "value adding" feature of the anonymity stuff, but also the "it runs itself" feature of quite possibly providing quite a lot of its participants enough offsite storage and anonymous packet routing just in node to node barter that we could all have a useful number of usefully stable offsite backups without having to sit down and figure out how much bitcoin to send each other at all. The system has the virtue that we can all have these things for ourselves by barter up to the point where someone wants to receive more service than they provide. Those people will become thereby the potential "customers" who might go shopping around among those of us who have stably, long term, reliably had service to spare that we can offer them.

Basically it should find out for us automatically who among us actually does have surplus available and who among us is suffering sufficient shortfall of provision to start thinking about spending some bitcoins having in a sense run out of actual service of their own to barter with.

With Tahoe-LAFS, not only are folks not anonymous but also we each have to jump through gosh knows what hoops to figure out how much of our resources which other person is consuming more of than we are consuming of their resources. The administrative overhead thus looks like it would be much higher  not onlly for those actually providing service but also for those wanting to leech or buy service (the potential "customers" among the participants).

-MarkM-

Lets translate that question a litttle to make it clearer what is being asked:

"How would tracking [quantity of product offered for sale] be any different from tracking [quantity of currency offered in return]? They're both just numbers..."

-MarkM-

There's no reason to restrict the market-making function to a single entity. While the network is small you could even use #bitcoin-otc to match up buyers and sellers. That gives you an open order book and a reputation system to work with. When the network grows more exchanges could appear and people (or their software agents) could choose the best ones to use.

Creating a market mechanism is really as simple as automating the steps that you'd use to buy and sell manually. If you were looking to rent hard drive space you'd go to a place like #bitcoin-otc, look at the offers that were available and the reputation of the sellers and choose one or more of them to form a contract with. After the contract ended you'd leave either positive or negative feedback as appropriate and expect the seller to do the same for you. If either party provided inaccurate feedback the other one would attempt to prove his innocence in order to protect his reputation. People with higher reputation scores could charge more for their services than those with low or nonexistent scores.

Automating this is a matter of encapsulating in software a set of rules for the buyer and a different set of rules for the seller.

So you are seriously suggesting DVD's as a potential competitor.  Who wants to wait for 200 DVD's to burn?  On top of that, who actually trusts that, out of 200 DVD's, none of them will have an error when you try to restore from them?  On top of that, there's the convenience issue of not being able to access your data immediately if you need it.  On top of that, what "offsite storage location" is going to offer you safe storage for zero cost?  A family member's house?  What if they lose them?

I agree that the pricing model is complicated.  It seems that at least three metrics would need to be calculated and available.  Storage space, bandwidth, and speed.  And determining a pricing matrix between the three would be quite arbitrary and, well, confusing.  So perhaps, reducing it to one metric (storage space), and imposing certain minimum standards on the other two would be appropriate.  A host much be able to provide download speeds of at least 100 KB/s, and must be able to provide bandwidth equal to or greater than the storage space rented on a per-month basis.  Anything above and beyond that would be a bonus.

Or, looking at it from a different perspective, perhaps a central agency (I know, naughty words here) could match up buyers and sellers as appropriate.  The agency would know who has what storage, bandwidth, and speed capabilities.  For each buyer, they might choose a host with lots of speed, a host with lots of storage, and a host with lots of bandwidth, then serve from each of the three depending on how often the person needed the data.  If they end up needing the data often, work with the host with lots of bandwidth.  If they end up needing the data infrequently, let them download at higher speeds.  Etc, etc.  Yes, it would be complicated, but maybe less complicated than trying to figure out a decentralized way of matching up buyers and sellers.

Certainly, the simplest method of conducting such a project is to have a page where people host files to be downloaded, with a bounty in the future if they need it.  This would mean the host with the most space would have the largest chance at being able to provide a file that is needed, and would also mean the host with the most speed would have the biggest chance at providing the file more quickly than anyone else.  Yes, it is uncertain for sellers of space, but it may only take a handful of recoveries to make up for it entirely.  I actually think this would be the most practical route to go, and it's an interesting idea that hasn't been done before (AFAIK).

I still don't see why you want to move away from Bitcoins though.  How would tracking a karma score be any different from tracking Bitcoins themselves?  They're both just numbers...

Don't get confused by the variable-names or metric-labels:

"Pay bitcoins if you need [RESOURCE_OR_SERVICE], receive bitcoins if you rent it out."

If what you need is, purely and solely, storage space, have you compared the monthly cost per terrabyte of writeable DVDs plus the oneshot upfront cost of installing enough DVD drives to put that much DVD space online all at once at your own location?

If what you need is, purely and simply, offsiteness of space, have you compared the monthly cost per terrabyte of sneakernetting or snail;mailing the DVDs to an offsite location?

If what you need is purely and simply, transport from onsite to offsite, again compare DVD-by-sneakernet.

If what you need is, purely and simply transport from offsite to onsite, again have you compared price of DVD-media transport?

It continues to sound though as if you are actually trying to palm of or fob off some dynamic combination of probably at least three of these pure and simply commodities onto a one-dimensional metric (known as bitcoin) that already carries the burden of also having to be equal to dynamic quantities of an endless number of other pure and simple things and complicatedly dynamic things and indeed any thing or service whatsoever.

If you really want to take it down to how many bits you receive from offsite, how many bits you send offsite, how many copies of each bit reside on each reliability of peer storage reachable at each possible bits per second rate at what percentage chance on any given second, minute, hour, day, week month or year, that is vastly complicated set of markets, where you or your agent needs to determine how many bits per second seconds how many bitmonths of how likely to be accessible when you need it storage is equal in value to you to how much more or less of any of those factors. How much do you value the snappiness of response, as measured in number of bitmonths stored? Does it differ according to how many bitcoins the material requested has been stored? And on and on and on. Is two bytes of storage worth more or less to you than two bytes per timespan for two timespans of transport? Is outgoing transport worth more or less to you than incoming trasport? Etc etc endlessly.

Once all these factors as to your personal preferences as to when you want how many more bits per second rather than how many bitmonths of remote storage in use and so on and so on have been all reduced down to a single scalar numeric value, then how many satoshis or bitcoins are  you willing to pay per how large a value of that scalar metric can be tested by offering you such a scalar on a market.

If however you insist that the scalar itself be labelled bitcoins, then it is reduced to a market in which the commodity being auctioned off is known as bitcoins due to your insistence that the scalar result of the reduction of the commodity to a single scalar must be bitcoins rather than being number of bits of data you have on offsite DVDs available by sneakernet using couriers, or speed of vehicle said couriers will use up to and including putting the DVD into a transporter so it materialises on your desk or directly changes the magenetism of your hard-drive.

It is too vast a field of not-the-same-thing that you are trying to boil down into one thing. Having boiled it down to one thing, whatever one thing you boil it down to so as to have a BTC<->ThatThing market, if you insist that thing ITSELF must ITSELF be known as bitcoins rather than being known as "general scalar metric of offsite-storage-network resource consumption quota" you will just end up with a totally confusing market, a "BTC meaning the digital currency"<->"Bitcoins meaning the metric of usage of network storage resources subsuming upload volume, download volume, storage volume and duration, bits per second sustained of download, bits per second burst of download, bits per second sustained of upload, bits per second burst of upload, and other ancillary factors".

It seems simpler and clearer to use some word other than "BTC" or "Bitcoin" or "Bitcoins" for that second metric, the commodity metric being offered for sale not AS bitcoins but IN RETURN FOR bitcoins.

In GNUnet, if I recall correctly, that commodity-provided-by-this-system metric is stored in a variable some coder or other happened to name "karma".

It is simply a human-and-compiler-readable label meaning "that scalar value which serves as the numeric metric of the commodity provided by this network/application".

Hindu or any other mysticism might only be some strange random reference that coder happened to find useful at coding time as a mnemnonic by which to refer to that variable/metric.

So for your purposes maybe it would suffice to use a preprocessor macro to change the name of that variable to QUANTITY_OF_SERVICE_DESIRED_BY_USER so that you will be able, upon reading the code, to make the mental leap to the aha moment that, in such a network, THAT is the variable you would be best served by spending your bitcoins to buy an increase in numeric value of the bits that variable refers to as interpreted as representing a number.

Or, you could chose to change its name to AMOUNT_OF_STORAGE_DESIRED everywhere instead, while people also interested in how much speed it amounts to rather than how many years bits offsite will survive if they don't request them for years could use some other label for it.

It is a complicated service involving redundancy and latency and bandwidth and on and on and on. Marketers would love to game you by offering a different product, different in any of the details that go into the whole generic concept of getting your bits offsite and being able to get them back onsite, whether by dialup or sneakernet or wireless or TCP or UDP or whatever, so they can claim to be a different product entirely, maybe not even a commodity (generic) product even maybe but rather a prestige boutique product incomparable to commodity offerings so they belong on a different market entirely and should not be compared to commodity offerings, especially not on price, since afterall their vast marketing budgets and ridiculously ignorant user-bases' need for hand-holding's accomodation by vast helpdesk centres precludes any comparison to any raw commodity.

To be a commodity that can be assigned a scalar value in bitcoins,  a thing has to be gotten down as close to a scalar value as possible first in order to have a single value, usually labelled quantity (as in X number of porkbellies, X number of grams of gold, X number of remote storage bitspeedmonthdirections or whatever), so that a market can be a "quantity of money" <-> "quantity of that very very specific commodity" market.

Notice even the grams of gold might be this that or the other fine-ness of gold, so even there there are more variables that in casual chat one takes for granted but that in fact a .999-fine grams of gold market is distinct from a .333-fine grams of gold market and so on.

To commoditise a thing you need to get it condensed down to the point where a market offering a scalar value of it per scalar value of a currency makes sense to the participants.

Already in Open Transactions we are seeing how fragmentation of markets scares off potential customers, with some potential customers thinking that having separate markets for ones of bitcoins, tens of bitcoins, hundreds of bitcoins (or s/bitcoins/porkbellies/ or s/bitcoins/shiploads of porkbellies/ or s/bitcoins/grams of porkfat/ or whatever, an assset is an asset is an asset, all are tradeable each against each) is in some way bad or awkward or "too complicated" or something even though in business it is actually very often the case that the price of grams of something, per gram, differs from the price of tons of that same thing, per gram.

So dont keep getting hung up on the name of the variable that serves in the code as the measure / metric of what it is that is being provided, instead consider whether that which is provided suits your needs and whether you wish to be provided with more of it or less of it than you are currently being provided. If you want more of "the service", get that number increased, whether by paying bitcoins to have it increased or by whatever other means. It is the measure aka metric of the service provided by that software.

-MarkM-

Because ... if each server is allowed to hand out the exact same bytes, the client can not tell 2 IPs pointing to the same disk from 2 distinct servers opening the door for said scammer that claims to do all the redundancy for you while he only has one server but 12000 accounts.

In the case of AWS, it is costly because it is so extremely redundant to absolutely paranoid levels. You can save 30% by choosing Reduced Redundancy Storage, but even then they store you data across several servers.

For Standard (paranoid) redundancy, they say "Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year. Designed to sustain the concurrent loss of data in two facilities."
And for Reduced Redundancy, they say "Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects. Designed to sustain the loss of data in a single facility."

So for the first option, you could literally wipe out two whole datacenters with missiles or whatever, and they would still have a backup copy.

I don't understand why any sort of concept of karma is necessary if people just use Bitcoins for it.  Pay Bitcoins if you need space, receive Bitcoins if you rent it out.

Why couldn't the client encrypt the data before sending it?  Why does each server need to encrypt it?

If cloud storage is superior, then why is it so expensive?  $125/TB/month?

In my opinion, this doesn't solve a real problem that anyone has in a way that could be even remotely useful.

Cloud storage will always be superior and the kind of problems people want to solve with this P2P storage idea could be solved much better and more cheaply by extending the basic concept of cloud storage.

This idea would create more problems than it would solve.

Not sure if this was addressed before but if I get payed per MB I host, each such server would need a different key to encrypt the data as else I could register 12000 servers with one actual server, mimic to provide redundancy but serve all from the same disc. Those that pay for 12 copies would end up with 11 copies on my "cluster" paying me 11x the money.

If karma works right, which it is intended to and thus hopefully in some version of the software will do, and might in fact already do, then it should condense nicely all that complication down to a simple do I need more karma with more other nodes or am I satisfied with the level of performance my existing karma provides.

-MarkM-

MarkM, I wish you could be a bit less wordy.  You have good thoughts, but your posts are incredibly difficult for me to follow, for some reason.

Anyhow, if I understand you correctly, your proposal is that people pay for files only when they need them?  So they say, "hey, I might need these files in the future, store them for me and I'll pay you $10 if I need them in the future".  Those with higher amounts of bandwidth would perhaps be able to provide those files more quickly, and thus more likely to receive the bounty for a file download in the future if it is necessary.  That's an interesting twist...  I just wonder if people would be willing to store files if future payment is unknown.

I am not sure how the original idea would work with regards to bandwidth vs storage.  Maybe one provider would only be able to provide 100 GB of bandwidth even though they have 1 TB of storage.  Or it might be vice-versa for another.  Perhaps a "bandwidth/storage" ratio would also be provided, along with how much storage is available.  And of course, an "uptime" indicator and data-checker would be a necessity.d

Okay then great. The store blocks on speculation model might work at least for some data. For example if you don't mind the data being stored in legible form, in the sense that anyone who would like a copy is welcome to buy a copy and is not likely to have any trouble executing it or displaying it or playing it - in general, "using it" - then heck thousands of nodes might eagerly store it for you forever just on the strength of the income they hope to get by selling it on to others.

As we have seen from the difficulty of finding hash collisions though, if the data is encrypted the chances are very low that anyone other than you will ever be paying us for a copy of some random block of an encrypted file only you know the key for, so the future looks kind of dim unless we do have some reason to believe you either will buy it back or might very unlikely buy it back but will pay well for it if you ever do. We'd store your data on speculation, much as some three-letter agencies maybe try to store every phone conversation that ever passes over international lines or some other vast collection of possibly largely/mostly useless information.

But is it important for us to care whether or not that is how karma points actually work inside the code/network?

Can we mere humans not get along fine simply deciding whether or not to buy our node some more karma points instead of worrying about the nitty gritty details of how this version or the next updated version of the software actually does its wheeler-dealer dealing to come up with the karma balances it assigns to its neighbors?

Don't we really just want our backups when our local system crashes? Or want our photographs back that we moved offsite without even keeping a local copy of?

Certainly I want offsite backups, real ones in the sense that I actually can get a copy from offsite if my onsite data ever does get lost thus create a need for restoring from backups and my onsite backups also are lost. It would be nice if some shopping app could shop around and find the best price for sufficiently reliable storage but its not as mission-critical as actually having such backups in place.

Right now I could pay bitcoins for karma points on other people's GNUnet nodes if my own GNUnet node turns out to have insufficient karma to accomplish this critical mission.

It happens though that I have not yet experienced sufficient shortfall of such points to be aware yet of any current need to boost my karma by paying for karma points. Maybe other people have?

-MarkM-

That's a pretty good description of the theory of operation of a futures market. Believe it or not such markets exists and function reasonably well.

Ok that is strange, I had the impression that since you cannot know until the the information is delivered back to you or some kind of N-knowledge proof (where N might even be zero even???) proves they do still have it when the time you paid for is up, and possibly even had it constantly between the start of the timespan and the end of the timespan, they they actually did store it.

Thus my impression had been that it is better to pay after the fact for performance than to believe bids up front.

Thus I thought an "I will pay X for block Y" much more robust.

I suppose maybe asking for estimates ahead of time for how much the network thinks it might cost to buy such a block some certain timeperiod after you submit it for consideration might help give you hints about how often to buy a copy for how much in order to ensure someone out there will be eager to sell it to you next time you ask for it, but really, if it turns out you can get it cheaper from numerous providers than the initial estimates you were given, why not go the cheaper route?

However, I guess you are more in the market for some vapourware-provided service than for something people can actually fire up an app to provide to you by the sound of it?

Is there any bounty offered for moving this vapourware onward toward people being able to actually store your, uh, how many bytes exactly for, uh, how long a timespan is it you want it stored for? Oh and how much are you offering for storing the data once the software to do this for you has been gotten up and running?

I am trying to move on to actually getting offsite backups up and running, having to wait for some vapourware first isn't very appealing. I'd prefer to get to a point where people can store data and can if they wish use bitcoins to encourage the robustness of the storage solution before waiting for some future development that might make doing that even easier than it already is using already working code.

-MarkM-

I have read a lot of such material on many distributed storage systems.

I guess if you are offering to buy a resource and can provide the software or provide a URL to download a free open source implementation of the software to provide you that resource, I'd consider installing it and selling you the resource maybe depending on how much of the resource you want to buy how regularly for how long.

These karma points are basically just another resource, they can be experienced as disk space and/or as bandwidth, a kind of spacetime resource maybe. If you don't want to buy that, point to code that explicates executably the precise resource it is you are looking to buy.

-MarkM-

Maybe it would help if you read up on some of the whitepapers of Mojo Nation and other similar projects. There's already a lot of material written on how market-based allocation of computing resources could work and it would save a lot of time to not rehash all that in this thread.

What service?

DIstributed storage tends to be crypted blocks, so maybe you want to buy blocks? Specific blocks, offering more for hard to find blocks than for blocks you can get swiftly and easily from umpteen providers?

-MarkM-

EDIT: Example: you pay nodes to let you send them a block. They wonder whether the block will ever pay anything more than what you paid to have them let you connect to them and deliver it. Some day you offer them bitcoins for a copy of it. You pay well, and pay them to let you tell them the bits of another block you might or might not want to buy a copy of at some later date. Etc.

I suspect a karma<->BTC market might well work better. A human might not hover all day over their node day-trading karma, but might some friday night upon noticing their node didn't have enough karma to fetch them the data they asked for on monday decide to give it some bitcoins so it can buy some karma to see if that can manage to get the data before the weekend is over...

The same way that any other paid online services work right now - the owner wants to buy the service.

I want to store X MB for Y hours so I publish this information and then then let the those nodes who want to sell me storage bid on it. My node uses a set of heuristics to pick the best bid taking into account all relevant information (price, reputation, bandwidth, etc). The system does this for every resource that you want to allocate.

Okay so lets rename karma as bitcoins. Or as satoshis or as millibitcoins or whatever.

Are you also proposing that the nodes mine bitcoins?

If not, how will any node ever actually convince its owner to provide it any bitcoins?

Are you planning to put some bitcoins in your node's wallet? If so, how many?

I figured, maybe wrongly, that most people who install the software to see if it can make them some pennies will most likely initially run it a while to see how many pennies it brings in before considering whether maybe their node simply isn't competetive enough to bring in any and from there whether they have any actual use for distributed storage themselves that might justify providing their node with some bitcoins instead of leaving it bring in bitcoins itself.

So maybe lets intially survey how many bitcoins are on offer, in order to consider whether to even bother firming up the details of the contemplated software so that folks can run it to harvest those offered coins...

...Or, we could use the already existing so called karma points for now, with an eye to figuring out whether in the end we will be calling them satoshis or bitcoins or millibitcoins or what.

Failing that, what exact services do you wish the markets to offer up for sale for bitcoins? Bits per second seconds of data-transfer maybe? Along with bits per second or day or hour or year or whatever of storage? Or, pay per block, you bid X satoshis for block ASghk235bbld6w566bnlwlsl66 and first node to deliver it gets paid? Or what?

-MarkM-