More useful-looking code:
https://bitbucket.org/blueluna/transmission_supervisor/wiki/Home
http://www.oneswarm.org/about.html
http://en.wikipedia.org/wiki/Opentracker
I am wondering how much of an advertising budget folks looking to sell storage space are willing to spend to find customers for such a service; what percentage of gross earnings will it take to maintain a given level of gross earnings or even increase gross earnings? What percent commissions should people who actually manage to bring a paying customer get, and should it be only a percentage of the actual payment they convince the customer to make instead of an on-going percent, so that salespeople will be motivated to keep in touch with customers encouraging them to renew when their initial payment has been exhausted?
It is all very well to say lets bring those looking for money together with those who have money and want to spend it, but the competition these days for people who even merely have money or might have money, let alone are actually willing to spend it, has gotten a lot more intense, and more expensive, than back in the gravy-train raining-soup days of the very very late twentieth century.
Thus budgeting for purchase of marketing from outsiders (buying adsense ads facebook ads and so on conceivably, maintaining email capture landing-pages and professional third party mailing-list services, purchase of raw hits, etc etc etc), and/or actual marketing by insiders - surfing manual-surf traffic-exchanges? Running auto-surfs in browsers 24/7? Or merely contributing money to pay for professional marketing?
I guess the implication here is am not seeing a whole lot of "heck yes I'll buy X gigabytes a month at Y bitcoins a month per gigabyte" type of comments so I am dubious of finding much paying demand.
Reverting to the Treasure Map model was largely because of having that impression; I had drifted toward lets see who is still online in a few months of freely sharing and see how many months consistent reliable storage hosts will end up having to be online before starting to see people actually offering money instead of merely offering storage of their own. I have also, y'see, an impression that people are more willing to offer storage than to offer money thus if they are reluctant to even make their storage available it is even less likely they will ever actually make any of their money available.
Maybe those with bitcoins can demonstrate control of addresses with bitcoins in them for a few months to show they not only have coin but are disciplined enough to still have coin a month or few down the line, while those who have space can demonstrate they not only have space but it can be utilised and still has in it in a month or few what was entrusted to it, then maybe we can get a better idea of whether this is going to be a workable matchmaking project?
Bearing in mind that if we can do this with off the rack software and already existing networks which might have "network effects" of their own already, the potential paying customers might well notice that what we have come up with could probably work sufficiently well without paying any actual money. Afterall look at the numbers of people doing bitswarming already, each "seeder" providing storage already if only while their own downloading is happening.
The only example we have encountered so far of someone who actually needs space but has none to offer turns out to be probably better served by begging borrowing or stealing a spare disk drive from her brother, so is not actually a case study in favour of this project afterall...
Maybe in reality most use-cases of wanting/needing off-site storage tend to be not really for storage at all but actually for sharing (with oneself and/or others), the desire to get the data off the site being to have it somewhere all one's friends can look at it and enjoy it and so on, you oneself can access it from anywhere and so on, thus maybe tend to be more about bandwidth in a way than storage?
Maybe those looking to make money with their storage capacity would be best off stocking it with content many people want but not quite enough people for that specific content to already be 24/7 available for free?
-MarkM-
Topic: [IDEA] Dirt cheap online storage - page 4. (Read 14501 times)
That website is pretty neat! Looks like an easy way for people to quickly and easily create a torrent and get it seeded at the same time.
I think a consistent payout is better than a treasure map that will pay out only to one person at one point in time. There's nothing wrong with some centralization to organize and achieve this.
I think a consistent payout is better than a treasure map that will pay out only to one person at one point in time. There's nothing wrong with some centralization to organize and achieve this.
It had been quite a while since last I had read up in Bittorrent, so I am wikipeding it now.
Take a look at http://burnbit.com/
Looks handy for anyone who can upload their file to their website to have their website be their initial-seed host.
I remember this part:
"Routers that use network address translation (NAT) must maintain tables of source and destination IP addresses and ports. Typical home routers are limited to about 2000 table entries[citation needed] while some more expensive routers have larger table capacities. BitTorrent frequently contacts 20–30 servers per second, rapidly filling the NAT tables. This is a common cause of home routers locking up.[42]"
DIstributed key word search sounds interesting and there is free open source code for it (see for example Tribler).
Maybe we don't need any centralisation at all. Adding money into the equation might be do-able just with a Treasure Hunt script, on top of which some GUI-lover can put a GUI, and a Bury Treasure script likewise which an initial-seeder (person hoping to get copies of a file to persist a while out in torrentland) can use to bury treasure in a file before initial seeding and to advertise a treasure map some time later.
("By the way dutiful retainers of my data, if you take a look at bytes X through Y, you might recognise that there, but for ROT13, is a bitcoin private key. Those of you who know not what ROT13 might signify are welcome to await my next clue, or, if impatient, to perform some research on the topic...")
-MarkM-
Take a look at http://burnbit.com/
Looks handy for anyone who can upload their file to their website to have their website be their initial-seed host.
I remember this part:
"Routers that use network address translation (NAT) must maintain tables of source and destination IP addresses and ports. Typical home routers are limited to about 2000 table entries[citation needed] while some more expensive routers have larger table capacities. BitTorrent frequently contacts 20–30 servers per second, rapidly filling the NAT tables. This is a common cause of home routers locking up.[42]"
DIstributed key word search sounds interesting and there is free open source code for it (see for example Tribler).
Maybe we don't need any centralisation at all. Adding money into the equation might be do-able just with a Treasure Hunt script, on top of which some GUI-lover can put a GUI, and a Bury Treasure script likewise which an initial-seeder (person hoping to get copies of a file to persist a while out in torrentland) can use to bury treasure in a file before initial seeding and to advertise a treasure map some time later.
("By the way dutiful retainers of my data, if you take a look at bytes X through Y, you might recognise that there, but for ROT13, is a bitcoin private key. Those of you who know not what ROT13 might signify are welcome to await my next clue, or, if impatient, to perform some research on the topic...")
-MarkM-
Torrents break the files into regular and specific piece sizes, and hash each piece with SHA-1. The list of hashes is included in the .torrent file. This is my understanding based on the wikipedia article on bittorrent.
Markm, exactly, and that is where my limited knowledge of torrenting has me hung up. I am not sure how parts of a file are downloaded and verified. It possible that the .torrent file contains the checksum or hash information of all the pieces of a particular file, so that each piece can be checked as it is downloaded. I need to do some reading on "how stuff works: torrenting" or something.
Sukrim, sounds like a decent summary of the conclusions made so far. Thanks for posting.
I was thinking about the potential seed speed issue as well... not sure how to solve that one, besides users reporting any hosts that do not meet the minimum speed requirement. Since torrent seeders can typically set whatever upload speed they like, and the speed set is transparent to any users downloading from more than one host, it would be difficult to track. But, if a way to see the download speeds from each host is implemented, as well as a way for users to report bad hosts to service.com, the threat of being permanently removed from participating in the network might be enough to ensure hosts at least provide the minimum speed required.
Smart (and malicious) hosts could allow quick uploads to service.com, but slow uploads to anyone else. That way, if service.com was checking their seed speed, they would appear fast. Service.com might counter by using multiple IP addresses, but those could become known by users making false reports, then recording any new downloading IP's to find out what service.com's other IP addresses are. Aside from using IP-changing VPN's around the world, I don't see a good way to counter this besides user feedback.
Sukrim, sounds like a decent summary of the conclusions made so far. Thanks for posting.
I was thinking about the potential seed speed issue as well... not sure how to solve that one, besides users reporting any hosts that do not meet the minimum speed requirement. Since torrent seeders can typically set whatever upload speed they like, and the speed set is transparent to any users downloading from more than one host, it would be difficult to track. But, if a way to see the download speeds from each host is implemented, as well as a way for users to report bad hosts to service.com, the threat of being permanently removed from participating in the network might be enough to ensure hosts at least provide the minimum speed required.
Smart (and malicious) hosts could allow quick uploads to service.com, but slow uploads to anyone else. That way, if service.com was checking their seed speed, they would appear fast. Service.com might counter by using multiple IP addresses, but those could become known by users making false reports, then recording any new downloading IP's to find out what service.com's other IP addresses are. Aside from using IP-changing VPN's around the world, I don't see a good way to counter this besides user feedback.
You would need the following:
service.com:
Gets Bitcoin payment + has a "upload file or torrent to seed" field on the page + displays a daily rate.
Offers a service.com integrity checker client (can be a fairly simply python script probably).
Offers a page with all files hosted + potential daily rate for seeding them.
uploader/user:
Either creates a torrent file and uploads it or simply uploads a file to be distributed to service.com
Service.com then creates a bunch of random check points for arbitrary points in the file (say: MD5 of Bytes 1337-31337, SHA1 of Bytes 23-42...) and puts it on the offer page, as soon as the BTC payment came in.
seeder/server owner:
Checks the offer pager + installs the integrity checker client and any bittorrent client.
If one or more offers are appealing, he loads the torrent into the BT client as well as into the integrity checker client.
This client then allows him to download the file quickly from service.com if he is the first seeder and handles payments as well as the random integrity/uptime checks (from service.com: "What's the MD5 of Bytes 1337-31337?" - "abc123" - "Correct!").
Something still to be determined though is how to make sure the seeders really seed with some guaranteed bandwidth. Maybe this can be done by having a modified bittorrent client on the leecher side that rates the seeders accordingly... At least the problem of how to make sure someone really has a complete file for a certain amount of time though is solvable this way in my opinion.
Something interesting to look at for potentially scoring seeders: http://en.wikipedia.org/wiki/Glicko_rating_system
Now how can we trust the downloaders though...
service.com:
Gets Bitcoin payment + has a "upload file or torrent to seed" field on the page + displays a daily rate.
Offers a service.com integrity checker client (can be a fairly simply python script probably).
Offers a page with all files hosted + potential daily rate for seeding them.
uploader/user:
Either creates a torrent file and uploads it or simply uploads a file to be distributed to service.com
Service.com then creates a bunch of random check points for arbitrary points in the file (say: MD5 of Bytes 1337-31337, SHA1 of Bytes 23-42...) and puts it on the offer page, as soon as the BTC payment came in.
seeder/server owner:
Checks the offer pager + installs the integrity checker client and any bittorrent client.
If one or more offers are appealing, he loads the torrent into the BT client as well as into the integrity checker client.
This client then allows him to download the file quickly from service.com if he is the first seeder and handles payments as well as the random integrity/uptime checks (from service.com: "What's the MD5 of Bytes 1337-31337?" - "abc123" - "Correct!").
Something still to be determined though is how to make sure the seeders really seed with some guaranteed bandwidth. Maybe this can be done by having a modified bittorrent client on the leecher side that rates the seeders accordingly... At least the problem of how to make sure someone really has a complete file for a certain amount of time though is solvable this way in my opinion.
Something interesting to look at for potentially scoring seeders: http://en.wikipedia.org/wiki/Glicko_rating_system
Now how can we trust the downloaders though...
Content-Addressable Storage (CAS) uses the content to address storage not the location in the world or on disk or whatever at which it is stored nor some nickname some user might find humanly appropriate characterising the file or its content.
This, any change in the content of a block / unit/ file / whatever of storage changes the identity/ID of that block /unit/ file / whatever.
Basically the hash is what one refers to it by, so anything that does not hash to the hash that was requested is not what was requested.
Doing this block by block instead of file means that any block can be requested.
Many systems work this way, including, I think, Freenet, Tahoe-LAFS, GNUnet, Twisted Storage, and camlistore. Maybe even bittorrent. If your bittorrent does not offer you a user-interface for asking for individual blocks or sectors or fragments or somesuch of a file, likely it is merely hiding that functionality from you the poor ignorant human, while using it itself to request specific portions of a file from specific hosts in order to download many portions at once from many hosts.
So maybe you just have to dig a little deeper into some kind of advanced mode or debug mode of your client, or complain to its developers that the user-interface is somehow not making clear to you how to tell it yourself which piece to get from where.
-MarkM-
This, any change in the content of a block / unit/ file / whatever of storage changes the identity/ID of that block /unit/ file / whatever.
Basically the hash is what one refers to it by, so anything that does not hash to the hash that was requested is not what was requested.
Doing this block by block instead of file means that any block can be requested.
Many systems work this way, including, I think, Freenet, Tahoe-LAFS, GNUnet, Twisted Storage, and camlistore. Maybe even bittorrent. If your bittorrent does not offer you a user-interface for asking for individual blocks or sectors or fragments or somesuch of a file, likely it is merely hiding that functionality from you the poor ignorant human, while using it itself to request specific portions of a file from specific hosts in order to download many portions at once from many hosts.
So maybe you just have to dig a little deeper into some kind of advanced mode or debug mode of your client, or complain to its developers that the user-interface is somehow not making clear to you how to tell it yourself which piece to get from where.
-MarkM-
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file. If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.
Ok, so service.com is the cluster co-ordinator for a bunch of storage nodes, or maybe is even further up a hierarchy than that.
How does the customer know how many copies of his/her file are seeded? Specifically, are storage.com's claims about that believed or does the customer get to see directly peer to peer with the hosts hosting the copies that they claim to be seeding it?
I ask because I am wondering if each host can add a firstbits string to the name by which they seed a copy.
For example say you want to store SgtSpike-Desktop-2012-05-31.bak.rar
I have a firstbits of 1DIgitalis, so I seed it as SgtSpike-Desktop-2012-05-31.bak.rar.1DIgitalis or whatever: basically I put my firstbits in there, or if filename length isn't a problem, then lets forget firstbits and use full bitcoin-addresses.
Now you should be able to ask bittorrent for a list of all files whose name start with SgtSpike? or maybe for any with keywords of SgtSpike and Desktop? Or if wondering what else I am hosting, all files with 1Digitalis in the name, or something like that?
Or does bittorrent have metadata even, like GNUnet and hmm not sure but I think maybe Tahoe-LAFS do, so a seeder can associate a whole bunch of keywords with a file without having to stuff them into the filename?
I am thinking here maybe we could eliminate service.com entirely even, simply by including in the keywords or filenames the offered reward for seeders from some offering seeder, the identity of the offering seeder, and the bitcoin address rewards for the copy having this name or keywords-set (description?)
Service.com could be useful as escrow though, so storage nodes do not have to trust some random seeder's offer just storage.com's claim that storage.com will make the offer good.
-MarkM-
I am not sure if torrents are capable of containing metadata, and if they are, if it would screw up the "copy" status and not just make it a whole new separate file if some of that metadata was changed. But you're thinking along the same lines as I am in that regard - the seeder would somehow advertise their bitcoin address whenever they are seeding a paid file. Service.com wouldn't be absolutely necessary, but I think it would be valuable to match up uploaders with those who are looking for the paid torrents, as well as acting as escrow to ensure that the uploader actually paid for the uploads.
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.
Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
Wouldn't you have to keep some data on hand to make sure the checksum or hash matched? In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
First someone takes a checksum of the file, storing that on their computer. They upload the file to the network, then delete their local copy (to free disk space). Once the person receives the file again once they need it, they simply take the checksum of the received data and see if it matches the previous one.
Is that overkill?
I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts. Gladamas touched on it, but I am not enough of a programmer to understand how it might work.
I'm not really a developer either, but this is what I understand: Alice has files to upload onto the network, and Bob has disk space to share. Alice encrypts her data with her public key (so only she can see her data). Alice sends the encrypted data and her public key to Bob. Bob then encrypts his Bitcoin address using his private key. He then sends that and his public key to Alice, who can then authenticate the message using Bob's public key.
Anyone else can then prove that Bob was the owner of the Bitcoin address by decrypting the message with Bob's public key. Bob can also prove that he owns that private key by having someone else encrypt a message with Bob's public key, then Bob decrypts it with his private key and verifies the contents of the message with the person who sent it.
I'm curious, though, how to stop Alice from, say, denying that Bob ever sent her his Bitcoin address, or if Alice simply refuses to pay.
That's why I think a central entity needs to be involved to manage payments. Otherwise, hosts would have to rely on the reputation of the uploader to ensure file seeds are paid.
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file. If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.
Ok, so service.com is the cluster co-ordinator for a bunch of storage nodes, or maybe is even further up a hierarchy than that.
How does the customer know how many copies of his/her file are seeded? Specifically, are storage.com's claims about that believed or does the customer get to see directly peer to peer with the hosts hosting the copies that they claim to be seeding it?
I ask because I am wondering if each host can add a firstbits string to the name by which they seed a copy.
For example say you want to store SgtSpike-Desktop-2012-05-31.bak.rar
I have a firstbits of 1DIgitalis, so I seed it as SgtSpike-Desktop-2012-05-31.bak.rar.1DIgitalis or whatever: basically I put my firstbits in there, or if filename length isn't a problem, then lets forget firstbits and use full bitcoin-addresses.
Now you should be able to ask bittorrent for a list of all files whose name start with SgtSpike? or maybe for any with keywords of SgtSpike and Desktop? Or if wondering what else I am hosting, all files with 1Digitalis in the name, or something like that?
Or does bittorrent have metadata even, like GNUnet and hmm not sure but I think maybe Tahoe-LAFS do, so a seeder can associate a whole bunch of keywords with a file without having to stuff them into the filename?
I am thinking here maybe we could eliminate service.com entirely even, simply by including in the keywords or filenames the offered reward for seeders from some offering seeder, the identity of the offering seeder, and the bitcoin address to which to send rewards for the copy having this name or keywords-set (description?)
Service.com could be useful as escrow though, so storage nodes do not have to trust some random seeder's offer just storage.com's claim that storage.com will make the offer good.
-MarkM-
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.
Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
Wouldn't you have to keep some data on hand to make sure the checksum or hash matched? In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
First someone takes a checksum of the file, storing that on their computer. They upload the file to the network, then delete their local copy (to free disk space). Once the person receives the file again once they need it, they simply take the checksum of the received data and see if it matches the previous one.
I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts. Gladamas touched on it, but I am not enough of a programmer to understand how it might work.
I'm not really a developer either, but this is what I understand: Alice has files to upload onto the network, and Bob has disk space to share. Alice encrypts her data with her public key (so only she can see her data). Alice sends the encrypted data and her public key to Bob. Bob then encrypts his Bitcoin address using his private key. He then sends that and his public key to Alice, who can then authenticate the message using Bob's public key.
Anyone else can then prove that Bob was the owner of the Bitcoin address by decrypting the message with Bob's public key. Bob can also prove that he owns that private key by having someone else encrypt a message with Bob's public key, then Bob decrypts it with his private key and verifies the contents of the message with the person who sent it.
I'm curious, though, how to stop Alice from, say, denying that Bob ever sent her his Bitcoin address, or if Alice simply refuses to pay.
I think you're a bit confused about how torrenting actually works.
There would be no concept of users on the seeding side, only files and payments. The site would organize the torrents available, and show how much (currently) a seeder would be paid per day to host a particular file, assuming no change in the current number of seeders of the file.
Again, torrenting is a pull operation, not a push operation. A seeder would have to request to download a file from the uploader in order to download it - the uploader couldn't push a file onto the seeder's system.
I think torrent software is usually set up by default to seed anything that is downloaded. In fact, it usually seeds the portions of the file it has WHILE downloading.
Essentially, this is how it would work:
1) An uploader wishes to back up encrypted group of files b1.rar.
2) The uploader creates a .torrent file for b1.rar.
3) The uploader funds his account at service.com with 1 BTC, and states that he will pay 0.01 BTC/day split between all users seeding b1.rar.
4) Various users visit service.com, notice the uploader's proposal, and downloads the .torrent file. This .torrent file tells their computers where they can download pieces of b1.rar (which would be from all seeders - note that the uploader is a seeder as well).
5) The hosts start seeding the full file as soon as it is fully downloaded.
5) Once the uploader is satisfied with the number of hosts seeding replicas of his file, he takes his file offline and no longer seeds it.
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file. If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.
There would be no concept of users on the seeding side, only files and payments. The site would organize the torrents available, and show how much (currently) a seeder would be paid per day to host a particular file, assuming no change in the current number of seeders of the file.
Again, torrenting is a pull operation, not a push operation. A seeder would have to request to download a file from the uploader in order to download it - the uploader couldn't push a file onto the seeder's system.
I think torrent software is usually set up by default to seed anything that is downloaded. In fact, it usually seeds the portions of the file it has WHILE downloading.
Essentially, this is how it would work:
1) An uploader wishes to back up encrypted group of files b1.rar.
2) The uploader creates a .torrent file for b1.rar.
3) The uploader funds his account at service.com with 1 BTC, and states that he will pay 0.01 BTC/day split between all users seeding b1.rar.
4) Various users visit service.com, notice the uploader's proposal, and downloads the .torrent file. This .torrent file tells their computers where they can download pieces of b1.rar (which would be from all seeders - note that the uploader is a seeder as well).
5) The hosts start seeding the full file as soon as it is fully downloaded.
5) Once the uploader is satisfied with the number of hosts seeding replicas of his file, he takes his file offline and no longer seeds it.
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file. If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.
Thank you! Okay, I now agree that bittorrent is a strong candidate.
Quite a few other systems can split the load and also migrate blocks towards regions where those blocks are popular though, which I have not heard of bittorrent doing. My impression has been that with bittorrent you deal with an entire file at a time, so any host that has the file has the file?
And the only migration in response to regional/directional demand is when someone not only downloads the file but also chooses to make downloaded files available for others to get from them?
Actually, can you even have all downloads automatically seeded for others to get from you, or do you (at least with whatever client it is that makes you feel so sure everyone already knows how to use it) have to explicitly pick from among your downloads specific files on a case by case basis that you want to "seed" for others to retrieve from you?
Block by block storage could let more people get in on the money even when the paying customer only wants a few copies. Some kind of RAID type system or even the Nimbus.io system could split the data up in such a way that X out of N nodes would have to die to cause loss of data. We could make some copies be full copies, putting all the blocks of a file on one host - all its eggs in one basket - and then also have some scattered-blocks copies also to share the wealth, basically trying to make sure all our storage-providers get at least some token amount of data to store.
The migration features of some distributed storage systems could also provide trickles for people who are along the way toward regions where a particular file is most popular, as blocks migrate toward the region where they are in high demand.
However okay if bittorrent already provides some of that stuff, by being able to automatically make available anything a node downloads, lets move on for now until I look again more closely at the distributed storage systems that do resemble bittorrent in those ways.
So, the money. Well, disk quotas still are on the table as a means by which someone providing space can limit the amount of space they are providing, if for some reason bittorrent does not already include setting how much disk space mximum you wish it to use. How does bittorrent know who is using how much space though? Maybe we need to look at bittorrent server software which google shows plenty of. Maybe if you run a bittorrent server instead of just a client, you will find it lets you allocate space to various users and stuff like that?
If bittorrent does not maintain some kind of concept of which users are using how much space, does it even maintain some concept of users at all, associating each file with a user?
If not you might have to run one instance per user whose usage you wish to account separately.
I am thus starting to suspect that instead of a peer to peer storage-node model we might have to go for a peer to peer cluster model, in which storage nodes only have one customer, which is their cluster co-ordinator node. Normal people who just want to sell some space could deal directly with their nearest cluster-coordinator or pick some other hopefully not too distant cluster co-ordinator.
We then don't need to worry about normal home-user types having to deal with complicated issues, they can simply try to set up a link of some kind with their chosen cluster-coordinator and see if they notice enough satoshis coming in at the end of each month to motivate them to stick it out another month or even maybe to do some marketing of the service to try to bring in some customers.
-MarkM-
EDIT: Maybe since security has come up we should from the get-go assume storage nodes will be virtual machines, anyone wanting to be a storage node simply fires up some standard virtual machine specific to the purpose? That gives them a disk space max and CPU usage max right off the bat without having to get into granular by user/group usage quota systems.
Quite a few other systems can split the load and also migrate blocks towards regions where those blocks are popular though, which I have not heard of bittorrent doing. My impression has been that with bittorrent you deal with an entire file at a time, so any host that has the file has the file?
And the only migration in response to regional/directional demand is when someone not only downloads the file but also chooses to make downloaded files available for others to get from them?
Actually, can you even have all downloads automatically seeded for others to get from you, or do you (at least with whatever client it is that makes you feel so sure everyone already knows how to use it) have to explicitly pick from among your downloads specific files on a case by case basis that you want to "seed" for others to retrieve from you?
Block by block storage could let more people get in on the money even when the paying customer only wants a few copies. Some kind of RAID type system or even the Nimbus.io system could split the data up in such a way that X out of N nodes would have to die to cause loss of data. We could make some copies be full copies, putting all the blocks of a file on one host - all its eggs in one basket - and then also have some scattered-blocks copies also to share the wealth, basically trying to make sure all our storage-providers get at least some token amount of data to store.
The migration features of some distributed storage systems could also provide trickles for people who are along the way toward regions where a particular file is most popular, as blocks migrate toward the region where they are in high demand.
However okay if bittorrent already provides some of that stuff, by being able to automatically make available anything a node downloads, lets move on for now until I look again more closely at the distributed storage systems that do resemble bittorrent in those ways.
So, the money. Well, disk quotas still are on the table as a means by which someone providing space can limit the amount of space they are providing, if for some reason bittorrent does not already include setting how much disk space mximum you wish it to use. How does bittorrent know who is using how much space though? Maybe we need to look at bittorrent server software which google shows plenty of. Maybe if you run a bittorrent server instead of just a client, you will find it lets you allocate space to various users and stuff like that?
If bittorrent does not maintain some kind of concept of which users are using how much space, does it even maintain some concept of users at all, associating each file with a user?
If not you might have to run one instance per user whose usage you wish to account separately.
I am thus starting to suspect that instead of a peer to peer storage-node model we might have to go for a peer to peer cluster model, in which storage nodes only have one customer, which is their cluster co-ordinator node. Normal people who just want to sell some space could deal directly with their nearest cluster-coordinator or pick some other hopefully not too distant cluster co-ordinator.
We then don't need to worry about normal home-user types having to deal with complicated issues, they can simply try to set up a link of some kind with their chosen cluster-coordinator and see if they notice enough satoshis coming in at the end of each month to motivate them to stick it out another month or even maybe to do some marketing of the service to try to bring in some customers.
-MarkM-
EDIT: Maybe since security has come up we should from the get-go assume storage nodes will be virtual machines, anyone wanting to be a storage node simply fires up some standard virtual machine specific to the purpose? That gives them a disk space max and CPU usage max right off the bat without having to get into granular by user/group usage quota systems.
Eh, I've had some experience with VPN's, and my experience has shown that they are complicated to set up and maintain, they are somewhat unreliable, and they do not allow some requisite features necessary for this type of project (i.e., allowing continuation of file transfer after interruption). Plus, allowing anyone into your file system, even if it seems to be contained, is a rather glaring security hole. And I am uncertain that multiple VPN connections can be made at the same time without consequence.
Here's why I think torrenting would work well:
- After multiple hosts have the files, it reduces the load on any single host to make additional copies of the data.
- It already allows resuming interrupted downloads, managing download speeds and bandwidth usage, etc.
- No security holes or issues.
- It is scalable - if someone wants their data to be ULTRA safe, they can pay such a ridiculously large sum of money on a daily basis to each person seeding the file that hundreds or maybe thousands of people will host said file to "get in on the money".
- If using VPN's, the hosts couldn't just allow anyone to download from them, as they wouldn't want to put their VPN information out in public. This would mean that the burden of creating additional copies of the original content would lie with the uploader. What happens if the uploader no longer has the content, and one of the hosts decides to move on to other files? With torrenting, this would be fine - a host might decide it is economical to start seeding a particular torrent, and download from the other hosts already seeding the content. With VPN's, the uploader would have to re-download the information from one of the VPN's, then reupload it to a new one.
- Torrenting is already cross-platform without any special configuration.
I don't see VPN's working well unless hosts were willing to allow anyone to download from their VPN-hosted files. In which case, why use a VPN at all, and not a more standard FTP or HTTP protocol?
I'll emphasize this point again though: Torrenting reduces the load of any particular host, as it (more or less) evenly splits the load between all seeders. If a new person wants to download a particular file, and 5 hosts are already seeding it, then each host only has to use bandwidth equal to 1/5 of the total size of the file. With VPN, FTP, or HTTP, there is no easy way to split up the load between different hosts. With torrenting, it is automatic.
Bittorrent doesn't already do what I want, because it provides no incentive for other people to hold my backups. But, adding incentive for other people to hold my backups make torrenting a very viable platform. Also, there is no reason that torrent technology cannot be used for 1 host just as easily as thousands.
Here's why I think torrenting would work well:
- After multiple hosts have the files, it reduces the load on any single host to make additional copies of the data.
- It already allows resuming interrupted downloads, managing download speeds and bandwidth usage, etc.
- No security holes or issues.
- It is scalable - if someone wants their data to be ULTRA safe, they can pay such a ridiculously large sum of money on a daily basis to each person seeding the file that hundreds or maybe thousands of people will host said file to "get in on the money".
- If using VPN's, the hosts couldn't just allow anyone to download from them, as they wouldn't want to put their VPN information out in public. This would mean that the burden of creating additional copies of the original content would lie with the uploader. What happens if the uploader no longer has the content, and one of the hosts decides to move on to other files? With torrenting, this would be fine - a host might decide it is economical to start seeding a particular torrent, and download from the other hosts already seeding the content. With VPN's, the uploader would have to re-download the information from one of the VPN's, then reupload it to a new one.
- Torrenting is already cross-platform without any special configuration.
I don't see VPN's working well unless hosts were willing to allow anyone to download from their VPN-hosted files. In which case, why use a VPN at all, and not a more standard FTP or HTTP protocol?
I'll emphasize this point again though: Torrenting reduces the load of any particular host, as it (more or less) evenly splits the load between all seeders. If a new person wants to download a particular file, and 5 hosts are already seeding it, then each host only has to use bandwidth equal to 1/5 of the total size of the file. With VPN, FTP, or HTTP, there is no easy way to split up the load between different hosts. With torrenting, it is automatic.
Bittorrent doesn't already do what I want, because it provides no incentive for other people to hold my backups. But, adding incentive for other people to hold my backups make torrenting a very viable platform. Also, there is no reason that torrent technology cannot be used for 1 host just as easily as thousands.
Okay, so you have local area network sharing but are concerned about its security.
This many years or decades in has it not by now been given an option to encrypt the data on the wire?
If not, then a Virtual Private Network might be required, to set up host and customer with a private local area that happens to extend across the internet. That might well *not* be already right there on your system setup menus/icons on Windows. For Linux there are lots of them, I would ask my package manager aka installer to search then try to choose which one to actually use somehow maybe partly by whether it seems one I have heard of a lot or seems to be the usual vanilla standard or whatever... Aha I see "OpenVPN", the "Open" part has me sold on it at first sight.
Google WIndows VPN:
Configure a VPN Connection Using Windows XP
www.windowsecurity.com/.../configure-vpn-connection-windows-xp...
18 Mar 2005 – In this article we will learn how to configure a connection to a virtual private network (VPN) using Windows XP. This article will show you how to ...
Google goes on list other flavours of WIndows. (Link above is an error 404 but hey, back in 2005 with Windows XP this stuff already existed.)
For disk quotas googling "Windows disk quota" shows immediate results for various windows 2000+, windows server, and windows 7 systems, one might have to dig deeper for older systems and they might well have deliberately tried to keep such "professional" features out of old cheap home-user versions intended to lure professionals into upgrading.
But, okay, that is more suitable to you and your sister than to random strangers, whom you of course wisely do NOT want on your local area/net.
If Bittorrent already does what you want then great. If not I still do not really understand the fascination of it, because previously you have been writing about having one to three hosts host the customer's data. If the goal is to have hundreds or thousands or hundreds of thousands of hosts host it, then yes, I too would look at torrent technology. But for the probably for our purposes more common case of 1 to 99 hosts, with 1 to 25 or less likely far more common, I am not convinced of a need to resort to torrent.
-MarkM-
This many years or decades in has it not by now been given an option to encrypt the data on the wire?
If not, then a Virtual Private Network might be required, to set up host and customer with a private local area that happens to extend across the internet. That might well *not* be already right there on your system setup menus/icons on Windows. For Linux there are lots of them, I would ask my package manager aka installer to search then try to choose which one to actually use somehow maybe partly by whether it seems one I have heard of a lot or seems to be the usual vanilla standard or whatever... Aha I see "OpenVPN", the "Open" part has me sold on it at first sight.
Google WIndows VPN:
Configure a VPN Connection Using Windows XP
www.windowsecurity.com/.../configure-vpn-connection-windows-xp...
18 Mar 2005 – In this article we will learn how to configure a connection to a virtual private network (VPN) using Windows XP. This article will show you how to ...
Google goes on list other flavours of WIndows. (Link above is an error 404 but hey, back in 2005 with Windows XP this stuff already existed.)
For disk quotas googling "Windows disk quota" shows immediate results for various windows 2000+, windows server, and windows 7 systems, one might have to dig deeper for older systems and they might well have deliberately tried to keep such "professional" features out of old cheap home-user versions intended to lure professionals into upgrading.
But, okay, that is more suitable to you and your sister than to random strangers, whom you of course wisely do NOT want on your local area/net.
If Bittorrent already does what you want then great. If not I still do not really understand the fascination of it, because previously you have been writing about having one to three hosts host the customer's data. If the goal is to have hundreds or thousands or hundreds of thousands of hosts host it, then yes, I too would look at torrent technology. But for the probably for our purposes more common case of 1 to 99 hosts, with 1 to 25 or less likely far more common, I am not convinced of a need to resort to torrent.
-MarkM-
I'm a Windows user, as is my sister. I suppose I have just not heard of whatever this built-in file sharing capability Windows has though. Unless you are talking about local area network shares (mapped drives and all of that)? I've never heard it touted as a good way to share files publicly on the internet though - that seems insecure and a way to make yourself ripe for abuse, if it is even possible.
Also, remember, this isn't about my sister. I could find a number of solutions for her, including giving her one of my hard drives to use (though it wouldn't be as safe as having multiple people store it online). This is about finding a general-purpose low-cost online storage solution.
Also, remember, this isn't about my sister. I could find a number of solutions for her, including giving her one of my hard drives to use (though it wouldn't be as safe as having multiple people store it online). This is about finding a general-purpose low-cost online storage solution.
Windows Shares, or SAMBA, or whatever such capabilities are now called in Windows, existed in Windows 3-point-something or earlier.
What operating system are you using / used-to?
What operating system is your sister using / used-to?
I am talking about built in stuff that should be right in your system setup menus/icons somewhere already just waiting for you to turn it on, or in the case of Linux distributions conceivably for you to click on in the install packages tool to have them installed for you.
Certainly far more normal and standard that strange third-party stuff such as bittorrent or retroshare or whatever. It is native to the system / distribution / operating-system.
-MarkM-
What operating system are you using / used-to?
What operating system is your sister using / used-to?
I am talking about built in stuff that should be right in your system setup menus/icons somewhere already just waiting for you to turn it on, or in the case of Linux distributions conceivably for you to click on in the install packages tool to have them installed for you.
Certainly far more normal and standard that strange third-party stuff such as bittorrent or retroshare or whatever. It is native to the system / distribution / operating-system.
-MarkM-
I still don't see a good reason to deviate from a popular and known platform (torrenting) to something that doesn't even exist yet (Windows Shares?). That's just making things more complicated for the sake of making things more complicated.
Regarding software development, well, this is only idea exploration at this point. I am getting close to the point where I'd feel comfortable with dropping some coin on a programmer developing the idea though.
Regarding software development, well, this is only idea exploration at this point. I am getting close to the point where I'd feel comfortable with dropping some coin on a programmer developing the idea though.
Okay well also consider that even Windows users might already have as part of Windows something called something along the lines of Windows Shares, or SAMBA shares, or something like that, which should be able to provide them the ability to specify directories they would like others to be able to share, and to mount other people's such directories so they can just drag and drop stuff. The shared directories are "actually" shared, at least once they are mounted.
In Unix type systems, maybe including OS/X, there is a pretty standard remote mount system (NFS: Network File System) that lets you similarly make directories available for remote mounting, though how exactly you tell it who gets access and how is somewhat different.
There is (or certainly long ago used to be) SAMBA code for Unix type systems so that for example an ISP could offer Windows users mountable home-directories on the ISP's unix/linux/etc based servers.
Hosts that are Linux based even have available quota systems so they can assign quotas to users, and have 'cron' scheduling daemon that runs things on specified schedules. Such hosts could have cron run a script that looks up for each user a bitcoin account the balance of which will be used as the number of units of storage to set that user's quota to, and another, monthly script that deducts bitcoins from that account based on the maximum amount of storage that user had actually used in the last month.
Presumably "we" need not worry about what people who want to sell space would prefer in the way of software to be such a seller, except in proportion to the number of bitcoins they offer as bounty for each specific script or function etc they specify? Folk who can handle the technical stuff themselves can get up and running selling space to bring in coins first, limited possibly by the need for pretty front ends for the buyers of space to use in order to maximise the coins brought in by/to the project; then over time as the sellers find it worth their coins to have prettier software for their end made such software will or will not get made.
Basically if software is to be developed we want first to have coin in hand with which to motivate software developers to develop it?
Meanwhile those who are able to raise coin by setting up as sellers using existing software can attack the problem of what exactly do the people who want to buy space need (other than coin) in order to be able to buy it? Such as ability to tell their operating system to make available one or more directories as "shares"? Coupled possibly with the ability to tell their router to allow people coming in from the net looking for shares/specific-shares to reach which port of which machine to get those shares?
-MarkM-
In Unix type systems, maybe including OS/X, there is a pretty standard remote mount system (NFS: Network File System) that lets you similarly make directories available for remote mounting, though how exactly you tell it who gets access and how is somewhat different.
There is (or certainly long ago used to be) SAMBA code for Unix type systems so that for example an ISP could offer Windows users mountable home-directories on the ISP's unix/linux/etc based servers.
Hosts that are Linux based even have available quota systems so they can assign quotas to users, and have 'cron' scheduling daemon that runs things on specified schedules. Such hosts could have cron run a script that looks up for each user a bitcoin account the balance of which will be used as the number of units of storage to set that user's quota to, and another, monthly script that deducts bitcoins from that account based on the maximum amount of storage that user had actually used in the last month.
Presumably "we" need not worry about what people who want to sell space would prefer in the way of software to be such a seller, except in proportion to the number of bitcoins they offer as bounty for each specific script or function etc they specify? Folk who can handle the technical stuff themselves can get up and running selling space to bring in coins first, limited possibly by the need for pretty front ends for the buyers of space to use in order to maximise the coins brought in by/to the project; then over time as the sellers find it worth their coins to have prettier software for their end made such software will or will not get made.
Basically if software is to be developed we want first to have coin in hand with which to motivate software developers to develop it?
Meanwhile those who are able to raise coin by setting up as sellers using existing software can attack the problem of what exactly do the people who want to buy space need (other than coin) in order to be able to buy it? Such as ability to tell their operating system to make available one or more directories as "shares"? Coupled possibly with the ability to tell their router to allow people coming in from the net looking for shares/specific-shares to reach which port of which machine to get those shares?
-MarkM-
You could sign, using a bitcoin client, a message to your sister claiming that your machine and the bitcoin address used to sign the message are owned/controlled by the same person/company.
However, does she even get told who asked for her files? And if she displays a list of what files you have on your end is your machine identified similarly to the way it is identified in her logs of who has downloaded which of her files?
Since IP addresses are not usually stable for many home users, does bittorrent have some kind of node-identity that it uses for such identifications/logs?
-MarkM-
This is the part I am really unsure of. I know IP addresses identify torrent clients, but beyond that, I am not sure what other information they can make public about themselves or request from others. Gladamas seemed to indicate that it would be easy to attach a Bitcoin address to a particular torrent seeder, but I don't know the specifics about how that would work. However, does she even get told who asked for her files? And if she displays a list of what files you have on your end is your machine identified similarly to the way it is identified in her logs of who has downloaded which of her files?
Since IP addresses are not usually stable for many home users, does bittorrent have some kind of node-identity that it uses for such identifications/logs?
-MarkM-
Jump to: