Pages:
Author

Topic: Blockchain + DHT = secure email-like messaging. - page 2. (Read 4640 times)

sr. member
Activity: 389
Merit: 250
tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"
On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.
Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.
donator
Activity: 452
Merit: 252
tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"
On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.
sr. member
Activity: 389
Merit: 250
tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"
On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.
sr. member
Activity: 389
Merit: 250
Great post!
Yes it's all very generic, what makes my proposal reasonable (hopefully) is the combination of features. Once again, I want to build a secure messaging analogous to email with all its features but without any centralization and asymmetric encryption ingrained in the system.

1. You have to exchange keys with current PGP setup. Also you need to have SMTP and POP servers to actually process your messages. If they want to mess with your messages they can.
The proposal is to use public keys as receiver addresses, so no keys should be exchanged. And build a strict peer-to-peer system.
2. The problem with storage can be solved more or less as it's sometimes done with Bittorent private trackers - if you wanna use the system you have to give back, that is you have to store messages too. People store tones of multimedia files at their home PC's, probably storing 100 mb 1 GB of data is all right. Bitcoin block chain is 6 GB now.
Another way to approach this is the nodes who store the data for a fee, this also would be a way to monetize the system. For example heavy users would be required to pay a fee through bitcoin which will be redistributed between storing nodes. 
3. The way to approach the spam problem would be PoW, analogous to the way its made in Bitmessage. Also a peer would be allowed to block messages from certain nodes. in this case the messages will be dropped from the system
4. This is more like crypto twitter. Actually the spam problem won't be so drastic here since you need to know the public key of the node if you want to subscribe to its messaging stream, it can't message you directly.
5. Appropriate DHT setup is able to solve the problems of peers going offline, as it is proven in practice by various P2P systems. As for central servers - I believe that complete decentralization is the way to go, Internet is inherently decentralized system which certain parties always try to centralize.
There have to be countermeasures to this. Just imagine a sheer amount of private information any webmail provider has access to. I'm not sure that this is the ideal setup of human communication, when a third-party always has access to anything it wants. It creates a new hierarchy, which actually is not necessary, all communication CAN be peer-to-peer indeed.

Freenet is cool but it's more file sharing application. Also there won't be an explicit blockchain here, everything can be done through correct DHT setup.

once again thank you for very insightful comments!

While such a combination of features would be useful, I just think that they may be impractical together.

1. I'm not sure what you mean here, if a server alters an encrypted message it (should be, depending on the encryption scheme) is rendered unreadable. It's possible to cause the message to be unreceivable, but not to alter a message after encryption during transit. If you mean MITM because of poor key exchange, then that's a fundamental problem with key distribution not PGP in particular. The same problem (having to make sure key exchange is secure) would exist in any scheme.
2. Bittorrent users derive direct benefit from storing and sharing material. On public trackers you have the perpetuated availability of material the peer wanted in the first place. On a private tracker it maintains ration in addition to the previous. While there is also altruism (which would be all I can see this relying on), it's not infinite, and given only 100mb (or any finite quantity) per peer you will still eventually run out of space. The blockchain is useful to anyone who stores it because that's how you prove what has happened and already there a large number of "light" nodes that only store the block hashes and verify things separately as needed and don't store many blocks at all as well as several implementations (including the satoshi client) that are looking to prune the spent tx data to reduce the storage requirement (pruned data would be ~120 MB right now if memory serves). For larger file storage clients like Freenet (while file storage you can just think of files as binary messages (see: Usenet file sharing)) the incentive is that you store some extra data so others will save your extra data, though no strict parity is enforced and there is a certain reliance on altruism (and for this reason I don't think this sort of project is anywhere close to impossible, just harder). Since I brought up Usenet, there are basically superpeers that act as peers with each other and central servers for end users; end users normally pay for large data storage provided by central servers.
3. PoW does help, but the higher you make it the harder (relatively) it is for a normal user to send a message, the lower you set the bar the easier it is to spam. So while there is a tradeoff but it's definitely a solid place to start; allowing a user to establish how hard it is to send them messages might work. Rejecting messages from a node may be somewhat meaningless, you want to allow new users to be able to participate reasonably quickly and there's no way to select between a new user and a new identity for a malicious node. In addition to all that routing messages through the p2p portion means the physical source node might not be important, the identity of the sending node is all you can conserve on a practical level.
4. I was thinking this was along the lines of a PM system with optional twitter-like broadcasts, but if it's twitter and includes direct messaging a few things shift priority, but overall allowing any messaging TO an identity opens up all the same problems if it's the most common behaviour or not. Sounds kind of like most of the forums on Freenet though, everyone publishes their board posts to their personal broadcast stream and based on your direct trust level or indirect trust through WoT you choose whether or not to download their messages. The expensive part of creating a new identity is bootstrapping yourself into the network and announcing your presence so other people begin to download your messages to begin with. If you wanted to apply a similar mechanic at all you could start with a heavy PoW problem to announce your identity to other peers.
5. I'm used to seeing DHT as a lookup method, but you can easily tie that to looking up where to find/who to ask about a certain message certainly. I'd say the central servers archetype evolved before p2p was necessary or practical; if you wanted to share something you hosted it, if you wanted mail you sent it to the mail server (similar to the post office I suppose) and it was also easier to manage and there was no reason for it to go away. A lot of hosted platforms (especially closed source hosted sites like google, centralized mining pools, or anything requiring any central authority or a protected/hidden database to run) are either easier or only possible inside a centralized environment. Some things can be moved easily to a decentralized setup (file sharing from FTP to Bittorrent, central mining pools to P2Pool, etc) and some things are harder to move over (Anything abusable, anything that requires strong input checking, required hidden or protected databases). I'd also say that part of the reason webmail providers have so much information they don't need to have access to is that it's easier and established. There's no established default email or text encryption standard for transmission and the default is plaintext. If you could switch everything to encrypted transmission by default the webmail providers wouldn't have anything to begin with (other than sender/receiver and timing, which is also what the NSA was collecting on phone calls and is still powerful information and not eliminated in a p2p setup) but it's harder for end users, grandma doesn't care about PGP and grandma couldn't use PGP and wouldn't be able to use email if it were the default and the only option for her would likely be a webmail provider either receiving plaintext or decrypting it for her server side which means they have the contents anyway (you could do it client side, but then grandma has to carry her private key around and she can't do that either, also it's easier server side).

All told you've definitely addressed a lot of things, I'm having to insert a fair amount of qualifiers to things now, and a lot of problems can be considered corner cases (that I'd still personally like to see addressed in an active implementation, now is too soon for some of them before larger details are finalized). The biggest problem I still have is storage, I'm not sure the balance between indefinite holding and total potential storage is there yet, at some point it just runs out. Though I don't know that a two day window is enough for everyone either :-) If nothing else I can keep playing devil's advocate at least if you want to try to flesh things out more.
donator
Activity: 452
Merit: 252
tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"
newbie
Activity: 25
Merit: 0
Great post!
Yes it's all very generic, what makes my proposal reasonable (hopefully) is the combination of features. Once again, I want to build a secure messaging analogous to email with all its features but without any centralization and asymmetric encryption ingrained in the system.

1. You have to exchange keys with current PGP setup. Also you need to have SMTP and POP servers to actually process your messages. If they want to mess with your messages they can.
The proposal is to use public keys as receiver addresses, so no keys should be exchanged. And build a strict peer-to-peer system.
2. The problem with storage can be solved more or less as it's sometimes done with Bittorent private trackers - if you wanna use the system you have to give back, that is you have to store messages too. People store tones of multimedia files at their home PC's, probably storing 100 mb 1 GB of data is all right. Bitcoin block chain is 6 GB now.
Another way to approach this is the nodes who store the data for a fee, this also would be a way to monetize the system. For example heavy users would be required to pay a fee through bitcoin which will be redistributed between storing nodes. 
3. The way to approach the spam problem would be PoW, analogous to the way its made in Bitmessage. Also a peer would be allowed to block messages from certain nodes. in this case the messages will be dropped from the system
4. This is more like crypto twitter. Actually the spam problem won't be so drastic here since you need to know the public key of the node if you want to subscribe to its messaging stream, it can't message you directly.
5. Appropriate DHT setup is able to solve the problems of peers going offline, as it is proven in practice by various P2P systems. As for central servers - I believe that complete decentralization is the way to go, Internet is inherently decentralized system which certain parties always try to centralize.
There have to be countermeasures to this. Just imagine a sheer amount of private information any webmail provider has access to. I'm not sure that this is the ideal setup of human communication, when a third-party always has access to anything it wants. It creates a new hierarchy, which actually is not necessary, all communication CAN be peer-to-peer indeed.

Freenet is cool but it's more file sharing application. Also there won't be an explicit blockchain here, everything can be done through correct DHT setup.

once again thank you for very insightful comments!
Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email
1. This is a given for any private messaging system, it's also generally a more easily solved issue given existing systems like PGP or more generally with most modern asymmetric crypto.
2. An admirable trait, but to avoid storing it locally while keeping it accessible means that it still needs to be stored somewhere. Generally this is easiest when done by a central server or something of the sort, but to keep it decentralized means that someone else has to store their messages, your messages, as well as everyone elses. Not only that but since you want to be fault tolerant of a single peer leaving it has to be stored across multiple peers. All this storage also has to be pushed onto people that derive no inherent benefit from storing your unreadable messages. And how long is indefinitely? Until the account is inactive for a certain period (this also opens the can of worms that everyone has to know if the identity is active), until the account asks to be removed (if ever?), until someone runs out of storage space (highly random)? This is likely a large portion of the reason that bitmessage only keeps things for 2 days.
3. Spam is the bane of any messaging system, but to allow for number 1 anyone with your key must be able to message you. Generally publicly publishing your private key is the simplest way to allow others to communicate with you. To preserve secrecy of identity they have to be able to send it without disclosing too much information about who they are until you receive their message. All these together (and with number 2) mean that you're open to several different DDoS attacks (sending a large number of messages for you to decrypt and check if they're worth receiving, even if all you end up doing is running against a spam filter. Storing spam mail. Manually tuning a spam filter of some sort. Transmitting spam for yourself and others. And I would guess several others not mentioned here).
4. This generally sounds like a useful feature, but be care has to be taken with implementation to avoid possible abuses. Very possible though.
5. This just makes all the rest of it more difficult as you have to make sure other people are willing to volunteer the resources, requires certain tolerances to peers leaving the swarm, and means that you can't depend on a lot of the nice features a central server allows you (while avoiding the problems a central server creates of course).

All that said, you may want to see if an existing setup would work just as well, throwing the blockchain idea at everything doesn't always help anything at all. One existing setup that you may want to have a look at is Freenet (a P2P censorship resistant, data storage network where data can include text messages of course) and the various forum implementations therein (FMS, Freetalk, Frost, and the email-like plugin that I forget the name of). They're all fairly different but may help you skip past past issues they've already encountered and build upon areas they've had success in. Some of the things in particular to take away are tradeoffs in storage, availability, duration of messages, and privacy. Web of trust may also be a useful thing to look at, and is featured in most of the forums.

Naysaying aside, good luck if you go forward.
sr. member
Activity: 389
Merit: 250
Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email
1. This is a given for any private messaging system, it's also generally a more easily solved issue given existing systems like PGP or more generally with most modern asymmetric crypto.
2. An admirable trait, but to avoid storing it locally while keeping it accessible means that it still needs to be stored somewhere. Generally this is easiest when done by a central server or something of the sort, but to keep it decentralized means that someone else has to store their messages, your messages, as well as everyone elses. Not only that but since you want to be fault tolerant of a single peer leaving it has to be stored across multiple peers. All this storage also has to be pushed onto people that derive no inherent benefit from storing your unreadable messages. And how long is indefinitely? Until the account is inactive for a certain period (this also opens the can of worms that everyone has to know if the identity is active), until the account asks to be removed (if ever?), until someone runs out of storage space (highly random)? This is likely a large portion of the reason that bitmessage only keeps things for 2 days.
3. Spam is the bane of any messaging system, but to allow for number 1 anyone with your key must be able to message you. Generally publicly publishing your private key is the simplest way to allow others to communicate with you. To preserve secrecy of identity they have to be able to send it without disclosing too much information about who they are until you receive their message. All these together (and with number 2) mean that you're open to several different DDoS attacks (sending a large number of messages for you to decrypt and check if they're worth receiving, even if all you end up doing is running against a spam filter. Storing spam mail. Manually tuning a spam filter of some sort. Transmitting spam for yourself and others. And I would guess several others not mentioned here).
4. This generally sounds like a useful feature, but be care has to be taken with implementation to avoid possible abuses. Very possible though.
5. This just makes all the rest of it more difficult as you have to make sure other people are willing to volunteer the resources, requires certain tolerances to peers leaving the swarm, and means that you can't depend on a lot of the nice features a central server allows you (while avoiding the problems a central server creates of course).

All that said, you may want to see if an existing setup would work just as well, throwing the blockchain idea at everything doesn't always help anything at all. One existing setup that you may want to have a look at is Freenet (a P2P censorship resistant, data storage network where data can include text messages of course) and the various forum implementations therein (FMS, Freetalk, Frost, and the email-like plugin that I forget the name of). They're all fairly different but may help you skip past past issues they've already encountered and build upon areas they've had success in. Some of the things in particular to take away are tradeoffs in storage, availability, duration of messages, and privacy. Web of trust may also be a useful thing to look at, and is featured in most of the forums.

Naysaying aside, good luck if you go forward.
newbie
Activity: 25
Merit: 0
Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email
newbie
Activity: 25
Merit: 0
I think what is really cool with email is the fact that your messages are not deleted. Otherwise it would have been replaced with IM long ago Smiley
Email is extremely old-fashioned and ridden with virtually unsolvable issues, such as spam. But it still exists Smiley
legendary
Activity: 1764
Merit: 1000
All right I have to check on that, but anyway you can't store messages more than two days. So bitmessage is a great idea but in functionality it is somewhere closer to slow chat service than to email.

what if I told you that chatting is just a faster e-mail? Cheesy
newbie
Activity: 25
Merit: 0
All right I have to check on that, but anyway you can't store messages more than two days. So bitmessage is a great idea but in functionality it is somewhere closer to slow chat service than to email.
sr. member
Activity: 420
Merit: 250
Naturally Bitmessage is very promising, but my personal gripe with it that it doesn't provide the full email functionality, by deleting messages from block-chain within two days. Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option. In Bitcoin you have only transaction stored in the chain, in my opinion it corresponds to storing in the blockchain only the message sender and recepient address, but not the whole message.
I don't think you completely understand how Bitmessage works. It doesn't use a blockchain in any way. The nodes store everything they've seen in the past two days, and serve it when requested. Data older than 2 days is automatically deleted by default. However, the client still retains all of your messages and received broadcasts on local storage.
If your message isn't acknowledged by the receiving party within the 2 day window, it is resent. So it will be received even if they don't come online for a while.
Quote from: tradeaway
Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option.
If Bitmessage in it's current state becomes really big, you're completely right. But this is being planned for with the use of streams. This splits addresses into multiple networks for scalability.
newbie
Activity: 25
Merit: 0
I think you all folks agree that some sort of ubiquitous protected messaging solution which cannot be wiretapped by design is probably what the world needs right now. After recent events it becomes clear that this is not a matter of privacy per se, this is the matter of a better society structure, in which no group of people is able to control society, by having access to basically all information it wants.

Naturally there are tons of privacy tools right now, but really a few of completely decentralized. Decentralization is  crucial to system security on the basic design level.

Naturally Bitmessage is very promising, but my personal gripe with it that it doesn't provide the full email functionality, by deleting messages from block-chain within two days. Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option. In Bitcoin you have only transaction stored in the chain, in my opinion it corresponds to storing in the blockchain only the message sender and recepient address, but not the whole message.

Returning to the email functionality, it seems that one of the most solid points in favor of traditional email is that you store messages for indefinite period of time, and they can be fetched from POP server again if you erase them locally. In spite of all spam problems this is what keeps SMTP protocol afloat with no immediate demise in sight.

So the question arises, if you don't want to store the messages in block-chain, where do you want to store them?

I propose using the old and proven solution of distributed file storage used in Bittorent protocol. In this case blockchain would contain the sender and receiver respective addresses, and the message hash.  When a receiver wants to check the messages sent to her she extracts the hashes of the messages, and does a hash based search through the network. There are established algorithms for this  such as DHT in Bittorent (magnet links) or Kademlia.

So the process of message transfer would be the following:
- Receiver shares its public key
- Sender fetches it, encrypts a message, adds the address of the sender and puts the block containing both addresses and the message hash in the block chain.
- Clients responsible for storing messages in a certain hash range download the message (or its parts) from the sender and store the encypted message.
- When a receiver wants to read the message it obtains its hash from the chain, and finds the peers storing it. After that it downloads the message and decrypts it with its private key.

Naturally tons of technical details are dropped from this simple description, such as provision of permanent message storage, handling huge messages, coping with abusing peers and such.  But hopefully the idea is clear, you use block-chain as the message register, and distributed hash tables as a tool to store the message in a distributed way.  You obtain bullet-proof system able to store messages indefinitely, which cannot be tapped without access to the receiver private key.

Also additional functionality can be easily added, for example  if sender doesn't want to store the message indefinitely  it can be dropped from the system.  Also it's quite obvious that similar set-up can be used for distributed file storage. It's also possible to monetize the system by imposing a fee on large file transfers and their storage.

The main advantages of the approach are:
-Manageable blockchain
-Using cheap storage for using the message data, and using the "expensive" storage (blockchain) for storing only the message information.
-Web-interfaces can be easily constructed, analogous to traditional webmail.
-We get more or less usual email functionality without messing with PGP and key exchange.

We would like to start open-source project along these lines, and welcome everyone interested to join us.  Also any comments on the system design will be greatly appreciated.
Pages:
Jump to: