Topic: Data-stream encryption (Read 384 times)

I don't really care about the TCP packet size, just the chunk/stream size received for decryption which might not be the same size as what was sent
unless you hang around waiting to see if more data arrives a few seconds later.

This makes encrypting large volumes of data sent over the wire more difficult on top of the performance problem of eating up the CPU  and
frankly I don't have the answer yet.

While 65535 bytes is the theoretical maximum for a 'tcp packet', which by the way is stream based (why do you care about packet size when using TCP?),
lower layers are not able to process such big packets without errors.

In the end a single packet sent over a tcp connection is always lower than those 65535 byte.

The point is you do have to deal with it with a stream cipher, if you want a protocol that is in any way reliable. TCP checksums are not sufficient, and without chunking and authentication you won't know something is wrong until after the whole file is sent, and then the whole file must be sent again.

http://www.itprotoday.com/management-mobility/isoosi-ieee-8022-and-tcpip


But 2mb are still sent as 2mb blocks from sockets regardless of the trouble in keeping a thread open to send data in little block or encryption
and the lower levels deal with it most of the time and there is not much i can do about it.

14.4k modem   When i started we only had 1200 baud and being able to send faxes involved an upgrade to the modem and these
things were expensive I can tell you but still it was better than Prestel and it's teletext type graphics.

I'm not sure what you mean by ISO reference model. TCP has a 2 byte checksum which can easily be accidentally fooled. Malicious bridge nodes (in Tor terminology) can easily switch all kinds of bits that TCP cannot detect or they can just modify the checksum to match. That is why you need authentication.

https://en.wikipedia.org/wiki/Maximum_transmission_unit

"For example, a 1500-byte packet, the largest allowed by Ethernet at the network layer (and hence over most of the Internet), ties up a 14.4k modem for about one second. Large packets are also problematic in the presence of communications errors. If no forward error correction is used, corruption of a single bit in a packet requires that the entire packet be retransmitted, which can be very costly."

Now imagine you sent your 2MB image without chunking and you have the same problem as a TCP packet.

The lower levels of the ISO reference model deal with bit's being flipped, it happens all the time but all a socket can receive in one chuck is 65535 bytes
so really I am sending images and bigger files just like web-site pump out data and that works.


Yes I know my way around Tor, been an exit node myself before now plus hosted a onion site however it's
slow, has no redundancy to host sites if the server node is down plus a few other little problems like only having a few well known
exit nodes and they are all black-listed plus something I don't want to go into here.

Down the road I will be dealing with Whats-app type messages, file sharing, hidden sites but most of it's based on end to end encryption
using a public key generated for each node without needing any centralization however I don't really think this will be possible if it becomes
popular but at least I admit it up front.

One day people will laugh that they had emails addresses instead of a public key knowing that big brother is reading everything we say.

Sure it will, and it will work much better than attempting to encrypt it all in one chunk. The maximum packet size is around ~1500 bytes as it is. If you send a 2MB file and one bit gets flipped somewhere along the way, without chunking the entire thing will have to be resent. There is very little overhead to doing this.

It sounds like you want to do something similar to Tor, maybe you should investigate what it does?

I don't think people are quite picking up the point here or maybe I didn't put it across correctly but we are not talking about
encrypting small amounts of data like a 2k whats-app message but something more like 100k needed for the HTML on a web-page
and then six javascript files within the page followed up by say ten 2mb .PNG/Gif images files and a CSS file

Performance is critical here because if these requests are not serviced within 2-5 seconds then the browser will try sending the request again
using multi threaded requests so both the client and server nodes need to also be multithreaded also and the exit nodes need to start pumping the data
back across the network even before they have the whole reply from the web-site.

Things are made worse because often data is chunked which more or less means streamed and you have relay nodes possibly in the middle
plus having to service 100 requests all at the same time so buffering up the data to be encrypted in memory and keeping much state information
simply is not an option.

Data can be sent over TCP/IP as a 30K message but can be received as three 10k sections with a second delay between them all
but you still have to start feeding the web-browser as soon as you receive the first 10k or else it might timeout so I have no option
but to use on-the-fly stream encryption and it must perform like Greece lightning.

What i do know is that Microsoft AES encryption works nearly as fast as I can iterate over a byte array
using un-safe code from C# with C++ style pointers so I don't know if specialist hardware is being used to
get the performance for AES encryption but that's the type of speed that I need and using Maths.Mod or BigINT's just won't
give it to me

Microsoft's AES encryption memory leaks and I am not trusting back-door bill gates with any keys or data given that
I don't know whats being copied and uploaded back to the NSA/CIA servers all night long.





Yes got safe key exchange using Bitcoins Secp256k1 on a control channel and the data-streams have good encryption
on the HTTP Requests send to the server nodes but these requests are only between 400-2000 bytes in size so no trouble to encrypt.


8K chunks are no use when
sending 2mb images over the internet and often using relays, not going to work that's not.


Google is trying to use UDP instead of TCP/IP for web-pages with the Chrome browser but I am not doing anything apart from
DNS using UDP:53

I keep the control channel and a number (Six just now) of these bi-directional streams open per client so that a node sitting behind a firewall
can still act as a server but in general the number of streams needed to service a web-browser can reach as high as 30 steams at a time

satoshi was a Windows developer, the early Bitcoin releases were only released in Windows... so don't underestimate people that are good coders in Windows, as long as the project is open source and other developers can keep developing in Linux later on as well then it should be good. I recommend that people learn programing in Linux for projects of such nature tho.

Isn't AES used for the wallet.dat encryption? I would like more options for the encryption of the wallet.dat file, give us the ability to select cascaded encryption Veracrypt style and the option to mask the funds too so you need to enter the password to see the funds.. anyway this is off topic I will create a thread later.

To send encrypted data over a vulnerable protocol (like the internet) you need:

1) key exchange (diffie-hellman usually)
2) a cipher (like salsa20 + initialization vector per chunk)
3) data integrity checks/authentication (hmac / poly1305)

You generally need to use TCP and you need a small maximum message size (chunk) - around 8kb or so typically - otherwise you have to resend huge chunks in case of corruption. You can use UDP which it sounds like you want, but then you need to reinvent a lot of TCP on top of UDP.
You can't use some hardcoded key because then anyone can look at your software and decode all communications. You need a different IV per message or observers can easily determine your key. You need authentication of the data or you will not know if the peer sent bad info or if it was accidentally/maliciously changed in transit.

The NaCl and LibSodium libs implement most of this for you in "secretbox". It's also available in the Go "x/crypto" library. There might be a C# implementation, I don't know.

in my quest to find what i was looking for I took the code here, nice sweet and simple it is and it used maths.mod to encrypt data
and I got it up and running in seconds ready for a bit of benchmark testing on 1GB of data (1mb blocks).

https://www.codeproject.com/Articles/17703/Stream-Based-Encryption-for-NET

I stopped the program after about sixty seconds and it had only managed to encrypt/decrypt
500mb of data so i think you can have good encryption or fast performance but you cannot have both.

BigINT are useful for encryption but again I just cannot get the performance I want using this method.

My current plan is to semi encrypt the data, tweak the key and pass it over to Microsofts black box
encryption but I won't be using AES encryption because the .NET version leaks memory which you only get
to see if you call it half a million times or so which might sound a lot but a proxy server can reach these
type of number in little over an hour.

 

That sounds like how many machine code instruction cycles does it take to encode each bit which is
related to the speed of the CPU so you want a low number of cycles and a high clock rate to get any speed

https://github.com/TangibleCryptography/Secp256k1/blob/master/Secp256k1.Core/ECEncryption.cs

The ECPoint gets encoded with with a AES key and IV



Later these are used for the signature signing


So the hard coded value seem to be correct and I think I had looked in to this before so I just took it that the rest of the
code would work with bit-coin since it contained the Base58 and Hash160 code but somewhere along the way it seems
like it come off the rails

I am not sure what code you are looking at.
The ECPoint you have linked (https://github.com/TangibleCryptography/Secp256k1/blob/ee6ddaa59d3c61295bb24bcc903bb16fb528b933/Secp256k1.Core/ECPoint.cs) consists of this:



Feel free to directly link to the AES function.
But as i have already mentioned, AES is used for symmetric encryption.
The bitcoin protocol doesn't require anything to be encrypted.

While signing message/transaction is theoretically like encrypting (with private- and public key swapped), there is no AES (or anything comparable) used.

You also seem to be a little bit confusing with your term of 'key exchange'.

I'd suggest you read this: https://en.wikipedia.org/wiki/Key_exchange


Regarding the IV of (any) AES encryption:
The IV can be whatever you want. So it seems that someone just decided to use parameters from the secp256k1 as IV.

Read here for more information: https://en.wikipedia.org/wiki/Initialization_vector

---

What do you mean by 'recognize' ?
You should read my answer to your previous post more carefully:


Those are the values for the secp256k1 curve.

Look here: https://en.bitcoin.it/wiki/Secp256k1

Sorry my layman question ... but it came to my mind ...

stream ciphers encrypts bit by bit ... how does it relates to CPU clock ?

Thanks for the link and you gave me the right answer in the end after looking around and finding very little code that would run in windows let alone
in C# that would stream or was open source and would give me any performance let alone not being bloatware that falls back on BouncyCastle or being open source.

you see I can encrypt something in Dot-spyware-net in about five lines of code but i don't trust them so what i think I might do is to semi encrypt
the stream, chuck it to Microsoft's black box code after reversing the key or something and then reverse the process during decryption.

I need to do some banchmark testing but why re-invent the wheel when only a spoke is broken and I might even throw in some compression
whilst I am about it.

what does concern me is the open source Secp256k1 code that I picked up and was planing on using for key exchange because it's starting
to look tainted now and I do confess that I don't really understand the maths behind points on a wave and this bit of code is starting to look
suspect to me.


Does anyone recognize these hard coded strings ?

Indeed ... I am not in the academic field .. but yes sounds that the branch is huge ...

Lecture 3: Stream Ciphers, Random Numbers and the One Time Pad by Christof Paar
https://youtu.be/AELVJL0axRs

ps-> Christof Paar, textbook is awesome ... worth a try ..

Yes i am starting (I think) to get the picture and the code I got this impression from was from here
https://github.com/TangibleCryptography/Secp256k1

Seems to tick all the boxes if you care to look

See second post here https://bitcointalk.org/index.php?topic=2826752.msg%msg_id%

I think this code was hanging around in the forum from about 2014
https://bitcointalk.org/index.php?topic=574882.0;prev_next=prev

The ECPOINT in his code uses key exchange from Secp256k1 for AES key/iv as you can see so
I am not sure whats going on here so maybe you can cast some light on this and excuse me for
saying that Bitcoin used AES

I started to suspect something might be wrong (late in the day) when I noticed the name "Brainwallets"
and I would give you a merit for correcting me but I don't seem to get many to give away around here so
it will just have to be a kiss 

You are neglecting a whole branch of cryptography: https://en.wikipedia.org/wiki/Stream_cipher .

The shelf is reasonably well stocked, recent overview: https://en.wikipedia.org/wiki/eSTREAM .

Bitcoin does not use AES for signature. AES is used for symmetric encryption.
Bitcoins signature algorithm is the ECDSA (DSA on elliptic curve). And Secp256k1 refers to the curve which is used.

---

Besides AES, bitcoin also doesn't have any key exchange.
An key exchange is required when you have to actually exchange a symmetric encryption/decryption key.
Asymmetric encryption made that obsolete with the drawback of being significantly slower.
But the bitcoin protocol does not include any (encrypted) information exchange via the network.