Topic: Node.js decrypt file and verify gpg file (Read 2339 times)

You may want to take a look at a pure javascript implementation of open pgp, the standard behind gpg. https://github.com/openpgpjs/openpgpjs

A rsa key is usually packaged using pkcs 1 which defines the serialization of both pub and secret keys.

I suppose you would need to dig through GPG's code to figure out how it is packaged. I will take a look at that.
Edit: Openssl cannot sign and encrypt it seems. Here is a starting point for figuring out how GPG does that: http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=blob;f=g10/sign.c;h=ed8ac7328a151ee9bd57f5b6ca41cb25e6330731;hb=refs/heads/STABLE-BRANCH-1-4#l954

I verified it incorrectly, that was the problem. BTW I am not using GPGTools since I don't have a mac. Rather I am just using GPG command line on Ubuntu, which should be the same.

I'm looking at it and trying to figure it out. In the meantime, this article should help: http://www.sysmic.org/dotclear/index.php?post/2010/03/24/Convert-keys-betweens-GnuPG%2C-OpenSsh-and-OpenSSL
That is unfortunate. I don't know node.js, so I can't help with implementation. I can however help with logic and conceptual stuff.

Well, that's one of the questions I was trying to get answered.  I assumed that by looking at someone's code that handled this, I'd gain some insights into how GPGTools was packaging all the info together. I suppose I could find the answers elsewhere, but I was hoping this would be a faster way to learn.


Do you have a private key in GPGTools?  You can't sign without a private key.  If you do have a private key, then why don't you think it is signed?


Great!  I've searched and can't seem to find a way to do this.  Every time I find any discussion where anyone else is asking anything similar the result is a bunch of self-declared "experts" telling them that openssl and GPG are "not the same thing" and that they shouldn't be doing what they are trying to do.  Show me how, and I'll be very happy.


Not a choice I get to make.  I've been informed that I'm to use node.js if at all possible.  I'll need to convince someone that it can't be done with node (and that it can be done with some other language) before I can choose a different language.

How does GPGTools package the sig, encrypted data, and the key? An example would help.
edit: when I use your command in the OP, it does not produce a signed file, only encrypted.

You should not need two different RSA key pairs. RSA keys are RSA keys, and it doesn't matter how they are formatted in the export, they are still RSA keys. You just need to figure out how to export them in the same format that can be imported to two different software. Openssl uses PEM format while PGP does its own thing. It should be possible to export the raw hex of the key and then convert that hex to the proper format before importing it into your software.

Any particular reason to use Node.js? What about Java or Python which are also supported by Lambda?

Ok, so here's my problem...

I'm trying to learn, and I'm pretty good at reading computer programs.  I can't find a good source of reading material or potential teacher, so I figured I'd have someone write me a program as a starting point, and once I got those basics in my mind, I'd have a much easier time figuring out what to ask, what to search for, and how to get answers for a point that I thought I needed to make with someone, but I'm starting to think that I was wrong in the first place.

I think I've got about 3 main issues that I'm trying to deal with, but I couldn't figure out where to ask the questions to get the right answers and searching with Google wasn't helping since I didn't even know enough to be able to ask the right questions.  I've also made some assumptions that may turn out to be false.

I'll lay out what I'm struggling with here since a conversation seems to have started (even though Marketplace/Services probably isn't the "appropriate" palce for such a conversation).

I'm working on a project.

I've been told that I'm required to receive multi-megabyte sized files that have been encrypted.  I will have some sort of RSA private key that I can use for decryption. The senders will all have received the RSA public key that they use for the encryption out of band ahead of time.  The senders will also each have some sort of RSA private key that they will use for a signature of some sort to prove that they are the sender of the encrypted data. They will provide me with the associated RSA public key out of band ahead of time.  The filename will indicate who the sender is so that I'll know which public key to use for signature verification.

Upon receiving the file I need to first verify the signature and if the signature isn't valid I can log this fact and toss the file in the trash.  If the signature is valid, I need to decrypt the file.

That all sounds nice and simple so far....

Here's where it starts to get a bit tricky.  Some senders want to use openssl command line for encrypting and signing the files.  Some senders want to use command line GPGtools for encrypting and signing the files. Some senders want to use some sort of java library (bouncycastle?) for encrypting and signing the files.  Some senders want to use some sort of node package for encrypting and signing the files.  I get to specify the symmetric cipher used to encrypt the data.

The signed, encrypted files will be dropped off in an Amazon Web Services (AWS) S3 (simple storage service) bucket.  The decrypted files will need to be stored in a different AWS S3 bucket.  I can use any AWS service I like to handle the verification and decryption, but unless absolutely necessary it is preferred that I not need to have a persistent elastic compute cloud (EC2) instance.  The current suggestion is to have the S3 trigger Lambda with node.js code to handle the signature verification and decryption.  It is also preferred that the signature, session key, and encrypted data NOT be three separate files.  Therefore, they all need to be transferred together in a single file.

The data that is being dropped off on S3 may also (instead) be pushed to a web API using https POST.  The web API would receive the data stream (which would contain the sig, encrypted session key, and encrypted data) and write the stream to S3 (which would then bring us right back to where we were when the sender just drops it off there themselves.

I don't need all of this written for me.  I can handle most of it. However, as far as I know, GPGTools is not available within Lambda.  I'm not sure yet whether or not openssl is. So, if I'm going to use node.js in Lambda to verify and decrypt, and if the sender is potentially going to use GPGTools to create the file, then I need help understanding is what that data stream is likely to look like and how I should parse it.

If I push back and say that if I'm going to do this then the senders will all need to be forced to use openssl command line, then is there a standard way of packaging the sig, encrypted session key, and encrypted data together?  Someone I'm working with is pushing for us to just declare our own way of storing this stuff together (base64 encode each of the 3 pieces, then concatenate them together in a specified order with a period as a delimiter (sig.payload.session_key).  I "feel" like there "ought to be" a standard way of packaging those three pieces of data together, and I don't want us to be re-inventing the wheel.  I've pointed out to the guy I'm working with that GPGTools already packages them all together, but neither he nor I can find a way to do it with openssl command line.  He says if I can show him a node program that can verify and decrypt a GPGTools file (without needing gpg installed), that he'll consider that perhaps we shouldn't be "rolling our own" way of packaging this data together.

In case it wasn't clear from that mess of words, the 3 problems that I "think" I have are:

• How can I verify and decrypt a file with node.js in Lambda that has been created with command line GPGTools?
• Is there a standard way that the sig, encrypted data, and encrypted session key should be packaged together when created with command line openssl
• Do I need to have completely different RSA key pairs for each of the methods that the senders might use (an openssl RSA key pair, and a GPGTools RSA key pair), and is there potential that there might be additional RSA key pairs that I might need beyond those two?

I think the program that was my "Y" in the XY Problem was essentially intended to answer the first two questions.  It would show me how to decrypt a GPGTools encrypted file without gpg being installed, and I thought it would demonstrate how GPGTools was packaging up the file (though now I'm not so sure).

Is there a reason why you need to use the openssh keyfiles (specifically) for gpg messages? If you can get by with using only pgp keys, then the issue is solved.

I'm glad to hear this, and if I can find a way to get the ssh RSA keys into gpg (or get the gpg RSA keys into ssh) I'll send over data that you can test with.

Unfortunately, I hadn't realized how difficult that was until I saw your post:


I tried encrypting the file, and then realized that I didn't know how to get the RSA key from gpg into the openssl (or openssh) format that I thought I needed.

Then I used ssh-keygen to create the keys, but couldn't figure out how to get them into gpg.  After a couple of hours with Google search engine I came to the realization that (as OnkelPaul has stated), "what I'm trying to do is a bit weird".  If I can figure this key thing out, I may still want the program, but if I can't find a way to provide the data that I said I'd provide, I'm not sure that the program will be useful to me at all.

Indirectly, yes.  But that might just be because I lack enough knowledge to know what I should be doing instead.


In my case its more about not knowing who I can have that chat with.  I need someone that is likely to know the answers to my questions, and who is willing to take the time to chat with me.  IRC, skype, email, telephone, meet for drinks, method of communication isn't important to me.  Learning how to navigate this mess is.


This I understand.  The core of my problem is a bit more complex than just automatically handling encrypted and signed messages, but perhaps it just feels more complex to me because I'm coming at it the wrong way. After a conversation with a more knowledgeable individual I might come to the realization that I've made this all more complicated in my mind than it needs to be.

At the moment, a more accurate analogy than "wash my fur but don't make me wet" might be:

"My fur is dirty, and I don't have any access to soap and water.  If I did, I'd easily clean it myself.  I have a vacuum cleaner, a hair brush, a leaf blower, a roll of tape, and a can of talcum powder. Can anyone help me figure out a way to use some or all of these to reduce the quantity of dirt my fur until I can gain access to soap and water?"


I agree with this wholeheartedly and have already put a lot of effort into avoiding that as much as I can.

I have a solution ready for you. PM me for details. OnkelPaul is right about openssl vs gnupg keys being difficult. I need to know more about your requirements to 100% tune this to your needs, but like I said, I have something working in PGP.

I'd guess that about 40%-50% of what makes cryptography so difficult to tackle is the sheer number of incomprehensible and incompatible formats that all try to solve the problem of allowing for future expansion, in case old algorithms get out of favor and new ones are required. I'd love to have a better grasp at how the same primitive data (key pairs, signatures, encrypted streams) are handled in the different frameworks, but I've got far too little time to learn all that stuff

That simplifies things a lot of course, as the number of cases to consider is greatly reduced!
Is the SSH/PGP mix part of your intended demonstration? Otherwise it might be easier for whoever implements a solution to stick with just one "world" - either the PGP-based stuff or the SSL stuff. Tools for both worlds are plentiful (and equally arcane on both sides.)


The SE article http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem notes that interactive chat (e.g. IRC) might be better suited. But IRC is not everybody's taste.
If the core of your problem is to automatically handle encrypted and signed messages, ruling out the well-tested solutions OpenSSL and GnuPG is somewhat like "wash my fur but don't make me wet". In cryptography and security, cooking up your own solutions is almost inevitably calling for trouble. For a demonstration it might suffice, but in production I'd rather use something well-tested.


Phew, that's good to know . I noticed that you had me on the list, and I decided not to argue it because it's your list and your rules, but I must admit I was a little sad about it. I still see your point even though I don't fully agree with it.

Onkel Paul

Yep, I'm learning this pretty quickly.


I've just today discovered this as well.  Unfortunately all of my cryptography knowledge is limited to what I've learned from being engaged with the bitcoin community here at bitcointalk.org.  I've got lots of questions and am finding it very difficult to figure out where/how to get answers.


I've given the command that would be used with gpg to encrypt.  It will just use whatever cipher is default, since it won't be specified. I can certainly specify one if that helps.


Since I am both the sender and the recipient for the purposes of this demonstration, I'm confident that the sender knows my SSH RSA public key.


Well that's disappointing, but I'm beginning to realize that what I have here is probably an XY problem.  Without a reliable source to learn about X, I've apparently invented a Y that is perhaps leading us all down the wrong path.


I didn't realize that it was weird when I created this post, but as I've been trying to research it a bit myself I'm slowly coming to the realization that it's weird (see XY problem).  Unfortunately I don't know where to turn to have a conversation about what I actually need to accomplish.

I figured I could pay someone to write what I thought was a simple program, and that I could then use the program as an example to figure out what it was doing and how to make use of the node packages used. I'd then have the knowledge I needed for the point I was trying to make with someone.


Nah, you're not on my ignore list.  I've come across enough well thought out posts from you that you've been kept off.

You should be aware that SSH RSA keys and PGP public/private key pairs are somewhat different beasts.
Typically, GPG keyrings are in .gnupg (secring.gpg and pubring.gpg), not in the .ssh directory.

The code that you want might involve a conversion between those two different key formats which seems to be a bit tricky. A Node.js program to do the job would also need to implement a plethora of symmetric ciphers since it is unspecified which of the available ciphers in gpg is goin to be used.
In addition, it is somewhat unlikely that the sender knows your SSH RSA public key which they'd need to encrypt a message to you.
Surely not a 2-3 hours thing.

That all said, I'm pretty much aware that
1) you probably already know that what you're trying to do is a bit weird, and you still want to do it
2) you'll ignore my posting anyway since it carries a signature ad

Onkel Paul

I've already done that. I need to make sure it will work with your key settings.

It should work to create your own gpg key and encrypt a test file for yourself using the same command except substitute the email for your own

One more question: is your id_rsa encrypted? That could be a complication.

Send me the sample input files (pastebin is fine). I've already got a prototype going.

Like I said, what I'm looking for isn't too complicated.  I'll provide an encrypted signed file and a couple of RSA keys, the node program just needs to decrypt and verify the signature on the one file one time. It doesn't even need to be OS-neutral (although if I get to choose the OS, I'd prefer that it at least work on OS X El Capitan).

My assumption was that if anyone was already familiar with the appropriate node packages that they'd be able to put something like this together in an hour or two.  I'd be happy to pay 0.1 BTC for it.  If it was going to take longer or be more complicated than my assumptions then I'd be willing to negotiate a fair price.

I can write a full *pure* node.js solution for you (OS-neutral, no binaries need to be installed). I've implemented this in the past. I can probably have it done in a few days if necessary. What are you offering? (PM is okay too.)

That might be ok.  I was really hoping for something actually written in javascript by a human an dnot something written in C or C++ and then converted to javascript. IF there are no other options though, I'd be willing to consider this.


No good.  gpg is not installed on the system that would be running this, therefore, it wouldn't be possible to spawn gpg in the background.


As long as I can install that node package from registry.npmjs.org, this would be an acceptable solution.  If I have to pull the C code from github and compile it into a node module myself, then this solution doesn't meet my needs.

Do you want the actual decryption and verification to be done in pure javascript?

I poked around a bit and found a few options depending upon what the point to be proven is:

https://github.com/manuels/unix-toolbox.js-gnupg which is a portion of the unix-toolbox.js project which utilizes Mozilla's emscripten project to compile C/C++ to pure JS for running in the browser and should be usable from nodejs I'd presume

https://www.npmjs.com/package/gpg which spawns the native gpg in the background

https://github.com/eschulte/node-gpg which utilizes native c code from the GnuPG Made Easy Library compiled directly into a node module for use from javascript