Author

Topic: [CURE] Illustrated Protein Folding EC2 Setup Guide: 300k PPD per Server! (Read 4277 times)

legendary
Activity: 1713
Merit: 1029
Wow, if these 0.35/h prices are correct it seems to me like renting these is highly profitable? Has anyone tried using this?

The $0.35/hr is actually a few cents higher than I'm currently paying ($0.29/hr right now). It's fairly profitable currently, but also a great way to acquire CUR right now Smiley
legendary
Activity: 1713
Merit: 1029
Has anyone tried this on the g2.2xlarge instances? I tried and it looks like the Nvidia drivers aren't the correct ones. I don't know enough about linux to fix that myself, even after some googling. The g2.2xlarge spot instances are less than half the price of the c3.8xlarge ones, so I was just interested in seeing how they compared.


Good question!!! The GRID GPUs are fairly low performance for the price. LasersHurt did a test run of these and got really low PPD numbers, so c3.8xlarge or cc2.8xlarge with BigAdv is the way to go.
newbie
Activity: 15
Merit: 0
Has anyone tried this on the g2.2xlarge instances? I tried and it looks like the Nvidia drivers aren't the correct ones. I don't know enough about linux to fix that myself, even after some googling. The g2.2xlarge spot instances are less than half the price of the c3.8xlarge ones, so I was just interested in seeing how they compared.
full member
Activity: 322
Merit: 102
Wow, if these 0.35/h prices are correct it seems to me like renting these is highly profitable? Has anyone tried using this?
legendary
Activity: 1713
Merit: 1029
Added pictures. In response to the above, EC2 has a TON of servers, and it's very likely folders won't be competing for spot instances. This is more expensive per point than GPU folding, assuming you already have the hardware, of course.
o3u
sr. member
Activity: 393
Merit: 250
Money comes, money goes
this is pointless now, it could damage the whole folding operation, i thought the point was to submit more WU?

With this as people compete for EC2 instances, they'll keep killing each others... and no one will profit. I'd advise everyone to not use this as it could do more harm than good.

This can be avoided by not using spot, and using regular instances. Prices are higher but risk of instance being killed goes away. Either way, this will likely not be profitable in any form.

I'll stick to GPU, this is a waste of money,
sr. member
Activity: 282
Merit: 250
this is pointless now, it could damage the whole folding operation, i thought the point was to submit more WU?

With this as people compete for EC2 instances, they'll keep killing each others... and no one will profit. I'd advise everyone to not use this as it could do more harm than good.

This can be avoided by not using spot, and using regular instances. Prices are higher but risk of instance being killed goes away. Either way, this will likely not be profitable in any form.
o3u
sr. member
Activity: 393
Merit: 250
Money comes, money goes
Hello! While folding on GPU hardware is a great way to earn Folding@Home points for protein research, if you do not have GPUs, or want to do a lot of folding, EC2 can be an alternate option to rack up those folding points.

Make sure you sign up for a folding@home account and an account at cryptobullionpools.com. You won't get QRB (Quick Return Bonus) for BigAdv WUs until your account has submitted 10 WUs successfully. I recommend you use the google NACL extension to quickly get 10 WUs before you start your EC2 work if you are using a new folding@home account. Make sure your account at cryptobullionpools and at folding@home are IDENTICAL.

Note: there is an above-normal risk for using EC2 on Curecoin. Curecoin EC2 folding is most profitable with Bigadv WUs, which can take up to nearly 24 hours to complete. Unlike mining where shares are submitted multiple times per minute, a partially-completed WU is worthless. If your server is destroyed before a WU is finished, you get no credit for that WU. Due to the nature of EC2 spot pricing, your servers can be bought out from under you. If you were 80% done with a WU and it costed you $6 of computing power to get to that point, then that server is out-bid from you, you lost that $6. You were awarded no points by Stanford, and thus no CURE by the payout pool for folding.

How does EC2 pricing work?
EC2 spot pricing is based on an auction-style bid system. People put a maximum price they are willing to pay per hour for a server of a certain type. If the current spot price is lower than that, they get a server provisioned. The second the bid price exceeds the price they have set for the maximum price they will pay, the server is decomissioned and redeployed, losing any data you had on the system. As a result, you can put a higher bid price to have a higher chance of keeping that system through price increases. Make sure you would be actually willing to pay the price you put in, though, as you may end up paying that. You will never pay more per hour than your maximum bid price.

How does EC2 work?
EC2 allows you to pay for server space by the hour. You can rent a server, fold on it until it completes a bigadv WU and returns it, and then cancel the server, paying for only the hours that the server belonged to you for.

Why spot pricing?
Spot pricing is significantly cheaper than on-demand instances. Spot pricing for a c3.8xlarge (16 cores, 32 threads) is currently around $0.26/hr/server in Oregon. On-demand (guaranteed no shutdown) are $1.68/hr/server.

What is Paravirtualization? What is HVM? Which should I choose?
If you want maximum performance (an additional ~1 or 2%, nothing significant), go with Paravirtualization of Ubuntu 14.04. However, in the very likely chance that you want to create one server and then duplicate it, HVM is for you, as you can easily create and deploy images without having to mess with S3. HVM still performs extremely well.

Getting Started:
I will publish a more comprehensive guide tomorrow complete with images and long-winded explanations. However, for now, a bit of knowledge about EC2 is assumed. You can find tutorials for this kind of stuff all over the forum.

--> Decide how much you want to spend per day, in USD. Remember the above risk. Divide this number by 24 to get the hourly rate you wish to spend. Divide this number by the number of servers you want to run to see what your price per hour per server should be. Currently, in Oregon and North Virginia, a c3.8xlarge instance (the best one) costs around $0.26 per hour. However, setting the price somewhere in the $0.35 to $0.42 range will make it far less likely that a spike in price will cause you to lose your server, and thus your investment back to the start of the current WU.

--> Click on "Request Spot Instance", and choose Ubuntu 14.04. If you are unsure which option to pick, go with HVM.
--> Click next, sort by ECU, and then choose the c3.8xlarge option (which has 108 ECUs)
--> Next, enter the price you calculated above. Make sure it is not lower than the price you see listed in the above boxes. In most cases, you want to set it high enough that small spikes in price will not cause you to lose your server (s).  
--> If you need to, configure security and a private key file. Download this file, use PuTTYgen to convert it to a .ppk, then load it into PuTTY as a private key file for authentication. Google is your friend. Tomorrow, I will have pictures, maybe a video. Smiley

-->Finish, and request your spot instance(s). Wait for them to be fulfilled (1 minute to 10 minutes).
-->Go to your Instances page, and copy the IP address for each one into PuTTY, making sure your .ppk file is chosen for Auth options. The username is 'ubuntu'.
--> Once you SSH into your server:

Code:
sudo -i
apt-get update
apt-get install htop

mkdir /etc/fahclient


apt-get install nano

wget 1.curecoinmirror.com/config.xml -O /etc/fahclient/config.xml
nano config.xml

In config.xml, use the arrow keys to navigate down to fill out your username and passkey. Double-check this! Control   + X to exit nano, press 'y' and enter to save.


Code:

wget https://fah.stanford.edu/file-releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb
sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb

Now, you can monitor progress by typing:

Code:
nano /var/lib/fahclient/log.txt

Press Control + V to scroll down to the bottom, where you will find the current percentage.

If you want to turn your server off, finish your current WU, otherwise your account loses that WU:

Code:
nc 127.0.0.1 36330
finish

You can open the program 'htop' to monitor CPU usage.
this is pointless now, it could damage the whole folding operation, i thought the point was to submit more WU?

With this as people compete for EC2 instances, they'll keep killing each others... and no one will profit. I'd advise everyone to not use this as it could do more harm than good.
legendary
Activity: 1713
Merit: 1029
Hello! While folding on GPU hardware is a great way to earn Folding@Home points for protein research, if you do not have GPUs, or want to do a lot of folding, EC2 can be an alternate option to rack up those folding points.

For this tutorial, you need an amazon EC2 account, and a computer. This tutorial is done at a keyboard of a Windows computer. Linux and Mac OS X are extremely similar. This will be using the Oregon region. You can select other regions to test pricing. Each region requires the creation of a different private key.

Step 1:  
Log in to your EC2 account. Click on "Request Spot Instance".


Step 2:
Click "Select" for Ubuntu Server 14.04 LTS (PV). HVM will also work, but will perform about 1%-2% slower.


Step 3:
Choose the c3.8xlarge instance type (108 ECUs).


Step 4:
Enter the maximum per-hour price you are willing to pay. Make sure it is at least 25% greater than the above-shown "Current Price" options.


Step 5:
Click the "Launch" button.


Step 6:
Select "Create a new key pair" from the dropdown menu, then name your keypair, and click the "Download Key Pair" button. NOTE: if you have already set up one instance before, you can skip down to step 25, and just double-click on the saved instance.


Step 7:
After downloading the private key (.pem) file, click on "Request Spot Instance".


Step 8:
Make sure that EC2 has successfully created your Spot Instance request (A green check-mark in a green confirmation box will appear, as shown). Then click on "View Spot Requests".


Step 9:
Click on the "Instances" option, as shown below. At first, nothing will show up. Leave this window as is for now.


Step 10:
Download PuTTYGen from http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe, then open it up. Click on the "Load" button.


Step 11:
Change the file selector to "All Files (*.*)" to allow you to choose the .pem file you downloaded from Amazon. Then browse for and select the keyfile, and open it up.


Step 12:
You will get a message about the successful import. Click "Ok" to continue.


Step 13:
Inside of PuTTYGen, make sure the SSH-2 RSA radiobutton is selected (this is the default), and click on "Save Private Key".


Step 14:
You will get a warning prompt about saving the private key without a password. This is fine, click "Yes".


Step 15:
You will get a "Save As" prompt. Choose a place to save the keyfile, and name it something simple.


Step 16:
Download PuTTY from http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe and open it up. Then, click on the "Data" option under "Connection" on the left options pane.


Step 17:
On the "Auto-login username" box, enter "ubuntu".


Step 18:
Click on the + sign next to "SSH" under "Connection" in the left options pane.


Step 19:
Click on the word "Auth" (not the plus) under "SSH" under "Connection" in the left options pane.


Step 20:
Click on the "Browse..." button next to the box for "Private key file for authentication".


Step 21:
An "open file" dialog prompt will appear. Browse to and select the new keyfile (.ppk) that we generated with PuTTYGen. Click on the "Open" button.


Step 22:
Click on the text at the top labeled "Session" in the left options pane.


Step 23:
Type a name in the "Saved Sessions" box. Then, click the "Save" button.


Step 24:
Go back to the EC2 "Instances" page. You may need to click on the refresh button (either in the browser or on the page) in order to see your server. Note the IP.


Step 25:
Enter this IP address into PuTTY, under the "Host Name (or IP address)" box.


Step 26:
A new window will pop up. You may be asked about trusting a fingerprint, accept. Type into the window:
Code:
sudo -i



Step 27:
Now, type:
Code:
apt-get update



Step 28:
Now, type:
Code:
apt-get install htop



Step 29:
Now, type:
Code:
mkdir /etc/fahclient



Step 30:
Now, type:
Code:
wget 1.curecoinmirror.com/config.xml -O /etc/fahclient/config.xml

Code:
https://i.imgur.com/yT6Gk8e.png

Step 31:
Now, type:
Code:
nano /etc/fahclient/config.xml



Step 32:
Now, use the arrow keys to move through the file. Enter your passkey and username as shown below.


Step 33:
Now, type:
Code:
wget https://fah.stanford.edu/file-releases/public/release/fahclient/debian-testing-64bit/v7.4/fahclient_7.4.4_amd64.deb



Step 34:
Now, type:
Code:
sudo dpkg -i --force-depends fahclient_7.4.4_amd64.deb



Step 35:
Press the "Enter" key on your keyboard to accept for whether FAHClient should be automatically started.


Step 36:
Wait 30 seconds.
Now, type:
Code:
nano /var/lib/fahclient/log.txt



Step 37:
Press Control + v four times to scroll down the log. Get to where it shows your Project. If it is 8101, 8102, 8103, 8104, or 8105, you successfully got a BigAdv WU. Alternately, check that is says 0xa5 to the left of the colon. Press Control + X when you are done.

You are done setting up! However, to keep your folding@home account in good standing, always finish your WUs. If you want to shut down your EC2 server, type:
Code:
nc 127.0.0.1 36330
finish
exit

You can look in the log again, and scroll all of the way to the bottom to see the current percentage:
Code:
nano /var/lib/fahclient/log.txt

Again, use Control + X to exit.

On average, each percentage point takes 14 minutes. It takes less for faster projects such as 8104 (averaging around 12 minutes), and longer for slower projects like 8101 (17 minutes).
You can type:
Code:
htop
To see your current processor usage. When finished, all of the 32 cores will drop from full 100% utilization to near-zero. When you see this, it is safe to terminate your EC2 instance from the EC2 panel without losing any folding@home work.
Jump to: