Creating an Electrum (pruning) Server Amazon EC2 instance and AMI.I chose to run an Electrum server on EC2 because I already use this service for other work. It's not the best bang-for-buck performance wise but it does have it's advantages. It's proven to be quite robust in my past use and I like that I only pay according to my use as I start/stop instances. It's easy to make snapshots and experiment with.
This step-by-step tutorial shows you how to create an Electrum Server AMI (Amazon Machine Image). An AMI is a ready-to-run virtual machine that you can start when desired, or leave running always. Using spot instances you accept some variability in uptime for a much lower running cost. At this time the monthly cost to run an AMI like this is roughtly $6.84 ($5.04 cpu + $1.80 storage). It's possible to have a small data transfer fee as well (if very busy).
I use timkay's handy cmd line tool here. EC2 operations can also be done easily from the AWS (Web) Console, point and click style, but I won't explain here which buttons and clicks are equivalent. Each command has some equivalent on the web interface.
1. If you want the timkay ec2 cmd line tools.cd ~
wget http://github.com/timkay/aws/raw/master/aws
nano .awssecret (put your EC2 access key and secret key here)
chmod 600 .awssecret
sudo perl aws --install (creates links for short names)
rm aws
ec2din --simple (test it works)
You can put --simple in a ~/.awsrc file if you always want simple listings.
2. Start an Ubuntu 12.04 server spot instance. I suggest using a high CPU instance (c1.medium) for compiling and initial blockchain download (saves much time, costs a few cents more), and then switching down to a regular small instance (m1.small) for economy during normal operation (t1.micro is just too small). The blockchain is on a separate data volume so can be detached and attached as needed. This is the process I follow below.
Before requesting a spot instance you'll want to be sure you have a security group with the right port ranges open. If you have one already you can specify the group. If not then this will create a group for electrum with suitable ports, and then use that group when startign the instance.
ec2addgrp electrum -d "allow electrum ports"
ec2auth electrum -P icmp -s 127.0.0.0/8
ec2auth electrum -P tcp -p 0-65535 -s 127.0.0.0/8
ec2auth electrum -P udp -p 0-65535 -s 127.0.0.0/8
ec2auth electrum -P tcp -p 22 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 8333 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 8081-8082 -s 0.0.0.0/0
ec2auth electrum -P tcp -p 50001-50002 -s 0.0.0.0/0
ec2dgrp |less -S (check your results)
To request a spot instance we specify AMI, max price, type, security group and login key name. This AMI code is for the Ubuntu 12.04 64bit virtual server image.
ec2rsi ami-9c78c0f5 -p 0.02 -t c1.medium -g electrum -k MyKey
ec2dsir --simple (check for status of spot request)
ec2din --simple (check if instance is running, takes a few minutes sometimes, and copy the public url output)
3. Login to the new instance.You can make login cmd simpler by editing your .ssh/config file to, eg.
Host *.compute-1.amazonaws.com
User ubuntu
IdentityFile ~/.ssh/MyKey.pem
Then login with only,
4. Prepare your tools - build.For Ubuntu 12.04 then you need to add this line to your /etc/apt/sources.list (for python-leveldb).
deb http://archive.ubuntu.com/ubuntu precise-backports main universe
sudo apt-get update
sudo apt-get install git htop build-essential libssl-dev libboost-all-dev libdb5.1++-dev python-leveldb python-setuptools
cd /usr/src
sudo git clone http://github.com/bitcoin/bitcoin
sudo git clone http://github.com/spesmilo/electrum-server
sudo easy_install jsonrpclib
sudo chown -R ubuntu: *
cd bitcoin
patch -p1 <../electrum-server/patch/patch
cd src
make -f makefile.unix USE_UPNP=-
sudo cp bitcoind /usr/bin/
sudo ln -s /usr/src/electrum-server/server.py /usr/bin/electrum
5. Create a data volume and attach to server. (to hold blockchain data)
At this writing the bitcoin/electrum data is about 6GB so I'm creating an EBS data volume of 10G. This will be big enough for a while anyway.
Again these cmds use the timkay tools; you can use the AWS Console, your choice.
ec2din |less -S (copy the zone id and instance id for your server)
ec2cvol --size 10 --zone (create volume, choose zone to match your instance)
ec2dvol |less -S (check for status, repeat until the new volume is available, copy the volume id given)
ec2attvol -i -d /dev/sdf (attach to your instance as device /dev/sdf)
sudo mkfs -t ext4 /dev/xvdf (for some odd reason /dev/sdf maps to /dev/xvdf)
mkdir /home/ubuntu/.bitcoin
sudo mount /dev/xvdf /home/ubuntu/.bitcoin
mkdir /home/ubuntu/.bitcoin/electrum_db
6. Configure and start bitcoin.Let it run until the blockchain is up to date (a long time!).
cd ~
sudo chown ubuntu: .bitcoin
cat << END > /home/ubuntu/.bitcoin/bitcoin.conf
rpcuser=electrum
rpcpassword=
END
bitcoind -daemon
You should see a messge that bitcoind is starting and you can monitor it with the getinfo cmd. The blockchain should be far along before you can start Electrum, but you can continue with SSL certs and config files while waiting.
bitcoind getinfo
7. Create an SSL certificate and key.There are 4 connection modes in Electrum client TCP,TCP/SSL,HTTP,HTTPS. A certificate is required for the server to start. You can use a self-signed one for TCP/SSL. For HTTPS to work you will need one that is recognized by the client. So either install your own root on the client or get one from a recognized authority. StartSSL.com provides free, but well recognized server certificates. I describe briefly how to create your own self-signed for TCP/SSL mode below.
openssl genrsa -des3 -out server.key 2048 (use any pwd as we will remove it after)
openssl req -new -key server.key -out server.csr (answer as suitable, common name is usually your domain name)
sudo openssl rsa -in server.key -out /etc/ssl/electrum.key (this step removes pwd)
sudo chown ubuntu: /etc/ssl/electrum.key (since electrum runs as ubuntu we need the key readable by ubuntu)
sudo chmod 600 /etc/ssl/electrum.key (make sure only owner can read)
(now create the certificate, good for 1 year)
sudo openssl x509 -req -days 365 -in server.csr -signkey server.key -out /etc/ssl/electrum.crt
rm server.key server.csr
8. Configure and start Electrum. Start out as private (no irc) for testing.
sudo su
cat << END > /etc/electrum.conf
[server]
host = (make sure the hostname resolves or will not bind)
native_port = 50000
stratum_tcp_port:50001
stratum_http_port:8081
ssl_certfile = /etc/ssl/electrum.crt
ssl_keyfile = /etc/ssl/electrum.key
password =
banner = Welcome to Electrum!
irc = no
cache = yes
backend = leveldb
[bitcoind]
host = localhost
port = 8332
user = electrum
password =
[leveldb]
path = /run/shm/electrum_db
END
cat << END >/usr/bin/start-electrum
#!/bin/bash
nohup /usr/bin/python -u /usr/bin/electrum &>> /var/log/electrum.log &
END
chmod +x /usr/bin/start-electrum
touch /var/log/electrum.log
chown ubuntu: /var/log/electrum.log
cat << END >/etc/logrotate.d/electrum
/var/log/electrum.log
/home/ubuntu/.bitcoin/debug.log
{
rotate 5
copytruncate
daily
missingok
notifempty
compress
delaycompress
sharedscripts
}
END
exit
Edit /etc/electrum.conf to have correct host name and passwords. You can place any "wall" or banner text in the file /etc/electrum.banner.
You can start electrum, but if it catches up with bitcoind block count it will shut down. I started electrum when bitcoind had reached about 170,000 blocks, and this worked for me. Electrum won't start listening for clients until it is fully up to date.
start-electrum
Check the log file for results. It should show initializing the database, and catching up blocks.
less /var/log/electrum.log
You can stop Electrum (but it doesn't work until it's caught up and starts listening).
electrum stop
9. Wait for Electrum to catch up (takes a quite while) and test with a client.
Bitcoin took around 7 hours for me to fully download the blockchain to 211,860. Electrum had reached around block 194,000 by that time. It took roughly another 2 hours for Electrum to fully catch up.
10. Configure both servers to start at boot time.We initialized electrum with it's data in shared memory (/run/shm/electrum_db) for speed, but we need to move this to the data volume for persistence. So first thing is to move it over, and edit /etc/electrum.conf for the new location.
electrum stop
mv /run/shm/electrum_db /home/ubuntu/.bitcoin/
sudo sed -i 's/\/run\/shm/\/home\/ubuntu\/.bitcoin/' /etc/electrum.conf
Create an init config so both start on boot. (be sure to change your hostname here)
sudo su
cat << END >/etc/init/electrum.conf
description "Start Electrum server"
start on runlevel [2345]
stop on runlevel [016]
exec su -c /usr/bin/init-electrum ubuntu
END
cat << END >/usr/bin/init-electrum
#!/bin/bash
hostname
hostname >/etc/hostname
bitcoind -daemon
until [ "`nc -z localhost 8332;echo $?`" == "0" ]; do sleep 5; done
start-electrum
END
chmod +x /usr/bin/init-electrum
cat << END >> /etc/fstab
/dev/xvdf /home/ubuntu/.bitcoin ext4 defaults 0 2
END
exit
Now stop electrum and bitcoind and try out the init manually to be sure it works.
electrum stop
bitcoind stop
sudo start electrum
htop
In htop you should see bitcoind running. The init script makes sure Electrum only starts after bitcoind is listening on rpc port 8332, so it may take a while (you can see the 5 second polling).
11. Create an AMI from this running instance. First, now is the time to make Electrum public if you want that for the AMI. Any changes to persist across all instances need to be made before creating the AMI.
sed -i 's/irc = no/irc = yes/' /etc/electrum.conf
It's possible to create an AMI from the command line but I prefer using the AWS console as I find it is less mistake prone. Open the AWS Console and go to Instance view. Select the correct instance. Choose Actions (above) and Create AMI Image. A panel of options pops up. Enter an AMI name (like Electrum) and description. Click "Create Image".
A snapshot will be created for each the root and data volumes we have. This process takes time and you can view progress on the "Snapshots" and "AMIs" views of the console by clicking the refresh button. During this process the running instance will reboot. After a while the snapshots will complete, and then the new AMI will change status to availble.
12. Start the new AMI as a small instance.Now you can try starting a working Electrum server from this AMI. This time we'll start a small instance as it costs less to run. Similar to before:
ec2rsi -p 0.01 -t m1.small -g electrum -k MyKey
ec2dsir --simple (check for status of spot request)
ec2din --simple (check if instance is running, takes a few minutes sometimes, and copy the public url output)
Login and check if electrum is up. Try a client to test availability.
Final steps and tweaks.The AMI created above is not ideal, but it's a starting point for tweaking to perfection. For one, it starts with a snapshot of the data volume and when stopped this new volume will be left existing. What we really need is for the AMI to start alone, attaching the data volume after boot. That way all updates and changes made to the data will persist on the original volume. Another issue is with DNS and IP addresses. This can be solved with an Elastic IP and associating it during the boot process. Finally, it would be nice if the hostname and IP could be given to the instance, when requesting the start up, allowing multiple instances and more flexibility. I have worked out these details but will save that for part two - Seeking Perfection, or upon request.
Please submit feedback or corrections below and I'll try to update as needed.