Sunday, May 25, 2008

Quick Start Guide with Amazon Elastic Compute Cloud (EC2) on Windows

I have been introduced to Amazon Elastic Compute Cloud (Amazon EC2) about 2 weeks ago, and I was quite impressed with it. What Amazon provide with EC2 is a virtually unlimited amount of virtual machines/servers for you to run any Linux based OS and install whatever application that you like in the virtual machine. Each virtual machine unit is called instances, and instances are saved as Amazon Machine Images (AMI).

I though it will be useful for the online community for me to put up a quick start guide to Amazon EC2. Let me know of any erros or problem that you encounter. Cheers...

AWS Account Setup


First, you need to go to Amazon Web Services site to create an AWS account and sign up to both the Amazon EC2 and S3 services.

Once you have completed the signing up process, you need to get your AWS access identifiers. Move your mouse over the Your Web Services Account button and select the View Access Key Identifiers as shown below.


Record both Your Access Key ID and Your Secret Access Key as shown below. Put these information somewhere safe where only you or an authorised person can access them. For example:
  • Your Access Key ID: 2B8WQAKHBMR2YBECYGGQ
  • Your Secret Access Key: K65RrZP8yrqz+PxCLNI+feDkaOiWFy93+df+63lCF

At the bottom of the thsi key identifiers page there is the
Your X.509 Certificate section. Click the Create New button to create a new X.509 certificate . Then, download the certificated and put it in a safe directory. In my case, it's in C:\ec2-keys directory.

You should end up with a PEM-encoded X509 certificate and a private key file named as per the examples below:
  • cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem
  • pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem
You also need your AWS account ID. Move your mouse over the and select Your Web Services Account and select Account Activity. At the top of this page, just under the Account Activity title, you should see a label named "Account Number", followed by a hyphenated number (something like 4952-1993-3132). Record this number and put it in a safe location.


SSH Client Setup (puTTY)

Next, you need to install an SSH client. Create a directory called putty in C:\Programs Files. Download the these PuTTY applications: PuTTY client, PuTTYgen and PSCP, and save them in the C:\Program Files\putty directory.

Also add PuTTY directory in the Windows Path variable. To do this, open the System Properties window, then select the Advanced tab as shown below.

Then in the System variables section, click on the Environment Variables button to open the window as shown below.

Scroll down to find Variable named Path and select it. Click Edit, and add the PuTTY directory at the end of the current Path value as shown below. Click OK and close the System Properties windows.



ElasticFox Setup


I'm using the ElasticFox Firefox plugin to manage my Amazon EC2 instances. Download and install ElasticFox from here, then restart Firefox.

Run the ElasticFox plugin. You should get a screen that looks like below.

Click on Credentials located on the top left corner of the ElasticFox interface. We need to put in our AWS identifiers. Enter any Account Name that you like, and then your AWS Access Key and AWS Secret Access Key. Click Add to save your details, and close the window.

Now, click on Account IDs and enter your account ID (the 12 digit number without the hyphens) and any Display Name you like. So, if your ID is 4952-1993-3132, the you should enter 495219933132. Click Add and close the window.

Next, we need to create an SSH key pair to be used when we perform SSH remote login to an AWS instance. In ElasticFox, select the Keypairs tab. Then click on the green Create a New Keypair button. Enter any keypair name you like and click OK. You'll be prompted to save the generated keypair. Give a name (*.pem file extension), and save the keypair to a location (in my case its C:\ec2-keys\foc2-keypair.pem). The newly created keypair name will appear on the screen as below.

The newly generated keypair need to be changed to a format that can be understand by puTTY. We will use puTTYgen to do this. Run puTTYgen (remember where you saved it before! In my case its C:\Program File\putty\puTTYgen.exe). Then load the key that you saved before. You will be informed when the keypair has been successfully loaded.

Click on the Save private key button to save the puTTY compatible key (*.ppk file extension). Choose Yes when it the puTTYgen warning appear. In my case, I saved it to the C:\ec2-keys\foc2-keypair.ppk. Close puTTYgen once the file has been saved.

Now, we need to tell ElasticFox where to find the key. Back to ElasticFox, click on Tools and change the SSH Key Template match the location where you saved the keypair file.

Next, we need to setup the permission for external access to our amazon instance. At this stage, I'm only going to open permission for port 80 (http server) nas port 22 (SSH client). On ElasticFox, select the Security Group tab.

In the Group Permissions section, click the green Tick button to grant a new permission. The following window should appear, and enter the first port which is port 80 as below.

Click add, and repeat the same procedure for port 22. The newly created permission should appear on the permission list once created.

Now that you've finished with the setup, we are going to try to run an instance.


Running an Instance

In ElasticFox, select the AMIs and Instances tab. In the Machine Images (AMIs) section, click on the blue Refresh button to list all the available public AMIs. The window should appear like below.

In this example, we're going to run a base install of Linux Ubuntu Hardy created by the excellent Alestic people. As shown below, enter hardy in the test box in the Machine Images (AMIs) section. This will list down all the available Ubuntu Hardy AMIs.
Select the latest Ubuntu Hardy AMI, in this example I am selecting AMI ID ami-6a57b203 (alestic/ubuntu-8.04-hardy-base-20080517.manifest.xml). Right click on the AMI, and select Launch instance(s) of this AMI. The following window will appear. Select your keypair in and click on Launch.

The launched instance will appear in the Your Instances section. Wait until the instance status changed to running. Keep refreshing the window (click the blue refresh button ) to update the status.

Once you instance is running, right click over your instance and select SSH to Public DNS Name. This is to connect to your instance via SSH using puTTY. If everything was setup properly, you should get the following screen.

CONGRATULATIONS!!! You have successfully run an instance and remote login to it via SSH. You're now free to do whatever you like to the instance. Just consider it as your own Linux PC that you can install any applications (Apache web server, MySQL etc).

If you need to create a webserver with PHP & mySQL (LAMP server) with an amazon instance, have a look at my LAMP server guide here.

And if you would like to set up a web server and mySQL cluster in Amazon EC2, have a look at my cluster guide here.

Savings Your Instance

First, we need to copy the PEM certificate and key files to the running instance. We will use PSCP to do this. Make sure that you know the location and the names of your
  • keypair file (eg. C:\ec2-keys\foc2-keypair.ppk)
  • AWS certificate file (eg. C:\ec2-keys\cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem)
  • AWS private key file (eg. C:\ec2-keys\pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem)
Also, get the Public DNS from ElasticFox in the Your Instances section. Also, you can copy the Public DNS by right clicking your running instance.

In my example, the Public DNS is ec2-67-202-32-74.compute-1.amazonaws.com

Open the windows command prompt (Start->Run->Open: "cmd"). Then enter the following commands:

pscp -i
<your keypair file> <your aws certificate file> <your aws private key file > root@<your public dns>:/mnt

In my example the command will be

pscp -i
C:\ec2-keys\foc2-keypair.ppk C:\ec2-keys\cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem C:\ec2-keys\pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem root@ec2-67-202-32-74.compute-1.amazonaws.com:/mnt

Note that I'm copying the files to the /mnt directory in the running instance.

Now connect again to the running instance using PuTTY and go to the /mnt directory. Both the certificate and private key file should be there.

Now, we need to create the currently running instance image. Enter the following command on the PuTTY client

ec2-bundle-vol -d /mnt -c <your aws certificate file> -k <your aws private key file> -u <your aws account number> -r i386 -p <your image name>

For example,

ec2-bundle-vol -d /mnt -c
cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -k pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 -r i386 -p my_hardy

It should now start creating the image as shown below. This may take a few minutes.


Once the image has been created, we are going to copy the images to Amazon S3 storage facilities. The command to do that is as below.

ec2-upload-bundle -b <s3 bucket name> -m /mnt/<your image name>.manifest.xml -a <access key id> -s <secret access key>

For example,

ec2-upload-bundle -b arope_ami -m /mnt/my_hardy.manifest.xml -a
2B8WQAKHBMR2YBECYGGQ -s K65RrZP8yrqz+PxCLNI+feDkaOiWFy93+df+63lCF

If you have an Access Denied error, it's possible that the bucket name that you enter is already exist. Try different bucket name.

Once the images has been copied successfully, we are going to use ElasticFox to register the AMI. To do this, in the Machine Images (AMIs) click on the green + button, a small window will prompt you to enter the AMI manifest path. The path entered should be in the following format: <s3 bucket name>/<your image name>.manifest.xml

Your new AMI should appear in the list now and you can run as many of these AMI instances as you like.

That's all, let me know for any errors and I'll put up some more guides as well. Watch this space.

21 comments:

Eric Hammond said...

When you are selecting an AMI, you seem to have your Elasticfox AMI panel too narrow to see the whole AMI manifest names.

There are a couple different classes of Ubuntu 8.04 Hardy AMIs in the alestic bucket. The "base" AMIs are small, clean, and boot very quickly. These are recommended.

By simply picking the last one you are probably picking a "desktop" AMI which is bloated with all the desktop software and takes a couple minutes to boot. It will also slow down rebundling the image because of the large size.

I recommend finding the right Ubuntu AMI ID on http://alestic.com and then paste the AMI ID into the Elasticfox AMI search box.

[Disclosure: I maintain these Ubuntu AMIs and http://alestic.com]

Nice guide by the way!

Hidayat S said...

Thanks Eric.. I appreciate your comment. I really love your base install AMIs. They are great. Keep up the good work. Cheers

Hidayat S said...

I've fixed the error as pointed by Eric. Let me know of any others. Thanks

Admin said...

Hi,

This is great. By the way, I had to specify x86_64 instead of i386_64 in the bundle.

Thanks,

Tom

Hidayat S said...

Tom, thanks for letting me know of the error. It supposed to be i386 for 32 bit instances (small instances) and x86_64 for 64 bit instances (large and extra large instances). I get caught up somewhere in the middle. Cheers

Dave said...

Thanks from me and my keyboard which saw sudden death approaching from my hands until I started following your instructions. I'm consolidating my knowledge, but will be back to look at some of your other guides very soon.
Dave and ...

Hidayat S said...

Thanks Dave, good luck and do let me know if you encounter any problems.

Anonymous said...

Great step by step guide for getting started on AWS EC2. Here are another visual tutorial for getting started on Amazon EC2 - http://www.searchblox.com/tutorial/sb_amazon_demo/sb_amazon_demo.htm

Shankar said...

Hi Hidayat,

great post and thanks a lot for this.. However I have a problem with setting up my elasticfox plugin.

I work for a company which has AWS account and I dont have access to the Key / Secret Key / Account Id.. I have the .pem files and I am able to launch / stop / ssh into the instances with just the .pem files and the keypair.

Is there a possibility to setup elasticfox with just the key files or do I need to get the account information?

thanks in advance

cheers
Shankar

Chintan said...

VERY KeWL!!!

I just spent past 8 hrs on a stupid Fedora core installing Apache n the stack.. this did it in matter of mins.. ok its 3:42aM EST.. i better go to sleep..finally!

UBUNTU R0X!

Keith said...

I have launched and fiddled with a bunch of instances but I couldn't figure out how to bundle and save them to S3. This is the first guide (and I've tried a few, let me tell you) that made it clear enough to help me be successful. Seeing my customized Ubuntu Linux server saving to S3 and then launching successfully through Elasticfox is a thing of beauty. Now I can stop pulling out my hair on this one. THANK YOU!!!!

Max Dama said...

Hidayat,

Your guide was excellent. I am now running ec2 without problems. There was one place that your guide gave me an error. I do not know enough about how everything works to know if it was a user error or a bug in the guide but here it is, on this command:
ec2-bundle-vol -d /mnt -c cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -k pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 -r i386 -p my_hardy

I had to use /mnt/ before the cert and pk filenames, so the corrected version would be this:
ec2-bundle-vol -d /mnt -c /mnt/cert-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -k /mnt/pk-HKZYKTAIG2ECMXYIBH3HXV4ZBZQ55CLO.pem -u 495219933132 -r i386 -p my_hardy\

Thanks again, I will probably post a link to your guide on my blog since I think EC2 is of interest to the algorithmic trading community.

Regards,
Max

Hidayat said...

Hi Max,

If you're in the /mnt directory, then you don't need to specify the /mnt/ for the pk and cert file.

Thanks.

Hidayat said...

Thanks for all you comments. I'm glad this guide helps you in some way. Cheers.

Owen said...

What a performance to get it running! In the end it worked, but there was a problem with the X.509 Certificate; when I tried to generate a new one, it said it would automatically wipe my AWS one. The which cert where list seems to imply you use only one, not both.

Jamie said...

Excellent guide! just what I was looking for. thanks!

Junaid said...

Hi, first I would like to thank you for the gr8 AMI provided for us to work with.. I am new to the Amazon AWS and also linux but I tried to create a duplicate AMI from the base AMI. My concern is that the elasticfox now displays my AMI after registering as Private But I want to know is this possible for any one to create instance from my AMI? does private mean I will be only able to create instances from it?

I would highly appreciate response from any of u being my Senior in the matter.

Regards
Junaid
mjunaiddar@gmail.com

Daniel said...

Excelente tutorial !! Gracias

Reaper said...

Great job on this.

An additional resource not to miss is the Elastic Fox User's Guide (PDF):

http://ec2-downloads.s3.amazonaws.com/elasticfox-owners-manual.pdf

andy said...

I always enjoy learning what other people think about Amazon Web Service and how they use it am wondering if you can check out my very own tool CloudBerry Explorer that helps to manage S3 on Windows . It is a freeware.

JKuria said...

Hey Hidayat, I just want to say a big thanks for your documentation. It has saved me what would have been several hours of figuring this stuff out myself.

Cheers,
J.

Now one more question. How do I navigate the file system? I realize the commands you've specified that have paths like /etc/... are succeeding but I can't seem to do "cd etc".