• High Scalability: Eucalyptus employs a software design in which scalability is achieved at two levels: front-end, transactional scalability and back-end, resource scalability. The new version of Eucalyptus provides increased back-end cluster scale improvements to support massive private and hybrid clouds.
• Support for iSCSI protocol for EBS volumes: Eucalyptus now supports Internet Small Computer System Interface (iSCSI) protocol for EBS volumes, which can make overlaying a Eucalyptus cloud on top of existing IT infrastructure even easier. This feature gives Eucalyptus users the flexibility to situate the EBS controller machine anywhere on the cloud, including outside the broadcast domain of the cloud nodes.
• KVM virtio support: Eucalyptus 2.0 supports KVM virtio, an efficient abstraction for hypervisors and a common set of I/O virtualization drivers. Users now have the flexibility to choose between emulated device drivers or direct kernel supported I/O devices via virtio for performance tuning.
• S3 versioning: Eucalyptus 2.0 extends its innate compatibility with AWS with support for S3 versioning. Now users can perform version control on the objects stored in Eucalyptus Walrus. Through a well-defined API, Eucalyptus users can retrieve specific versions of objects.

Also Eucalyptus introduced enhancements to its open.eucalyptus.com website to make submitting patches to the Eucalyptus open source code easier and more transparent. Seems like many of these came as the result of the latest controversy about eucalyptus and its open core model that resulted in the launch of the highly publicized OpenStack at OSCon. This demonstrates again (if it was needed) that it’s good to have several alternatives and this will only result in innovation and progress.

Until now. I mean, anyone with some experience can easily see that SofltLayer has already grown a lot and lowered their level of performance and support. Their tech people seem much less experienced and interested to help you out as they used to be, but this is not such a big deal because from how I see SolftLayer’s strength is their automation; they created a system designed to not need them so much. You can do everything yourself: from their control panel, or even from their api, and as long that works correctly all is good. You can order a server using api calls, you can cancel a server using the api, reboot it and you can even respond a ticket using api. Now with this merger I am am assuming that they are going to bring ThePlanet infrastructure to use SoftLayer automation; this is the only way that would make sense. SoftLayer is so much better than anything ThePlanet has, and there is no question in my mind this is what will happen. Still the concern remains and unfortunately for me, I don’t see the next ‘place’ for me to move if this will be needed. SolfLayer raised the bar so high, and other hosting companies don’t even dream to be close to that. SoftLayer was built by some of the original ThePlanet people (back in the days when it still was a great hosting company) and with their experience they knew exactly what they wanted to build. And they were right… They’ve done a great job.

I would be really interested if anyone knows or can recommend some other advanced hosting companies like SoftLayer? they need to have an api for everything and cloud computing solutions. I would love to try them. Let me know…

“Amazon VPC enables you to use your own isolated resources within the AWS cloud, and then connect those resources directly to your own datacenter using industry-standard encrypted IPsec VPN connections. With Amazon VPC, you can:

• Create a Virtual Private Cloud on AWS’s scalable infrastructure, and specify its private IP address range from any block you choose.
• Divide your VPC’s private IP address range into one or more subnets in a manner convenient for managing applications and services you run in your VPC.
• Bridge together your VPC and your IT infrastructure via an encrypted VPN connection.
• Add AWS resources, such as Amazon EC2 instances, to your VPC.
• Route traffic between your VPC and the Internet over the VPN connection so that it can be examined by your existing security and networking assets before heading to the public Internet.
• Extend your existing security and management policies within your IT infrastructure to your VPC as if they were running within your infrastructure.”

Besides the regular ec2 prices, we will have to pay for the VPN connection ($0.05 per VPN Connection-hour) and for the data transfer using the VPN tunnel ($0.10 per GB IN – and starting with $0.17 per GB OUT).

This is a great new service from Amazon that takes the cloud computing offerings to a new level. And this just when major competitors thought they were gaining on Amazon with their offerings, Amazon continues to innovate and launch great new services like this one .

The Amazon documentation explains how to get this information, basically just by using simple http get requests on the ip: 169.254.169.254, like for ex (for the metadata index):
curl http://169.254.169.254/latest/meta-data/
or for the custom data:
curl http://169.254.169.254/latest/user-data

Eucalyptus supports this great feature (starting with v1.4), but we obviously need to target a different ip to retrieve this information (as the amazon ip has nothing to do with our internal cloud ). We need to use the cloud controller IP for the request and the port it is bound (by default 8773 if you have not changed it). This will look like this (you need to run it from inside the actual instance):
curl http://<CC_IP>:8773/latest/meta-data
block-device-mapping/
security-groups
ami-manifest-path
ancestor-ami-ids
public-keys/
reservation-id
ramdisk-id
public-keys/0/
ami-launch-index
kernel-id
instance-type
local-hostname
local-ipv4
hostname
product-codes
public-ipv4
instance-id
public-hostname
ami-id
placement/

For the user data we can start the eucalyptus instance with:
ec2-run-instances <EMI> -d "myhostname" ...other params...
and later we can then retrieve that information from inside the instance using:
curl http://<CC_IP>:8773/latest/user-data
myhostname

While the documentation for Eucalyptus is getting better with every new version there are still missing parts like this one, and hopefully people looking for this information will find this article useful.

How to get the needed information (IPs, hostnames, etc.)

Amazon api provides us all the needed information. Any EC2 instance can get a lot of information about itself just by querying a web server using a REST-like API. Here is how we can get all the available metadata items:
curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

We have a direct interest in the public-ipv4 and local-ipv4 variables, but as we can see Amazon is providing many other useful information that can be used inside the instance for various purposes. For more details checkout the amazon docs.

Public and private IPs

This means that for getting the public and private ips we only have to make two calls like this:
curl http://169.254.169.254/latest/meta-data/local-ipv4and
curl http://169.254.169.254/latest/meta-data/public-ipv4

Hostnames

We need a way to identify from inside the instance what hostname this should be configured. There are probably several ways to do this, but the most common is to specify this when the instance is started using the –user-data option of the ec2-run-instances command (or the short form -d). This will pass the custom data and make it available to the instance.

You will probably want to customize this based on your needs. Myself I assumed that I will use the same domain for all instances and I need to pass only the hostname. Since I don’t need any other parameters to the machine I can just do this:
ec2-run-instances <AMI> -d "myhostname" ...other params...
If you use more user data, then you will probably use it like “hostname=myhostname <other_variables>”. (in this case you will need to update the script bellow like this: HOSTNAME=`echo $USER_DATA | cut -f 1 -d , | cut -f 2 -d =`)

Now making a http request like this will give us the hostname this instance should have:
curl http://169.254.169.254/latest/user-data

DNS Server configuration

Now that we have available inside the EC2 instance all the needed information, we need a way to allow the instance to update the DNS server. As I mentioned before, I will use for this a bind9 server, but this can by any other DNS server that allows this action to be scripted somehow (either using some api calls, or any way to use dynamic DNS update). For bind9 we will use the nsupdate utility to update the DNS server securely using the dnssec key mechanism.

Personally I don’t use the full domain for this, but delegate two subdomains (to the same nameservers) like this:
; ec2 zones:
ec2.domain.com.      NS      ns1.domain.com.
ec2.domain.com.      NS      ns2.domain.com.
ec2-int.domain.com.  NS      ns1.domain.com.
ec2-int.domain.com.  NS      ns2.domain.com.
and allow update access only to those zones. Of course if you prefer that you can give direct access to your full domain zone.

Generate a key using the dnssec-keygen utility like this:
dnssec-keygen -a HMAC-MD5 -b 512 -n USER user.domain.com.
and this will create two files like this:
Kuser.domain.com.+157+47950.key
Kuser.domain.com.+157+47950.private

Using the information from the public key add to your dns server configuration the key:

key user.domain.com. {
algorithm HMAC-MD5;
secret "xAw7F/axmVSxsZ+V4LAZnkeYObjOaJjbVKf21Zl4WhxtRHdlhqWSeCdd fIVR6MhC8LSQoim7NfkWD2j7WT5AHw==";
};

where secret is the value from the public key, that in my example looks like this:

cat Kuser.domain.com.+157+47950.key
user.domain.com. IN KEY 0 3 157 xAw7F/axmVSxsZ+V4LAZnkeYObjOaJjbVKf21Zl4WhxtRHdlhqWSeCdd fIVR6MhC8LSQoim7NfkWD2j7WT5AHw==

Finally we need to allow update access for the key:

zone "ec2.domain.com"
{
type master;
file "/etc/bind/zone/ec2.domain.com";
allow-update { key user.domain.com.; };
allow-query { any; };
};

zone "ec2-int.domain.com"
{
type master;
file "/etc/bind/zone/ec2-int.domain.com";
allow-update { key user.domain.com.; };
allow-query { any; };
};

Bind will need to be restarted after making these changes.

Using nsupdate to update the hostname

Next we will need to upload the key we created on the EC2 image (later we will save it inside the AMI once all runs well) and test to see if it is working properly.
cat<<EOF | /usr/bin/nsupdate -k Kuser.domain.com.+157+47950.private -v
server ns1.domain.com
zone ec2.domain.com
update delete test.ec2.domain.com A
update add test.ec2.domain.com 60 A <some_IP>
show
send
EOF

If this is working properly we can move on and put all this toghether in a script that will be running at the instance start time. If not, go back and see in your dns server logs if there are any issues why this is not working.

Finally automation

Now we just have to put all the pieces together and using a simple script like this will do the job:
ec2-hostname.sh:

#!/bin/bash

#you will need to have the key available in the instance in the same dir as this script
DNS_KEY=Kuser.domain.com.+157+47950.private
DOMAIN=domain.com

USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data`
HOSTNAME=`echo $USER_DATA`
#set also the hostname to the running instance
hostname $HOSTNAME.$DOMAIN

PUBIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/public-ipv4`
cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server ns1.$DOMAIN
zone ec2.$DOMAIN
update delete $HOSTNAME.ec2.$DOMAIN A
update add $HOSTNAME.ec2.$DOMAIN 60 A $PUBIP
send
EOF

LOCIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server ns1.$DOMAIN
zone ec2-int.$DOMAIN
update delete $HOSTNAME.ec2-int.$DOMAIN A
update add $HOSTNAME.ec2-int.$DOMAIN 60 A $LOCIP
send
EOF

You will probably want to run this at boot time either from rc.local or creating an initscript for it. I put all these EC2 related stuff under /usr/local/ec2 and in this case I just call it from rc.local with a line like this:
/usr/local/ec2/ ; sh ec2-hostname.sh

If all runs as you wanted you will probably want to save your AMI to include the script that will automatically update the dns hostname at instance boot time.

Hopefully you found this article interesting and it will be a starting point to create your own dns update script based on your needs.

Besides the obvious advantages of a virtual event, I would like to outline also some disadvantages: besides a rather poor sound quality for some speakers, the biggest minus for me was the inability to focus on a single talk uninterrupted. This because, being actually at work I had to deal with all the usual stuff that I have to do normally (the ones called emergencies). Being on a real event would imply other type of ‘noise’ but anyway .

One full week, with talks everyday for at least 8 hours is a big event. Being a virtual event it allowed everyone to choose what talks to attend. I can’t say there was anything spectacular that was announced, but more like everyone presenting their own existing products and views on the market.

I was a little disappointed by the keynote talk by Werner Vogels, CTO at Amazon.com called: “Ahead in the Cloud – The Power of Infrastructure as a Service” where he just presented the same old story that everyone I am sure heard so far about how amazon started building aws and how they ended up here, etc. Nothing about future plans, and he didn’t even take any questions. Contrary, his colleague from Amazon Web Services Jinesh Varia – Technology Evangelist (i love how this sounds) had a very interesting talk: “High Performance Compute Cloud” where he talked about the experiences in developing AWS so far and also on some of the cool upcoming developments they are working on (like monitoring, load balancing and auto-scaling). In case you missed this you can check it out online: http://cloudslam09.com/content/jinesh-varia-high-performance-compute-cloud-240.html

There were several good talk on private clouds and their implications, and also on cloud-hosted desktops (that has not really taken off yet and many are fighting on this segment to get an early advantage), like for ex:
Stephen Herrod: The Private Cloud: Enterprise-ready on and off premise.
Jeanne Morain: Universal Clients – enabling cloud computing for the Desktop – virtual evolution

In case you are interested in any of the talks they are all available online on the cloudslam site. (they are just adding them right now. If you see one missing, just check it out in a while and it should be there).

In conclusion this was a great event, definitely interesting and informative to attend. In case you missed it, you can now listen to any of the presentations as they become available online.

In my opinion here are the advantages of the newly released official Ubuntu images:

• officially support by Canonical (Eric has done a great job in patching and updating his images, but I am sure he has better things to do and let the Ubuntu team do this).
• custom kernels: for Intrepid 2.6.27 and Hardy 2.6.24 by having Amazon support in doing this (while alestic images were using the default Amazon Fedora kernel 2.6.21 image).
• apt mirrors in the ec2 cloud provided by Ubuntu: us.ec2.archive.ubuntu.com and eu.ec2.archive.ubuntu.com
• RightScale support for advanced integration with the RightScale platform for RightScale users.

Starting a default small instance Intrepid US image (check for the current AMI Ids):
ec2-run-instances ami-5059be39 -k my-keypair
ec2-describe-instances
ssh -i .ec2/id_rsa-my-keypair ubuntu@ec2-x-x-x-x.compute-1.amazonaws.com
sudo su -
Note: you have to login as ubuntu user and sudo as root.

Don’t forget to shutdown your instances when you are done, to avoid unneeded charges .

For more details: http://www.ubuntu.com/ec2

Amazon

“We’ve learned that some of our customers have needs which aren’t addressed by the spot pricing model. For example, some of them were looking for even lower prices, and were willing to make a commitment ahead of time in order to achieve this. Also, quite a few customers actually told us something even more interesting: they were interested in using EC2 but needed to make sure that we would have a substantial number of instances available to them at any time in order for them to use EC2 in a DR (Disaster Recovery) scenario. In a scenario like this, you can’t simply hope that your facility has sufficient capacity to accommodate your spot needs; you need to secure a firm resource commitment ahead of time.”

For example, a small ec2 instance that regularly costs $0.10 per hour, while if we commit for one year for a reserved instance paying a fee of $325, the hourly cost drops to $0.030, to an effective 24/7 cost usage of $0.067/ho. Obviously this is very interesting for people that run their sites full time on Amazon EC2 that can obtain a nice price reduction using this (check out the complete EC2 pricing info).
With the regular on demand prices, one small ec2 linux instance will cost (just the compute instance) approx 72$/mo. This means that if you run your site on a single instance full time, this will become interesting only if you intend to run it for more than 6 months. Anything less, is still better to run with the on demand pricing. Of course this is just talking about the pricing.

Also they added new API command line tools to support this new feature:

• ec2-describe-reserved-instance-offerings command lists the set of instance offerings that are available for purchase.
• ec2-purchase-reserved-instances-offering command makes the actual purchase of one or more reserved instances.
• ec2-describe-reserved-instances command displays a list of the instances that have been purchased.

In conclusion, this is an interesting move from Amazon, trying to get clients to commit on a long term basis, as opposed to the regular pay as you use model cloud computing became famous. This will be very interesting for people running 24/7 their sites on ec2, but for the rest that start instances as needed, in a true hourly based usage, this will not offer any advantage.
Just imagine Amazon started their cloud computing offerings with such prices; even if they are lower I am sure that people would have not seen this very different than the regular VPS offering where they have to pay a flat rate regardless of their usage. But now, if you put it like that, it doesn’t look so bad, right? and probably many startup owners will be quite happy on the price reduction they get by using the reserved instances.

Amazon

“With today’s launch, you can take advantage of the latest features for Amazon EC2 including multiple Availability Zones, Elastic IP addresses, and Amazon Elastic Block Store (Amazon EBS). In the near future, Amazon EC2 will also add support for Windows Server and SQL Server in the EU which is a new feature that was recently introduced on Amazon EC2 in the U.S.”

The new Amazon datacenter is located in Ireland, and from our tests the latency from a UK location is about 14ms (compared with 78ms for a US instance), and from the office here in Romania it averaged 66ms (compared with 125ms for a US instance). Respectively the EU instance has shown a ping latency of 118ms from a US location (compared with 46ms for a local US instance).

Prices

The prices for the EU resources are little more expensive compared with the US ones:
Standard (per instance hour consumed)
$0.11 for small instances
$0.44 for large instances
$0.88 for x-large instances

High CPU (per instance hour consumed)
$0.22 for medium instances
$0.88 for x-large instances

Data Transfer
$0.10 per GB – all data transfer in
$0.17 per GB – first 10 TB / month data transfer out
$0.13 per GB – next 40 TB / month data transfer out
$0.11 per GB – next 100TB
$0.10 per GB – over 150T

Elastic IP Addresses
$0.01 per non-attached Elastic IP address per complete hour
$0.00 per Elastic IP address remap – first 100 remaps / month
$0.10 per Elastic IP address remap – additional remap / month over 100

Our first EU EC2 instance

In order to use regions we need to use API Tools version 1.3-30349 and WSDL/API version 2008-12-01, or newer. To check your version run:
ec2-version
1.3-30349 2008-12-01

Let’s see what regions we have available:
ec2-describe-regions
REGION        us-east-1        us-east-1.ec2.amazonaws.com
REGION        eu-west-1        eu-west-1.ec2.amazonaws.com (where we can see the new EU region).
and the zones:
ec2-describe-availability-zones --region eu-west-1
AVAILABILITYZONE        eu-west-1a      available       eu-west-1
AVAILABILITYZONE        eu-west-1b      available       eu-west-1
By default the amazon api tools will run new instances in the US region, just like before. If we want to run our instances in EU we need to explicitly define this using:
export EC2_URL=https://eu-west-1.ec2.amazonaws.com
or using the –U command line option of ec2run
(ec2run <ami> -U https://eu-west-1.ec2.amazonaws.com)

Now we can start the instance as usual (keep in mind that ami’s from US are not automatically available to EU; check for their availability first; you might have to upload them to EU and the new api tools have a new command to help with this: ec2-migrate-bundle)

You will have to recreate your ssh keypair and security groups (region specific). Name the ssh keypairs differently if you use both zones to not have confusions:
ec2-add-keypair gsg-keypair-eu
and finally run the instance:

ec2-run-instances <ami-id> -k gsg-keypair-eu
And so we have our first running European EC2 instance   (that was used just for this article to test the latency from various locations).

This is a great news not only for European companies and developers but also for US ones that have a user base in Europe that will benefit from running local instances to achieve lower latency for their users.

Michael Crandell, CEO of RightScale, commented: “Over the next five years, spending on IT cloud services is expected to grow almost threefold, reaching $42 billion by 2012, according to industry research firm IDC. Specializing from its inception exclusively on cloud computing technology, RightScale is well positioned to maintain its leadership position as this industry matures. The company has two years of real-world cloud computing experience and thousands of live deployments. Over 200,000 servers have been successfully launched using the RightScale cloud management platform.”

Full Press Release: http://www.rightscale.com/news_events/press_releases/2008/08december2008.php