Recently I’ve worked on a project where we deployed a bunch KVM instances. Immediately we noticed horrible IO performance on all the guests instances. In this particular case the hosts and the guests were all Ubuntu 10.04 Lucid and were created with vmbuilder without any special settings using the ubuntu defaults. Here is a sample command similar to what we used to build the kvm images:

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

Read the rest of this entry »

A few days ago Opscode released a security fix for chef server 0.10.0 and 0.9.16 and this post will show how upgrade to chef-server 0.10.2. First start by backing up your data. Seriously. In the past I’ve had serious problems when performing similar upgrades (even a minor one like this that looks harmless), and even if now opscode are much better with this process it never hurts to be precautions. Since I use a rubygem install the next steps will focus on this type of installation; if you are using distribution or opscode packages this will not be very helpful as probably packages are not yet available for this upgrade; once they will replace the gem upgrade part with the deb/rpm upgrade and you should be set.

1. Stop all the chef related services

Here is a handy command that will stop all the possible chef server related services:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} stop
done

Read the rest of this entry »

After installing a clean Debian Lenny Xen system using xen-tools, I received this strange error when trying to connect using ssh to the machine:
PTY allocation request failed on channel 0
stdin: is not a tty

It looks like for some reason, xen-tools didn’t install the udev package. So in order to fix this issue, I had to connect (using the virtual console) to the xen machine:
xen console 1
and install udev:
apt-get install udev
strange enough the /dev/pts mount entry was present in /etc/fstab so all I had to do was to remount it with:
mount -a
(if you don’t have this entry make sure to add it in /etc/fstab:
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
and if the folder /dev/pts doesn’t exist create it first and after that mount -a).

This should fix the ssh problem and you should now be able to ssh into the xen machine. Next to see if this is fixed in xen-tools in Squeeze, and if not to file a bug for it.

Everyone knows and loves screen for running longtime scripts in the background without worrying that the ssh connection will drop and will have to run it again. Still, I have found myself many times in the situation where I started a process and needed to put it in the background and run something else on the console. Uff.. If only I started it with screen. But wait, there is hope. This quick tip will show how to put a process in the background and then start it back in foreground.

This works in bash and uses the ‘suspend‘ key (CTRL+Z) and the bg – background and fg – foreground commands. Let’s say we were running an intensive rsync command, and are wanted to check if we still have the available space on the disk without opening a new ssh session (yes, I know):
rsync -ar server:/source/ /destination/
^Z
Stopped

Let it run in the background:
bg
[1] rsync -ar server:/source/ /destination/ &

Now we can run some other commands like du:
du -h

We can see the background process with ps or jobs:
jobs
[1] Running rsync -ar server:/source/ /destination/

And finally we can bring it back to foreground with fg:
fg

Note: this works only on the running ssh/bash session and it will be closed once you exit. Logout should warn about open/running jobs and that they will be lost if exit.

Debian has a nice way to handle multiple java installations on a the same machine. Let’s say that for some reason you want to have sun-java 1.5 and also 1.6 installed on the server, we can easily configure the default one with the update-java-alternatives command (part of the java-common package). Here is how it can be used:

To see what versions of java we have installed on the system (from debian packages):
update-java-alternatives -l
java-1.5.0-sun 53 /usr/lib/jvm/java-1.5.0-sun
java-6-sun 63 /usr/lib/jvm/java-6-sun

We can see that the default version is 1.6 in my case (as it was the last installed):
java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

We can change the default version with: update-java-alternatives –jre -s <ver> , like:
update-java-alternatives --jre -s java-1.5.0-sun
and now the default is 1.5:
java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) Client VM (build 1.5.0_22-b03, mixed mode, sharing)

This is quite handy if you need to have multiple java versions installed, and need a quick way to change the default one (you can access any of them directly from their own path of course).

One of the great features of Amazon EC2 is the possibility to dynamically query and use instance specific metadata, or even custom data. This can be useful for various reasons, and the greatest advantage I’ve personally seen into this, is the possibility to allow the instance to have some information on how to configure itself when first booting (using chef or puppet, or some other configuration management tool).

The Amazon documentation explains how to get this information, basically just by using simple http get requests on the ip: 169.254.169.254, like for ex (for the metadata index):
curl http://169.254.169.254/latest/meta-data/
or for the custom data:
curl http://169.254.169.254/latest/user-data

Eucalyptus supports this great feature (starting with v1.4), but we obviously need to target a different ip to retrieve this information (as the amazon ip has nothing to do with our internal cloud ). We need to use the cloud controller IP for the request and the port it is bound (by default 8773 if you have not changed it). This will look like this (you need to run it from inside the actual instance): Read the rest of this entry »

s3sync is a great tool to synchronize local data with Amazon S3 for backups, or whatever other reasons you might want to put your data on S3. It is very simple to install (gem install s3sync) and use (s3sync -v -s -r –progress <source_dir> s3_bucket:<dir>); it runs very well and it can be easily scripted to do regular backups or even synchronize live data with S3. The only problem I found while using s3sync was that it can be very slow when uploading a lot of data (millions of files) to S3; this because the process is slow but also because it runs a single file at a time, and it doesn’t do several uploads in parallel. I would have loved for s3sync to do this out of the box, but unfortunately it doesn’t, but for my particular need I was able to do this by running more s3sync commands a the same time. It will not apply to your data (unless it is structured the same way as here; very unlikely), but it might give an idea on how you could do this your own data if it is structured in a feasible way.

Read the rest of this entry »

Many times you might want to fine tune the default permissions of the files created on a linux system. This is very simple and usually if you are using bash all you have to do is to define somewhere in the bash startup files (/etc/profile is a good place for this) a new value for umask like this:
umask 002
(this will allow by default group write permissions on the newly created files)

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

I needed to get a list of all the subdirectories that were owner by some other user than root under /var and their permissions/owner with full paths. My first thought was to use ls and something like this:
ls -dlR */
drwxr-xr-x  2 root root  4096 2009-06-05 06:25 backups/
drwxr-xr-x  8 root root  4096 2009-05-11 06:02 cache/
drwxr-xr-x  2 root root  4096 2009-05-06 04:49 ec2/
drwxr-xr-x 25 root root  4096 2009-05-25 14:55 lib/
...
will show the subdirectories just as I needed but only at one level. Using */*/ would show the next level, etc. This obviously is not a solution and unfortunately I had found no other way to do this with ls. Using:
ls -alR | grep ^d
drwxr-xr-x 15 root root  4096 2009-05-11 06:02 .
drwxr-xr-x 22 root root  4096 2009-06-03 15:02 ..
drwxr-xr-x  2 root root  4096 2009-06-05 06:25 backups
drwxr-xr-x  8 root root  4096 2009-05-11 06:02 cache
drwxr-xr-x  2 root root  4096 2009-05-06 04:49 ec2
drwxr-xr-x 25 root root  4096 2009-05-25 14:55 lib
....
works somehow, but since I don’t have the full paths this is useless.

Read the rest of this entry »

A while ago one of the major problems people faced to use Amazon EC2 into production environments was the dynamic state of the instances IPs. Every time one instance was started it was getting a new, dynamic IP. This has been addressed with the introduction of Amazon Elastic IP Addresses, but even when using this, the private IPs are still dynamic and most of the time people will want to communicate between several instances on the private allocated IPs and not on the public ones. This article will show how you can easily automate the process to update DNS hostnames for your EC2 instances, by adding to the AMI’s the logic for this. I will use for this a master DNS server running bind9, but this can be adapted to any other DNS server. Read the rest of this entry »