vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

In order to achieve good IO performance we had to use cache=’writeback’ for the instances and this will significantly increase the IO performance and bring it almost to host level performance, but in anycase much better compared with the old xen instances we had. Here is how you can enable writeback for an instance: stop the vm; edit the guestdomain and add cache=writeback in the driver section, save and start back the vm:
virsh --connect qemu:///system
stop guestdomain
edit guestdomain   <-- add cache='writeback' in the driver section
start guestdomain

Here is the how the disk part of my guest domain looks like after adding the cache writeback:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source file='/var/vms/vm2/ubuntu-kvm/tmphAUcOB.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>

In the process of debugging and searching for a fix for this issue, I’ve found out that it can also be useful to use elevator=noop as the default kernel io scheduler; this definitely helps, but not to the same extend as the cache writeback setting on the virtio disk. You can add elevator=noop to your kernel command line in your grub config, and I have this by default on all the instances.

Hopefully this will help you greatly improve IO performance for your KVM guests and will save you the time I’ve lost while trying to find a solution to this problem. Please feel free to share your experiences using the comment form bellow; also I’m curious if you have any other tips on how to improve this even more.

It looks like for some reason, xen-tools didn’t install the udev package. So in order to fix this issue, I had to connect (using the virtual console) to the xen machine:
xen console 1
and install udev:
apt-get install udev
strange enough the /dev/pts mount entry was present in /etc/fstab so all I had to do was to remount it with:
mount -a
(if you don’t have this entry make sure to add it in /etc/fstab:
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
and if the folder /dev/pts doesn’t exist create it first and after that mount -a).

This should fix the ssh problem and you should now be able to ssh into the xen machine. Next to see if this is fixed in xen-tools in Squeeze, and if not to file a bug for it.

Besides the obvious advantages of a virtual event, I would like to outline also some disadvantages: besides a rather poor sound quality for some speakers, the biggest minus for me was the inability to focus on a single talk uninterrupted. This because, being actually at work I had to deal with all the usual stuff that I have to do normally (the ones called emergencies). Being on a real event would imply other type of ‘noise’ but anyway .

One full week, with talks everyday for at least 8 hours is a big event. Being a virtual event it allowed everyone to choose what talks to attend. I can’t say there was anything spectacular that was announced, but more like everyone presenting their own existing products and views on the market.

I was a little disappointed by the keynote talk by Werner Vogels, CTO at Amazon.com called: “Ahead in the Cloud – The Power of Infrastructure as a Service” where he just presented the same old story that everyone I am sure heard so far about how amazon started building aws and how they ended up here, etc. Nothing about future plans, and he didn’t even take any questions. Contrary, his colleague from Amazon Web Services Jinesh Varia – Technology Evangelist (i love how this sounds) had a very interesting talk: “High Performance Compute Cloud” where he talked about the experiences in developing AWS so far and also on some of the cool upcoming developments they are working on (like monitoring, load balancing and auto-scaling). In case you missed this you can check it out online: http://cloudslam09.com/content/jinesh-varia-high-performance-compute-cloud-240.html

There were several good talk on private clouds and their implications, and also on cloud-hosted desktops (that has not really taken off yet and many are fighting on this segment to get an early advantage), like for ex:
Stephen Herrod: The Private Cloud: Enterprise-ready on and off premise.
Jeanne Morain: Universal Clients – enabling cloud computing for the Desktop – virtual evolution

In case you are interested in any of the talks they are all available online on the cloudslam site. (they are just adding them right now. If you see one missing, just check it out in a while and it should be there).

In conclusion this was a great event, definitely interesting and informative to attend. In case you missed it, you can now listen to any of the presentations as they become available online.

This is caused by the ‘new Xen console’ (xen now uses hvc0 for its console) and to fix it you have to add to your virtual machine xen configuration file one line: extra = “console=hvc0 xencons=tty”, restart the vm and it should be fine. In /etc/xen/<myvm>.cfg add this line:
extra = "console=hvc0 xencons=tty"

If you are using pygrub then in grub.conf on the domU add: console=hvc0 xencons=tty to the kernel line, like:
kernel /boot/vmlinuz-2.6.26-1-xen-amd64 ro root=/dev/sda1 console=hvc0 xencons=tty

Here are the xen packages lenny uses:

ii  libxenstore3.0                          3.2.1-2                    Xenstore communications library for Xen
ii  linux-image-2.6.26-1-xen-amd64          2.6.26-13                  Linux 2.6.26 image on AMD64, oldstyle Xen su
ii  linux-modules-2.6.26-1-xen-amd64        2.6.26-13                  Linux 2.6.26 modules on AMD64
ii  xen-hypervisor-3.2-1-amd64              3.2.1-2                    The Xen Hypervisor on AMD64
ii  xen-tools                               3.9-4                      Tools to manage Debian XEN virtual servers
ii  xen-utils-3.2-1                         3.2.1-2                    XEN administrative tools
ii  xen-utils-common                        3.2.0-2                    XEN administrative tools - common files
ii  xenstore-utils                          3.2.1-2                    Xenstore utilities for Xen

For full details: http://www.citrix.com/English/NE/news/news.asp?newsID=1687130

Manually create the /dev/fd link

This is a quick fix: we create the /dev/fd link manually:
ln -s /proc/self/fd /dev/fd
and bash completion will start working immediately.

Still, after the vm is restarted this link will be lost as /dev is not a persistent storage. We could of course add the link creation command to rc.local, but at this point I asked myself why udev was not creating that file automatically as it was present under /dev/.static/dev/fd?

Udev was missing

Then I realized that udev package was not installed, and this was the reason for the static devices not being created. I manually installed udev using:
aptitude install udev
and this was the solution for my problem. Hopefully you will find this post useful if you have the same problem, and it will save you the trouble to go into looking yourself for a solution for this.

If this is an isolated issue you can obviously just edit the udev rule generated and fix it (normally we want to have eth0 the existing interface and not something crazy like eth8, for ex.); this is found inside /etc/udev/rules.d/z25_persistent-net.rules

To completely disable this feature and no longer try to keep a static name for each device (in this case a static name for each mac address) we just have to remove the existing rules files (z25_persistent-net.rules) and also the generator rules that updates the rules file (z45_persistent-net-generator.rules):
rm -f /etc/udev/rules.d/z25_persistent-net.rules
rm -f /etc/udev/rules.d/z45_persistent-net-generator.rules

Note: since z45_persistent-net-generator.rules is just a link to the real file /etc/udev/persistent-net-generator.rules if you ever want to enable this back, you just have to recreate this link back and on the first boot the rules will be regenerated.