Recently I’ve worked on a project where we deployed a bunch KVM instances. Immediately we noticed horrible IO performance on all the guests instances. In this particular case the hosts and the guests were all Ubuntu 10.04 Lucid and were created with vmbuilder without any special settings using the ubuntu defaults. Here is a sample command similar to what we used to build the kvm images:

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

Read the rest of this entry »

After installing a clean Debian Lenny Xen system using xen-tools, I received this strange error when trying to connect using ssh to the machine:
PTY allocation request failed on channel 0
stdin: is not a tty

It looks like for some reason, xen-tools didn’t install the udev package. So in order to fix this issue, I had to connect (using the virtual console) to the xen machine:
xen console 1
and install udev:
apt-get install udev
strange enough the /dev/pts mount entry was present in /etc/fstab so all I had to do was to remount it with:
mount -a
(if you don’t have this entry make sure to add it in /etc/fstab:
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
and if the folder /dev/pts doesn’t exist create it first and after that mount -a).

This should fix the ssh problem and you should now be able to ssh into the xen machine. Next to see if this is fixed in xen-tools in Squeeze, and if not to file a bug for it.

Even though at some point it looked like debian lenny will not have full xen support (for the 2.6.26 amd64 kernel) in the end this was fixed and lenny supports fully Xen ever on amd64. Upgrading from 2.6.18 to 2.6.26 is very straightforward (though we were using xen-hypervisor 3.2-1 already) and the only problem noticed was that the console on the domU machines was no longer working: it was showing the output correctly, but you could not enter anything on the console.

This is caused by the ‘new Xen console’ (xen now uses hvc0 for its console) and to fix it you have to add to your virtual machine xen configuration file one line: extra = “console=hvc0 xencons=tty”, restart the vm and it should be fine. In /etc/xen/<myvm>.cfg add this line:
extra = "console=hvc0 xencons=tty"

Read the rest of this entry »

Citrix announced today that they will offer free licenses for their full XenServer virtualization solution. While Xen, the hypervisor itself was always open source, XenServer will not be released as open source, as it contains proprietary code, but will be free for anyone to download and use. I am sure that current XenServer5 clients that have paid more than $900 per server will be very ‘happy’ to hear this news . They will probably receive some special deals for Citrix Essentials, but still…

For full details: http://www.citrix.com/English/NE/news/news.asp?newsID=1687130

This post will show how to deal with the issue I had on a newly installed debian lenny xen virtual machine. I used xen-tools to create the vm using the deboostrap method, and all was fine. I installed the bash-completion package, as probably most of you, I can’t live without bash completion, but quickly found out that it was broken. Any attempt to perform a filelist completion was failing with this error:
vm11:~# tail -f /va<TAB>-bash: /dev/fd/62: No such file or directory
-bash: /dev/fd/60: No such file or directory
and this basically makes the bash completion useless. On a quick look I could easily see that the /dev/fd link was not there and that was the main cause of the problem. Still on one older lenny vm I had for a couple of months this was not happening (from what I can tell because it had an older version of the /etc/bash_completion file). There are several ways to fix this starting with the obvious one to downgrade /etc/bash_completion but I didn’t like that, so I looked for some other ways. Read the rest of this entry »