First we need to identify the file that is causing this issue; and for this we will verify all the packed objects and look for the biggest ones:
git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -5
(and grab the revisions with the biggest files). Then find the name of the files in those revisions:
git rev-list --objects --all | grep <revision_id>

Next, remove the file from all revisions:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>'
rm -rf .git/refs/original/

Edit .git/packed-refs and remove/comment any external pack-refs. Without this the cleanup might not work. I my case I had refs/remotes/origin/master and some others branches.
vim .git/packed-refs

Finally repack and cleanup and remove those objects:
git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune

Hopefully these steps will help you completely remove those un-wanted files from your git history. Let me know if you have any problems after following these simple steps.

Note: if you want to test these steps here is how to quickly create a test repo:
# Make a small repo
mkdir test
cd test
git init
echo hi > there
git add there
git commit -m 'Small repo'
# Add a random 10M binary file
dd if=/dev/urandom of=testme.txt count=10 bs=1M
git add testme.txt
git commit -m 'Add big binary file'
# Remove the 10M binary file
git rm testme.txt
git commit -m 'Remove big binary file'

Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

So let’s see how we can use veewee. I’m assuming you already have vagrant installed (and virtualbox), but if you don’t please install them first. To install veewee we just have to install the veewee gem:
gem install veewee
once you installed veewee you can see a new task added to vagrant: basebox.

Here is the list of the templates we get out of the box once we install veewee:
vagrant basebox templates
The following templates are available:
vagrant basebox define '' 'archlinux-i686'
vagrant basebox define '' 'CentOS-4.8-i386'
vagrant basebox define '' 'CentOS-5.6-i386'
vagrant basebox define '' 'CentOS-5.6-i386-netboot'
vagrant basebox define '' 'Debian-6.0.1a-amd64-netboot'
vagrant basebox define '' 'Debian-6.0.1a-i386-netboot'
vagrant basebox define '' 'Fedora-14-amd64'
vagrant basebox define '' 'Fedora-14-amd64-netboot'
vagrant basebox define '' 'Fedora-14-i386'
vagrant basebox define '' 'Fedora-14-i386-netboot'
vagrant basebox define '' 'freebsd-8.2-experimental'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386-netboot'
vagrant basebox define '' 'gentoo-latest-i386-experimental'
vagrant basebox define '' 'opensuse-11.4-i386-experimental'
vagrant basebox define '' 'solaris-11-express-i386'
vagrant basebox define '' 'Sysrescuecd-2.0.0-experimental'
vagrant basebox define '' 'ubuntu-10.04.2-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.04.2-server-amd64'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-amd64'
vagrant basebox define '' 'ubuntu-10.10-server-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-i386'
vagrant basebox define '' 'ubuntu-10.10-server-i386-netboot'
vagrant basebox define '' 'ubuntu-11.04-server-amd64'
vagrant basebox define '' 'ubuntu-11.04-server-i386'
vagrant basebox define '' 'windows-2008R2-amd64-experimental'

This means that we can build a box based on any of the above templates. That’s awesome! Let’s say we want to build a debian squeeze box using veewee; we would have to run:
vagrant basebox define 'debian-60' 'Debian-6.0.1a-amd64-netboot'
and this will create a folder definitions/debian-60 with the following files (the content of the veewee template):
definition.rb
postinstall.sh
preseed.cfg
we can modify/tune any of those files based on our custom needs. The file definition.rb is the main definition of the template. Here you would define the memory size, disk size, iso file, etc. The content is very easy to understand, but you would normally not have to change many things here. preseed.cfg is just a standard preseed file where you would customize the actual install process (you could change here the partitions or their type, timezone setup, etc). And finally postinstall.sh that is a bash script that will run at the end of the installation process and it will install ruby, gems , chef and puppet and also the virtualbox guest additions (needed for shared folders).

If you have the iso already place it in ‘currentdir’/iso. If not, veewee will download it and place it in the appropriate folder before starting the install process:
vagrant basebox build 'debian-60'
this will start the installation and you can see all the steps it takes (the keystrokes as they are entered, etc.). This can take a while… Once it is done you can validate the build with:
vagrant basebox validate 'debian-60'
(this will run a few basic tests to see if it can connect to the vm as user vagrant, if chef and puppet were installed, if the shared folders are accessible, etc).

And finally you can export it as a vagrant box with:
vagrant basebox export 'debian-60'
and add it to vagrant:
vagrant box add 'debian-60' debian-60.box
and now you can use it in vagrant with:
vagrant init 'debian-60'

That’s it. Very simple and now we have our own box built from scratch. As a side note, I found this very useful to test and troubleshoot preseed configurations . As you can see there are plenty of templates available in veewee but if you create a new one please consider to share it with others and send it to Patrick on github. I’m sure he will be happy to include it in newer versions of veewee. And if you found this useful don’t forget to thank Patrick for his great work on this awesome tool.

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

In order to achieve good IO performance we had to use cache=’writeback’ for the instances and this will significantly increase the IO performance and bring it almost to host level performance, but in anycase much better compared with the old xen instances we had. Here is how you can enable writeback for an instance: stop the vm; edit the guestdomain and add cache=writeback in the driver section, save and start back the vm:
virsh --connect qemu:///system
stop guestdomain
edit guestdomain   <-- add cache='writeback' in the driver section
start guestdomain

Here is the how the disk part of my guest domain looks like after adding the cache writeback:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source file='/var/vms/vm2/ubuntu-kvm/tmphAUcOB.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>

In the process of debugging and searching for a fix for this issue, I’ve found out that it can also be useful to use elevator=noop as the default kernel io scheduler; this definitely helps, but not to the same extend as the cache writeback setting on the virtio disk. You can add elevator=noop to your kernel command line in your grub config, and I have this by default on all the instances.

Hopefully this will help you greatly improve IO performance for your KVM guests and will save you the time I’ve lost while trying to find a solution to this problem. Please feel free to share your experiences using the comment form bellow; also I’m curious if you have any other tips on how to improve this even more.

1. Stop all the chef related services

Here is a handy command that will stop all the possible chef server related services:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} stop
done

2. Upgrade the chef-server gems

Simply run:
sudo gem update chef chef-server --no-ri --no-rdoc
and this should upgrade all the other gems it needs to. A sample output will look like this:
gem update chef chef-server --no-ri --no-rdoc
Updating installed gems
Updating chef
Successfully installed chef-0.10.2
Updating chef-expander
Successfully installed chef-expander-0.10.2
Updating chef-server
Successfully installed chef-server-api-0.10.2
Successfully installed chef-server-webui-0.10.2
Successfully installed chef-solr-0.10.2
Successfully installed chef-server-0.10.2
Gems updated: chef, chef-expander, chef-server-api, chef-server-webui, chef-solr, chef-server

Optional: if you want you can cleanup the system from old, unused gems with:
sudo gem cleanup

3. Start back the chef server services

Again in a single command, now to start them:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} start
done

That’s it, now you should be running the latest and greatest chef server version 0.10.2.

It looks like for some reason, xen-tools didn’t install the udev package. So in order to fix this issue, I had to connect (using the virtual console) to the xen machine:
xen console 1
and install udev:
apt-get install udev
strange enough the /dev/pts mount entry was present in /etc/fstab so all I had to do was to remount it with:
mount -a
(if you don’t have this entry make sure to add it in /etc/fstab:
devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=620 0 0
and if the folder /dev/pts doesn’t exist create it first and after that mount -a).

This should fix the ssh problem and you should now be able to ssh into the xen machine. Next to see if this is fixed in xen-tools in Squeeze, and if not to file a bug for it.

1. Update apt sources.list

The first thing we will do (after the backup of course) is to edit the /etc/apt/sources.list file and replace “lenny” with “squeeze“. Originally, this might look like this (for a system using the main US mirrors; your file might use a different local one):

deb http://ftp.us.debian.org/debian/ lenny main contrib non-free
deb-src http://ftp.us.debian.org/debian/ lenny main contrib non-free

deb http://security.debian.org/ etch/updates lenny contrib non-free

after replacing lenny with squeeze the file will look like this:

deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free
deb-src http://ftp.us.debian.org/debian/ squeeze main contrib non-free

deb http://security.debian.org/ squeeze/updates main contrib non-free

2. Perform the system upgrade

After updating the sources file, you will have to refresh the indexes with:
aptitude update

Next let’s manually upgrade the core apt packages (this will pull in some extra dependencies, and this is perfectly fine):
aptitude install apt dpkg aptitude

And finally we will perform the bulk of package upgrades with:
aptitude safe-upgrade
this will take a while depending on what packages you have installed (that will need to be upgraded) and on your internet connection speed.

Here are some changes that you might want to pay special attention during the upgrade:
- dash: squeeze uses dash instead of /bin/sh as the default system shell. This is the recommended way and it is supposed to be faster and improve the system overall performance. I recommend to use it and accept this change.
- grub2: is the default in squeeze and the upgrade will recommend to upgrade from your existing grub. This is a major change and for this reason initially the boot loader will be chainloaded in the existing menu.lst to verify it works fine. If you want to upgrade I would recommend to do the chainload and test it with at least one reboot before removing grub legacy.
- sysv-rc: to improve the boot process the upgrade will offer to move to dependency-based sequencing. This is irreversible but highly recommended. I had no problem with it and don’t see why anyone would not want this.
- UUIDs: the installer will recommend to switch from regular disk devices (aka /dev/sda*) to disk IDs (aka UUID=c6ecae74-6754-4ad0-986d-98dd9cbfd293) in various configuration places like /etc/fstab, /boot/grub/menu.lst. I personally don’t like that, but didn’t want to take the risk for the device to change its name with the new kernel and went with the change. If you do the same pls. doublecheck the changed files before rebooting.

Now it is time to reboot your system for the first time into squeeze. Fingers crossed and in a few minutes you will be running squeeze (as always an out of band impi console is handy when doing such upgrades)

3. Complete the upgrade

If you upgraded to grub2 and want to remove the chainloaded grub-legacy you can remove it completely from the system with:
upgrade-from-grub-legacy
and the output will look like this:
0
Installation finished. No error reported.
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-5-amd64
Found initrd image: /boot/initrd.img-2.6.32-5-amd64
Found linux image: /boot/vmlinuz-2.6.26-2-amd64
Found initrd image: /boot/initrd.img-2.6.26-2-amd64
done

You can now run:
aptitude full-upgrade
and this will complete some extra packages that were not straitforward and not seen as safe by the normal upgrade process (there might be none, depending on the state of your existing system). Evaluate them before moving forward. And finally once you are done reboot once more (to see that grub2 is working fine by itself) and you will be running the latest uptodate squeeze system.

4. (Optional) Convert your ext3 filesystems to ext4

This step is optional and if you are happy with the ext3 filesystem then you can safely skip it. Myself, I’ve done this most of the time so I thought it might be useful to add it here for anyone else interested to do the same.

First doublecheck if you are running the 2.6.32 squeeze kernel:
uname -a
Linux srv01 2.6.32-5-amd64 #1 SMP Wed Jan 12 03:40:32 UTC 2011 x86_64 GNU/Linux

Older than 2.6.28 kernels (like lenny 2.6.26 for example) don’t have native support for ext4, so this will fail for an older kernel. Because ext4 is backwards compatible with ext3, all we have to do is to change the mount definitions to ext4 in fstab:
vim /etc/fstab
and change the filesystem from ext3 to ext4 for any devices you might have. Make a note on the devices you change, and reboot the machine. After the reboot the machine will use the ext4 driver for the old filesystem even though it doesn’t take full advantage of the ext4 capabilities. To complete this and enable the extra ext4 features we will need to run for each filesystem:
tune2fs -O extents,uninit_bg,dir_index <device>
like, for example for /dev/sda1:
tune2fs -O extents,uninit_bg,dir_index /dev/sda1
this will only be activated when the filesystem is unmounted and we can achieve this with yet another reboot (especially for the root filesystem), and the system will auto fsck them (as the last command marked the filesystem as dirty) and perform the upgrade. I would highly recommend for this to have a remote console on the system as in most of the cases the auto fsck will fail and you will have to run it manually from the console.

Hopefully this howto will help you upgrade your debian system to squeeze with ext4. If you have encountered anything special during the upgrade feel free to share it with others using the comment box bellow.

This week the Debian project released the 6th update to its stable release Lenny, 5.0.6. All recent security updates have been added, as well as some other fixes. The linux-2.6 package was also updated for increased hardware support.

Backports service is now official

I was very happy to hear that the debian backports project is now an official debian project. I always used (and liked) the backports.org repository to easily bring in updated software to the stable release. Now, after it become an official project and not just a fun project of three developers will hopefully be even better and have more software added into backports much faster. Don’t forget to change your apt sources config to point to backports.debian.org (old backports.org mirror will still work for a while).
deb http://backports.debian.org/debian-backports lenny-backports main contrib non-free

Debian growth over time

Also on some unrelated news Romain Francoise published some interesting stats on the growth of the Debian archive over time:

• woody (2002): 8273 packages
• sarge (2005): 15195 packages (+83.7%)
• etch (2007): 18043 packages (+18.7%)
• lenny (2009): 22277 packages (+23.5%)
• squeeze (2010?): 28870 packages (+29.6%)

Wow… now that is really impressive.

Squeeze

Just like all the previous releases, this is another character from Toy Story – wheezy – a rubber squeeze toy penguin with a red bow tie (that appears only in the 2nd movie). This will be the first character selected as a Debian version name which has not appeared in all the movies.

Source: http://lists.debian.org/debian-devel-announce/2010/09/msg00000.html

This works in bash and uses the ‘suspend‘ key (CTRL+Z) and the bg – background and fg – foreground commands. Let’s say we were running an intensive rsync command, and are wanted to check if we still have the available space on the disk without opening a new ssh session (yes, I know):
rsync -ar server:/source/ /destination/
^Z
Stopped

Let it run in the background:
bg
[1] rsync -ar server:/source/ /destination/ &

Now we can run some other commands like du:
du -h

We can see the background process with ps or jobs:
jobs
[1] Running rsync -ar server:/source/ /destination/

And finally we can bring it back to foreground with fg:
fg

Note: this works only on the running ssh/bash session and it will be closed once you exit. Logout should warn about open/running jobs and that they will be lost if exit.