Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

In order to achieve good IO performance we had to use cache=’writeback’ for the instances and this will significantly increase the IO performance and bring it almost to host level performance, but in anycase much better compared with the old xen instances we had. Here is how you can enable writeback for an instance: stop the vm; edit the guestdomain and add cache=writeback in the driver section, save and start back the vm:
virsh --connect qemu:///system
stop guestdomain
edit guestdomain   <-- add cache='writeback' in the driver section
start guestdomain

Here is the how the disk part of my guest domain looks like after adding the cache writeback:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source file='/var/vms/vm2/ubuntu-kvm/tmphAUcOB.qcow2'/>
<target dev='hda' bus='ide'/>
</disk>

In the process of debugging and searching for a fix for this issue, I’ve found out that it can also be useful to use elevator=noop as the default kernel io scheduler; this definitely helps, but not to the same extend as the cache writeback setting on the virtio disk. You can add elevator=noop to your kernel command line in your grub config, and I have this by default on all the instances.

Hopefully this will help you greatly improve IO performance for your KVM guests and will save you the time I’ve lost while trying to find a solution to this problem. Please feel free to share your experiences using the comment form bellow; also I’m curious if you have any other tips on how to improve this even more.

To see what versions of java we have installed on the system (from debian packages):
update-java-alternatives -l
java-1.5.0-sun 53 /usr/lib/jvm/java-1.5.0-sun
java-6-sun 63 /usr/lib/jvm/java-6-sun

We can see that the default version is 1.6 in my case (as it was the last installed):
java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

We can change the default version with: update-java-alternatives –jre -s <ver> , like:
update-java-alternatives --jre -s java-1.5.0-sun
and now the default is 1.5:
java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) Client VM (build 1.5.0_22-b03, mixed mode, sharing)

This is quite handy if you need to have multiple java versions installed, and need a quick way to change the default one (you can access any of them directly from their own path of course).

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

The pecl memcache php extension has supported for a long time the memcache session.save_handler, but with the release 3.0.x (still in beta at this time) this brings in a set of interesting features for us:
- UDP support
- Binary protocol support
- Non-blocking IO using select()
- Key and session redundancy (values are written to N mirrors)
- Improved error reporting and failover handling

Installing the php memcache module is very simple and can be done either by using distribution repositories (the version we want to use 3.0.x will probably not be available) or by using pecl or manual compilation:

Using pecl:
pecl install memcache-3.0.4

or manually:
wget http://pecl.php.net/get/memcache-3.0.4.tgz
tar xvfz memcache-3.0.4.tgz
cd memcache-3.0.4
phpize
./configure
make
make install

Finally, we need to activate the module in php.ini. I normally prefer to create a new file for this memcache.ini inside the include directory of the php build (for ex. in debian this is under /etc/php5/conf.d/memcache.ini) like this:
extension=memcache.so
memcache.allow_failover = 1
memcache.redundancy = 1
memcache.session_redundancy = 2

To use memcached to store the php sessions we will have to edit php.ini and replace the default file handler settings with something like:
; Use memcache as a session handler
session.save_handler = memcache
; Use a comma separated list of server urls to use for storage:
session.save_path="udp://<memcache_server>:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

or if we don’t want to use this serverwide, we can just define it inside a php block like this:

$session_save_path = "tcp://<memcache_server1>:11211?persistent=1&weight=1&timeout=1&retry_interval=15, tcp://<memcache_server2>:11211";
ini_set('session.save_handler', 'memcache');
ini_set('session.save_path', $session_save_path);

Note: as you can see I used in the above example tcp and udp also. Please be sure that your memcached server has udp support enabled if you want to use that. Also ensure that the web server can connect to the memcache server/port (use proper firewall rules to allow this) in order for this to work.

After restarting apache, it will start using memcache to store the php sessions. If redundancy is needed (why not?) we will probably want to use the internal php memcache support to save the sessions to more servers, or if you prefer you can use an external solution to replicate the memcached server data – repcached.

In my opinion here are the advantages of the newly released official Ubuntu images:

• officially support by Canonical (Eric has done a great job in patching and updating his images, but I am sure he has better things to do and let the Ubuntu team do this).
• custom kernels: for Intrepid 2.6.27 and Hardy 2.6.24 by having Amazon support in doing this (while alestic images were using the default Amazon Fedora kernel 2.6.21 image).
• apt mirrors in the ec2 cloud provided by Ubuntu: us.ec2.archive.ubuntu.com and eu.ec2.archive.ubuntu.com
• RightScale support for advanced integration with the RightScale platform for RightScale users.

Starting a default small instance Intrepid US image (check for the current AMI Ids):
ec2-run-instances ami-5059be39 -k my-keypair
ec2-describe-instances
ssh -i .ec2/id_rsa-my-keypair ubuntu@ec2-x-x-x-x.compute-1.amazonaws.com
sudo su -
Note: you have to login as ubuntu user and sudo as root.

Don’t forget to shutdown your instances when you are done, to avoid unneeded charges .

For more details: http://www.ubuntu.com/ec2

Now this has been working fine for a long time now, using the original geoip module and patch-o-matic-ng method of installation (similar to what is very well described here). Still, this is unmaintained, and starting with kernel 2.6.22 it is no longer working. There is a patch that will make it work with a newer kernel, but if you run iptables 1.4.x this will again fail and even if there are some manual walkarounds this is still not the best solution.

The solution is called Xtables-addons. Xtables-addons is the successor to patch-o-matic-ng. Likewise, it contains extensions that were not, or are not yet, accepted in the main kernel/iptables packages. Xtables-addons is different from patch-o-matic in that you do not have to patch or recompile the kernel, sometimes recompiling iptables is also not needed.
The latest version 1.12 supports: iptables >= 1.4.1 and kernel-source >= 2.6.17.

The installation is very simple and requires only the following steps exemplified on a debian lenny machine (kernel 2.6.26 and iptables 1.4.2):

1. Install the needed dependencies: kernel headers and iptables dev:
aptitude install linux-headers-2.6.26-1-amd64 iptables-dev
libtext-csv-xs-perl will be also needed if you plan to update the database (normally you will want this to be able to update the db from time to time):
aptitude install libtext-csv-xs-perl

2. Download the xtables-addons package and the supplied database (or the sources to build your own):
wget http://switch.dl.sourceforge.net/sourceforge/xtables-addons/xtables-addons-1.12.tar.bz2
wget http://jengelh.medozas.de/files/geoip/geoip_iv0_database-20090201.tar.bz2

3. Configure and compile the package. There are several iptables modules included; you can leave them all enabled or choose to compile and install only the ones needed. For this edit the mconfig file and leave only the ones you want:
build_CHAOS=m
build_DELUDE=m
build_DHCPADDR=m
build_ECHO=
build_IPMARK=m
build_LOGMARK=m
build_SYSRQ=m
build_TARPIT=m
build_TEE=m
build_condition=m
build_fuzzy=m
build_geoip=m
build_ipp2p=m
build_ipset=m
build_length2=m
build_lscan=m
build_quota2=m

Compile and install:
./configure --with-xtlibdir=/lib/xtables
make
make install

this will add the iptables extension /lib/xtables/libxt_geoip.so and the kernel module in /lib/modules/<kernel>/extra/xt_geoip.ko

4. Now we have to put the geoip database files under the expected location (/var/geoip); this is hardcoded in the code, but you can change it if really needed and recompile. I would like to add that even if this uses the same geoip source (the free GeoLite Country database) as the original geoip iptables module, but the format has changed. You can either get the database from the source, or build your own with the supplied script. Once you have that copy the files to /var/geoip

wget http://www.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
unzip GeoIPCountryCSV.zip
./runme.sh
cp -R var/geoip/ /var/

That’s it! All you have to do is use the module based on your needs. The syntax is the same as the original geoip iptables module:
[!] –src-cc, –source-country country[,country...] = Match packet coming from (one of) the specified country(ies)
[!] –dst-cc, –destination-country country[,country...] = Match packet going to (one of) the specified country(ies)
NOTE:  The country is inputed by its ISO3166 code.

We use something like this to mark and send each type of traffic to its own destination:
iptables -t mangle -A PREROUTING -p tcp -m geoip --src-cc US -d <IP> --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp -m geoip ! --src-cc US -d <IP> --dport 80 -j MARK --set-mark 2

I hope you found this article useful, and as me, are grateful that Xtables-addons project took over the patch-o-matic-ng broken modules and made them available on current distributions. Xtables-addons was also accepted in debian repository (in testing) and this will make it even simpler to install and use in the future.

iotop can be downloaded either as source package or a rpm package. Starting with lenny, debian includes iotop in the main repository and it can be installed just as simple as running:
aptitude install iotopThis is very cool indeed and kudos to the debian team to include iotop in lenny

For other distributions, we can install it from source using the following simple steps (be sure to have python2.5 before trying this); I am using git and this will download the latest trunk version 3, if this is not what you want download the tar.gz (ver 0.2.1) and use the stable version instead.
git clone git://repo.or.cz/iotop.git
cd iotop
./setup.py install
after this iotop will be installed under /usr/bin.

Let’s see what are the program parameters:

iotop -h
Usage: /usr/bin/iotop [OPTIONS]

DISK READ and DISK WRITE are the block I/O bandwidth used during the sampling
period. SWAPIN and IO are the percentages of time the thread spent respectively
while swapping in and waiting on I/O more generally. PRIO is the I/O priority at
which the thread is running (set using the ionice command).
Controls: left and right arrows to change the sorting column, r to invert the
sorting order, o to toggle the --only option, p to toggle the --processes
option, q to quit, any other key to force a refresh.

Options:
--version             show program's version number and exit
-h, --help            show this help message and exit
-o, --only            only show processes or threads actually doing I/O
-b, --batch           non-interactive mode
-n NUM, --iter=NUM    number of iterations before ending [infinite]
-d SEC, --delay=SEC   delay between iterations [1 second]
-p PID, --pid=PID     processes/threads to monitor [all]
-u USER, --user=USER  users to monitor [all]
-P, --processes       only show processes, not all threads

We can run it interactively, just like top, by simply typing: iotop, or add some parameters like
iotop -o -d 2
We can also run it in batch mode using -b:
iotop -b -o -d 5
...
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.49 M/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
19215 be/1 mysql       0.00 B/s    1.24 M/s  0.00 %  2.58 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11670 be/1 mysql       0.00 B/s  152.28 K/s  0.00 %  0.64 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11656 be/1 mysql       0.00 B/s    1.11 M/s  0.00 %  0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

Other similar tools are iopp (presented in a previous post) and pidstat from newer sysstat packages (>= Sysstat 8.0.0). Stay tuned for a future post on here is how you can run iotop on debian etch (that uses by default python2.4).

Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):
git clone git://git.postgresql.org/git/~markwkm/iopp.git
cd iopp
cmake CMakeLists.txt
make

And install it:
make install DESTDIR=/usr
[100%] Built target iopp
Install the project...
-- Install configuration: ""
-- Installing /usr/bin/iopp
-- Installing /usr/share/man/man8/iopp.8
after this iopp will be installed into /usr/bin.

Let’s see what are the program parameters:
iopp -h
usage: iopp -h|--help
usage: iopp [-ci] [-k|-m] [delay [count]]
-c, --command display full command line
-h, --help display help
-i, --idle hides idle processes
-k, --kilobytes display data in kilobytes
-m, --megabytes display data in megabytes
-u, --human-readable display data in kilo-, mega-, or giga-bytes

And finally let’s test it:
iopp -i -u 5
pid rchar wchar    syscr    syscw reads writes cwrites command
2464    0B    0B        0        0    0B  8192B      0B kjournald
3364  515K    0B     1336        0    0B     0B      0B mysqld
4664    0B  134B        0        4    0B     0B      0B apache2
4685  803K  114K      454       80    0B     0B      0B collectd
20758   82K   82K      329      329    0B    84K      0B cronolog
26236   67K 3754B       59       27    0B     0B      0B apache2
...

The manual page explains each output field:

• pid: The process id.
• rchar: The number of bytes which this task has caused to be read from storage.
• wchar: The number of bytes which this task has caused, or shall cause to be written to disk.
• syscr: Count of the number of read I/O operations.
• syscw: Count of the number of write I/O operations.
• rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.
• wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.
• cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.
• command: Filename of the executable.

Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.

1. Create a new RAID array

Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
or using the compact notation:
mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1

2. /etc/mdadm.conf

/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
mdadm --detail --scan >> /etc/mdadm.conf
or on debian
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

3. Remove a disk from an array

We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
mdadm --fail /dev/md0 /dev/sda1
and now we can remove it:
mdadm --remove /dev/md0 /dev/sda1

This can be done in a single step using:
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

4. Add a disk to an existing array

We can add a new disk to an array (replacing a failed one probably):
mdadm --add /dev/md0 /dev/sdb1

5. Verifying the status of the RAID arrays

We can check the status of the arrays on the system with:
cat /proc/mdstat
or
mdadm --detail /dev/md0

The output of this command will look like:

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]

here we can see both drives are used and working fine – U. A failed drive will show as F, while a degraded array will miss the second disk -

Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
watch cat /proc/mdstat

6. Stop and delete a RAID array

If we want to completely remove a raid array we have to stop if first and then remove it:
mdadm --stop /dev/md0
mdadm --remove /dev/md0
and finally we can even delete the superblock from the individual drives:
mdadm --zero-superblock /dev/sda

Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
sfdisk -d /dev/sda | sfdisk /dev/sdb

(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).

There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm –help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.