Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

So let’s see how we can use veewee. I’m assuming you already have vagrant installed (and virtualbox), but if you don’t please install them first. To install veewee we just have to install the veewee gem:
gem install veewee
once you installed veewee you can see a new task added to vagrant: basebox.

Here is the list of the templates we get out of the box once we install veewee:
vagrant basebox templates
The following templates are available:
vagrant basebox define '' 'archlinux-i686'
vagrant basebox define '' 'CentOS-4.8-i386'
vagrant basebox define '' 'CentOS-5.6-i386'
vagrant basebox define '' 'CentOS-5.6-i386-netboot'
vagrant basebox define '' 'Debian-6.0.1a-amd64-netboot'
vagrant basebox define '' 'Debian-6.0.1a-i386-netboot'
vagrant basebox define '' 'Fedora-14-amd64'
vagrant basebox define '' 'Fedora-14-amd64-netboot'
vagrant basebox define '' 'Fedora-14-i386'
vagrant basebox define '' 'Fedora-14-i386-netboot'
vagrant basebox define '' 'freebsd-8.2-experimental'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386'
vagrant basebox define '' 'freebsd-8.2-pcbsd-i386-netboot'
vagrant basebox define '' 'gentoo-latest-i386-experimental'
vagrant basebox define '' 'opensuse-11.4-i386-experimental'
vagrant basebox define '' 'solaris-11-express-i386'
vagrant basebox define '' 'Sysrescuecd-2.0.0-experimental'
vagrant basebox define '' 'ubuntu-10.04.2-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.04.2-server-amd64'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386'
vagrant basebox define '' 'ubuntu-10.04.2-server-i386-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-amd64'
vagrant basebox define '' 'ubuntu-10.10-server-amd64-netboot'
vagrant basebox define '' 'ubuntu-10.10-server-i386'
vagrant basebox define '' 'ubuntu-10.10-server-i386-netboot'
vagrant basebox define '' 'ubuntu-11.04-server-amd64'
vagrant basebox define '' 'ubuntu-11.04-server-i386'
vagrant basebox define '' 'windows-2008R2-amd64-experimental'

This means that we can build a box based on any of the above templates. That’s awesome! Let’s say we want to build a debian squeeze box using veewee; we would have to run:
vagrant basebox define 'debian-60' 'Debian-6.0.1a-amd64-netboot'
and this will create a folder definitions/debian-60 with the following files (the content of the veewee template):
definition.rb
postinstall.sh
preseed.cfg
we can modify/tune any of those files based on our custom needs. The file definition.rb is the main definition of the template. Here you would define the memory size, disk size, iso file, etc. The content is very easy to understand, but you would normally not have to change many things here. preseed.cfg is just a standard preseed file where you would customize the actual install process (you could change here the partitions or their type, timezone setup, etc). And finally postinstall.sh that is a bash script that will run at the end of the installation process and it will install ruby, gems , chef and puppet and also the virtualbox guest additions (needed for shared folders).

If you have the iso already place it in ‘currentdir’/iso. If not, veewee will download it and place it in the appropriate folder before starting the install process:
vagrant basebox build 'debian-60'
this will start the installation and you can see all the steps it takes (the keystrokes as they are entered, etc.). This can take a while… Once it is done you can validate the build with:
vagrant basebox validate 'debian-60'
(this will run a few basic tests to see if it can connect to the vm as user vagrant, if chef and puppet were installed, if the shared folders are accessible, etc).

And finally you can export it as a vagrant box with:
vagrant basebox export 'debian-60'
and add it to vagrant:
vagrant box add 'debian-60' debian-60.box
and now you can use it in vagrant with:
vagrant init 'debian-60'

That’s it. Very simple and now we have our own box built from scratch. As a side note, I found this very useful to test and troubleshoot preseed configurations . As you can see there are plenty of templates available in veewee but if you create a new one please consider to share it with others and send it to Patrick on github. I’m sure he will be happy to include it in newer versions of veewee. And if you found this useful don’t forget to thank Patrick for his great work on this awesome tool.

The daemons noitd and stratcond are written in C, and the database used is postgressql, while the web interface is written in php. We will need to install a few dependencies to be able to compile noitd/stratcond:

Dependencies

apt-get install autoconf build-essential \
zlib1g-dev uuid-dev libpcre3-dev libssl-dev libpq-dev \
libxslt-dev libapr1-dev libaprutil1-dev xsltproc \
libncurses5-dev python libssh2-1-dev libsnmp-dev \
sun-java6-jdk

In case you don’t have already installed svn, we will need that also (to download the sources):
apt-get install subversion

Compiling

Next we will download the source and compile it. I used the trunk version bellow, but you might want use a stable tag (Urskek is the latest when writing this):
cd /usr/local/src/
svn co https://labs.omniti.com/reconnoiter/trunk reconnoiter
cd reconnoiter
autoconf
./configure
make
make install

Database

We need to instal PostgreSQL 8.4 and this is not available in lenny; the simplest way we can do this is to use the backports package. To enable the backports sources add this line
deb http://www.backports.org/debian lenny-backports main contrib non-free
to your /etc/apt/sources.list and run apt-get update to refresh your apt indexes. And finally add the backports key into apt:
apt-get install debian-backports-keyring

Now we can install postgres 8.4 using:
apt-get -t lenny-backports install postgresql

Next we’ll need to edit /etc/postgresql/8.4/main/pg_hba.conf and replace this line
local all all identwith
local all all trust
and restart postgres:
/etc/init.d/postgresql-8.4 restart
to allow local access without password.

Next we can install the dbs, using:
su postgres
cd /usr/local/src/reconnoiter/sql
psql
\i scaffolding.sql
\q

DB crontab

You can find a sample crontab in the sql folder that needs to be customized for our particular setup (/opt/psql835/bin -> /usr/bin/), and it should look like this:

su postgres
crontab -l
# m h  dom mon dow   command
#rollup jobs
* * * * * /usr/bin/psql -d reconnoiter -U stratcon -c "select stratcon.rollup_metric_numeric(rollup) from metric_numeric_rollup_config order by seconds asc;" >/tmp/rollup.log 2>&1
#cleanup jobs
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_numeric_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_text_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.check_status_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.check_archive', 'whence', 'day', 7);" 1>/dev/null
01 00 * * * /usr/bin/psql -d reconnoiter -U reconnoiter -c "delete from prism.saved_graphs where saved = false and last_update/dev/null

!!! I would suggest to run the daily crons manually once as they create the needed tables and in case you don’t have them you will receive various errors (I know I lost a lot of time trying to figure out why those were not there). This should be just like:
/usr/bin/psql -d reconnoiter -U reconnoiter -c "select stratcon.archive_part_maint('noit.metric_text_archive', 'whence', 'day', 7);"
and so on for each of the 5 daily crons from above.

keys

For a production install you will obviously create some real certs, but in my case where I was just testing this out the test certs build by the installer are just fine:
cd /usr/local/src/reconnoiter/test
cp test-*.crt test-*.key /usr/local/etc/

In dealing with the configuration files uuid’s you will find useful the uuidgen program from the uuid-runtime package:
apt-get install uuid-runtime
that will generate a new uuid on each run:
uuidgen
22426aa0-db34-437d-a891-e759ffbdb494

noitd

We will start from the sample configuration:
mv /usr/local/etc/noit.conf.sample /usr/local/etc/noit.conf
and edit/remove it as needed. Be sure to edit the sslconfig part with the keys you copied above. For more details on the config file look on the wiki.

Finally run noitd and see if all is good:
/usr/local/sbin/noitd -c /usr/local/etc/noit.conf -D
If there are no errors stop it, and run it in the background (without -D):
/usr/local/sbin/noitd -c /usr/local/etc/noit.conf

stratcon

For stratcon we are going to use a similar approach:
mv /usr/local/etc/stratcon.conf.sample /usr/local/etc/stratcon.conf
and then customize the config based on our needs. Set the keys in sslconfig section and the db details

    <dbconfig>
      <host>localhost</host>
      <dbname>reconnoiter</dbname>
      <user>stratcon</user>
      <password>unguessable</password>
    </dbconfig>    

the password is “stratcon” or whatever you set in scaffolding.sql if you customized it (a good idea ).
and finally run it with debuging in the foreground:
/usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf -D
and if all is good (you did ran the db creation task, right?) you can start it in the background without -D and move to the final step:
/usr/local/sbin/stratcond -c /usr/local/etc/stratcon.conf

Web interface

Finally the last step is to configure the web interface. For this we need to be sure we have apache/php installed and the pgsql module:
apt-get install apache2 libapache2-mod-php5 php5-pgsql
and to enable the rewrite module
a2enmod rewrite
Next we would copy the web files ui to some place like /var/www/ui and create a vhost for reconnoiter inside /etc/apache2/sites-available:

<VirtualHost *:80>
  DocumentRoot /var/www/ui/web/htdocs

  <Directory "/">
      Options None
      AllowOverride None
      Order allow,deny
      Deny from all
  </Directory>
  <FilesMatch "^\.ht">
      Order allow,deny
      Deny from all
      Satisfy All
  </FilesMatch>
  <Directory "/var/www/ui/web/htdocs/">
      php_value include_path /var/www/ui/web/lib
      php_value short_open_tag off
      Options FollowSymLinks Indexes
      AllowOverride All
      Order deny,allow
      Allow from all
  </Directory>

  LogLevel warn
  LogFormat "%h %l %u %t \"%r\" %>s %b" common

  ErrorLog /var/www/ui/web/logs/error_log
  CustomLog /var/www/ui/web/logs/access_log common

  AddType application/x-compress .Z
  AddType application/x-gzip .gz .tgz
  AddType application/x-httpd-php .php
  DefaultType text/plain

  RewriteEngine On
  RewriteRule ^(/data/.+)$ http://localhost:80$1 [P,L,QSA]
</VirtualHost>

enable the config:
a2ensite reconnoiter
and restart apache to enable it: /etc/init.d/apache2 restart

After this hopefully you can open the web interface and start adding some metrics. My goal was to test this with collectd and if there is interest I could describe the details on how to make reconnoiter work with collectd in another post. let me know…

~$ google
> help
Welcome to the Google CL tool!
Commands are broken into several parts: service, task, options, and arguments.
For example, in the command
"> picasa post --title "My Cat Photos" photos/cats/*"
the service is "picasa", the task is "post", the single option is a name of "My Cat Photos", and the argument is the path to the photos.
The available services are 'picasa', 'blogger', 'youtube', 'docs', 'contacts', 'calendar'
Enter "> help <service>" for more information on a service.
Or, just "quit" to quit.

There is a deb package available for ubuntu/debian_lenny, and source package for other distributions. The only requirement is Python 2.5 or 2.6 and the gdata python client library.
You can install the package manually just as any python package:
sudo python setup.py install

For MacOSX if you are running MacPorts, it is easier to install gdata and googlecl from ports (also easier to keep them updated in the future). The install should be as simple as:
sudo port install googlecl
in case you don’t see it in your source list, just run a port selfupdate:
sudo port selfupdate

Usage

There are several good examples on the project page, but basically each service has the ability to list, add and delete. Here are some simple examples for calendar:

List events for today only:
google calendar today

and the same with some nicer formatting:
google calendar today when,title --delimiter " | "
Jun 21 09:30 - Jun 21 10:00 | Very important call

we can also do a query after some strinng:
google calendar list --query velocity

and adding a new event:
google calendar add "Very important call today at 10am"

or delete an event:
google calendar delete --title "Meeting with John"

Calendar is definitely the most important one for me, but contacts and docs are also great to be controlled from the command line. I would like to see the same for Gmail in the future .

Now this has been working fine for a long time now, using the original geoip module and patch-o-matic-ng method of installation (similar to what is very well described here). Still, this is unmaintained, and starting with kernel 2.6.22 it is no longer working. There is a patch that will make it work with a newer kernel, but if you run iptables 1.4.x this will again fail and even if there are some manual walkarounds this is still not the best solution.

The solution is called Xtables-addons. Xtables-addons is the successor to patch-o-matic-ng. Likewise, it contains extensions that were not, or are not yet, accepted in the main kernel/iptables packages. Xtables-addons is different from patch-o-matic in that you do not have to patch or recompile the kernel, sometimes recompiling iptables is also not needed.
The latest version 1.12 supports: iptables >= 1.4.1 and kernel-source >= 2.6.17.

The installation is very simple and requires only the following steps exemplified on a debian lenny machine (kernel 2.6.26 and iptables 1.4.2):

1. Install the needed dependencies: kernel headers and iptables dev:
aptitude install linux-headers-2.6.26-1-amd64 iptables-dev
libtext-csv-xs-perl will be also needed if you plan to update the database (normally you will want this to be able to update the db from time to time):
aptitude install libtext-csv-xs-perl

2. Download the xtables-addons package and the supplied database (or the sources to build your own):
wget http://switch.dl.sourceforge.net/sourceforge/xtables-addons/xtables-addons-1.12.tar.bz2
wget http://jengelh.medozas.de/files/geoip/geoip_iv0_database-20090201.tar.bz2

3. Configure and compile the package. There are several iptables modules included; you can leave them all enabled or choose to compile and install only the ones needed. For this edit the mconfig file and leave only the ones you want:
build_CHAOS=m
build_DELUDE=m
build_DHCPADDR=m
build_ECHO=
build_IPMARK=m
build_LOGMARK=m
build_SYSRQ=m
build_TARPIT=m
build_TEE=m
build_condition=m
build_fuzzy=m
build_geoip=m
build_ipp2p=m
build_ipset=m
build_length2=m
build_lscan=m
build_quota2=m

Compile and install:
./configure --with-xtlibdir=/lib/xtables
make
make install

this will add the iptables extension /lib/xtables/libxt_geoip.so and the kernel module in /lib/modules/<kernel>/extra/xt_geoip.ko

4. Now we have to put the geoip database files under the expected location (/var/geoip); this is hardcoded in the code, but you can change it if really needed and recompile. I would like to add that even if this uses the same geoip source (the free GeoLite Country database) as the original geoip iptables module, but the format has changed. You can either get the database from the source, or build your own with the supplied script. Once you have that copy the files to /var/geoip

wget http://www.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
unzip GeoIPCountryCSV.zip
./runme.sh
cp -R var/geoip/ /var/

That’s it! All you have to do is use the module based on your needs. The syntax is the same as the original geoip iptables module:
[!] –src-cc, –source-country country[,country...] = Match packet coming from (one of) the specified country(ies)
[!] –dst-cc, –destination-country country[,country...] = Match packet going to (one of) the specified country(ies)
NOTE:  The country is inputed by its ISO3166 code.

We use something like this to mark and send each type of traffic to its own destination:
iptables -t mangle -A PREROUTING -p tcp -m geoip --src-cc US -d <IP> --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp -m geoip ! --src-cc US -d <IP> --dport 80 -j MARK --set-mark 2

I hope you found this article useful, and as me, are grateful that Xtables-addons project took over the patch-o-matic-ng broken modules and made them available on current distributions. Xtables-addons was also accepted in debian repository (in testing) and this will make it even simpler to install and use in the future.

1. Use Redmine 0.8.x stable

I lost some time with the current trunk version (r2571) until I realized that the importer is broken in trunk. Use the latest stable 0.8.x version, and if you want to use trunk, upgrade to it after you have the trac data inside redmine. While using trunk all was working fine (no error or anything strange), just the wiki pages were not imported in the redmine project. Anyway, I will assume you have a working clean installation of redmine 0.8.x before we will move on.

2. Import Trac data

My trac (0.11.2) database was using sqlite3; meaning I had to install first sqlite3-ruby:
gem install sqlite3-ruby
Next I ran the import script from inside the root redmine folder:
rake redmine:migrate_from_trac RAILS_ENV="production"
and entered the following:

Trac directory []: /var/trac/project
Trac database adapter (sqlite, sqlite3, mysql, postgresql) [sqlite]: sqlite3
Database encoding [UTF-8]:
Target project identifier []: project

if for your trac installation uses sqlite then use the original doc steps. For the target project identifier use a name for a new project in redmine that will have the trac data imported (small caps).

The script showed a summary of the imported objects, and hopefully you will find them inside the redmine new project. In my case most of the wiki pages were ok, but some had wrong formatting and needed to be manually fixed (minor issues). Overall I am very happy on how this worked and that I have now all my old trac data inside redmine.

3. Upgrade to trunk (only if you want to use trunk)

If you wanted to use trunk (like me) you can now migrate the redmine installation to trunk using svn (make backups first, etc):
svn sw http://redmine.rubyforge.org/svn/trunk .
rake db:migrate RAILS_ENV="production"
rake config/initializers/session_store.rb
and restart redmine, and you should be running trunk.

1. Kernel

The default etch kernel is 2.6.18 and even if this will not block us from installing iotop, there will be no result unless we run a kernel newer than 2.6.20. Luckily debian etch provides etchnhalf that made available a newer kernel 2.6.24 in etch. Another approach is to build your own kernel of course, where you will probably use the latest stable version at that time. Still if you run this on many machines it will probably wise to use the etchnhalf kernel or pack your own deb with a newer one. For our needs we will just use the 2.6.24 etchnhalf kernel:
aptitude install linux-image-2.6.24-etchnhalf.1-amd64 if you run on amd64 debian etch, or:
aptitude install linux-image-2.6.24-etchnhalf.1-686if you use the regular i386.

Note: there are some limitations that you should be aware of (please see debian etchnhalf release notes for more details), mainly there is no xen compatible image (meaning if you use xen don’t do this), and missing of bnx2 firmware from the build-in kernel (see my post about this).

Ok, once this is done, reboot and after your system is up and running with the new kernel we can move on:
uname -a
Linux srv02 2.6.24-etchnhalf.1-686-bigmem #1 SMP Fri Dec 26 04:53:15 UTC 2008 i686 GNU/Linux

2. Python 2.5

The default version of python in debian etch is 2.4 (while for lenny it is 2.5). Still python2.5 is available in etch also, and can be installed using:
aptitude install python2.5 python2.5-minimal python2.5-dev

That’s it. Most (so I don’t say all) of the python modules in the etch repo are working only with the default python version 2.4. For example for iotop build we need setuptools, but installing python2.5-setuptools fails for some reason; anyway the idea is that we were able to install python2.5 easily from the repo, but we will have to install the rest of the extra python 2.5 modules from source and point them to python2.5. I am doing this assuming you will want to keep python2.4 as the default system python version as probably you have some tools depending on it (like we have bcfg2 for example), so we don’t mess with that.

Let’s download setuptools for python2.5 and install it:
wget http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c9-py2.5.egg
sh setuptools-0.6c9-py2.5.egg

3. Finally iotop

All what is left is to install iotop, now that we have all the dependencies in place.
git clone git://repo.or.cz/iotop.git
cd iotop
and change inside setup.py the first line to use python2.5 like this:
#!/usr/bin/env python2.5
and run setup:
./setup.py install
hopefully this will install just fine, and you should have iotop running using the python2.5 binary.

I hope this post will be useful to users of debian etch showing them a quick way to install iotop on their existing system.

iotop can be downloaded either as source package or a rpm package. Starting with lenny, debian includes iotop in the main repository and it can be installed just as simple as running:
aptitude install iotopThis is very cool indeed and kudos to the debian team to include iotop in lenny

For other distributions, we can install it from source using the following simple steps (be sure to have python2.5 before trying this); I am using git and this will download the latest trunk version 3, if this is not what you want download the tar.gz (ver 0.2.1) and use the stable version instead.
git clone git://repo.or.cz/iotop.git
cd iotop
./setup.py install
after this iotop will be installed under /usr/bin.

Let’s see what are the program parameters:

iotop -h
Usage: /usr/bin/iotop [OPTIONS]

DISK READ and DISK WRITE are the block I/O bandwidth used during the sampling
period. SWAPIN and IO are the percentages of time the thread spent respectively
while swapping in and waiting on I/O more generally. PRIO is the I/O priority at
which the thread is running (set using the ionice command).
Controls: left and right arrows to change the sorting column, r to invert the
sorting order, o to toggle the --only option, p to toggle the --processes
option, q to quit, any other key to force a refresh.

Options:
--version             show program's version number and exit
-h, --help            show this help message and exit
-o, --only            only show processes or threads actually doing I/O
-b, --batch           non-interactive mode
-n NUM, --iter=NUM    number of iterations before ending [infinite]
-d SEC, --delay=SEC   delay between iterations [1 second]
-p PID, --pid=PID     processes/threads to monitor [all]
-u USER, --user=USER  users to monitor [all]
-P, --processes       only show processes, not all threads

We can run it interactively, just like top, by simply typing: iotop, or add some parameters like
iotop -o -d 2
We can also run it in batch mode using -b:
iotop -b -o -d 5
...
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.49 M/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
19215 be/1 mysql       0.00 B/s    1.24 M/s  0.00 %  2.58 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11670 be/1 mysql       0.00 B/s  152.28 K/s  0.00 %  0.64 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11656 be/1 mysql       0.00 B/s    1.11 M/s  0.00 %  0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

Other similar tools are iopp (presented in a previous post) and pidstat from newer sysstat packages (>= Sysstat 8.0.0). Stay tuned for a future post on here is how you can run iotop on debian etch (that uses by default python2.4).

Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):
git clone git://git.postgresql.org/git/~markwkm/iopp.git
cd iopp
cmake CMakeLists.txt
make

And install it:
make install DESTDIR=/usr
[100%] Built target iopp
Install the project...
-- Install configuration: ""
-- Installing /usr/bin/iopp
-- Installing /usr/share/man/man8/iopp.8
after this iopp will be installed into /usr/bin.

Let’s see what are the program parameters:
iopp -h
usage: iopp -h|--help
usage: iopp [-ci] [-k|-m] [delay [count]]
-c, --command display full command line
-h, --help display help
-i, --idle hides idle processes
-k, --kilobytes display data in kilobytes
-m, --megabytes display data in megabytes
-u, --human-readable display data in kilo-, mega-, or giga-bytes

And finally let’s test it:
iopp -i -u 5
pid rchar wchar    syscr    syscw reads writes cwrites command
2464    0B    0B        0        0    0B  8192B      0B kjournald
3364  515K    0B     1336        0    0B     0B      0B mysqld
4664    0B  134B        0        4    0B     0B      0B apache2
4685  803K  114K      454       80    0B     0B      0B collectd
20758   82K   82K      329      329    0B    84K      0B cronolog
26236   67K 3754B       59       27    0B     0B      0B apache2
...

The manual page explains each output field:

• pid: The process id.
• rchar: The number of bytes which this task has caused to be read from storage.
• wchar: The number of bytes which this task has caused, or shall cause to be written to disk.
• syscr: Count of the number of read I/O operations.
• syscw: Count of the number of write I/O operations.
• rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.
• wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.
• cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.
• command: Filename of the executable.

Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.

1. Create a new RAID array

Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
or using the compact notation:
mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1

2. /etc/mdadm.conf

/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
mdadm --detail --scan >> /etc/mdadm.conf
or on debian
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

3. Remove a disk from an array

We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
mdadm --fail /dev/md0 /dev/sda1
and now we can remove it:
mdadm --remove /dev/md0 /dev/sda1

This can be done in a single step using:
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

4. Add a disk to an existing array

We can add a new disk to an array (replacing a failed one probably):
mdadm --add /dev/md0 /dev/sdb1

5. Verifying the status of the RAID arrays

We can check the status of the arrays on the system with:
cat /proc/mdstat
or
mdadm --detail /dev/md0

The output of this command will look like:

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]

here we can see both drives are used and working fine – U. A failed drive will show as F, while a degraded array will miss the second disk -

Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
watch cat /proc/mdstat

6. Stop and delete a RAID array

If we want to completely remove a raid array we have to stop if first and then remove it:
mdadm --stop /dev/md0
mdadm --remove /dev/md0
and finally we can even delete the superblock from the individual drives:
mdadm --zero-superblock /dev/sda

Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
sfdisk -d /dev/sda | sfdisk /dev/sdb

(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).

There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm –help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.