Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Use the command line help (fpm –help) or the wiki to see full details on how to use it. I’ll show some simple examples on how to build some packages from various input sources that I’ve found useful myself.

1. Package a directory – output of a ‘make install’ command

This is how you would usually package an application that you would install with:
./configure; make; make install
For example, here is how you can create an rpm of the latest version of memcached:
wget http://memcached.googlecode.com/files/memcached-1.4.7.tar.gz
tar -zxvf memcached-1.4.7.tar.gz
cd memcached-1.4.7
./configure --prefix=/usr
make
so far everything looks like a normal manual installation (that would be followed by make install). Still we will now install it in a separate folder so we can capture the output:
mkdir /tmp/installdir
make install DESTDIR=/tmp/installdir
and finally using fpm to create the rpm package:
fpm -s dir -t rpm -n memcached -v 1.4.7 -C /tmp/installdir
where -s is the input source type (directory), -t is the type of package (rpm), -n in the name of the package and -v is the version; -C is the directory where fpm will look for the files.
Note: you might need to install various libraries to build your package; for ex. in this case I had to install libevent-dev.

If you are packaging your own application you can do this just by pointing to your build folder and set the version of the app. Here is an example for an deb package:
fpm -s dir -t deb -n myapp -v 0.0.1 -C /build/myapp/0.0.1/

There are various other parameters that you can use but basically this is how simple it is to build a package from a directory.
Here is an example on how to define some dependencies on the package you are building (using -d; repeat it as many times as needed):
fpm -s dir -t deb -n memcached -v 1.4.7 -C /tmp/installdir \
-d "libstdc++6 (>= 4.4.5)" \
-d "libevent-1.4-2 (>= 1.4.13)"

2. Ruby gems or python egg – converted to packages

You can create a deb or rpm from a gem very simple with fpm:
fpm -s gem -t deb <gem_name>
this will download the gem and create a package named rubygem-<gem_name>
For example:
fpm -s gem -t deb fpm
will create a debian package for fpm: rubygem-fpm_0.3.7_all.deb

You can inspect it with dpkg –info and you can notice that in this case it will fill nicely all the fields with the maintainer, and dependencies on various other gems. Very cool.

If you use python and want to package various python eggs this will work exactly the same and you will use -s python (it will download the python packages with easy_install first).

Overall FPM is a great tool and can help you simplify the way you are building your own packages. Check it out and let me know what you think and if you found it useful. And if you found this useful don’t forget to thank Jordan for his great work on this awesome tool.

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

The pecl memcache php extension has supported for a long time the memcache session.save_handler, but with the release 3.0.x (still in beta at this time) this brings in a set of interesting features for us:
- UDP support
- Binary protocol support
- Non-blocking IO using select()
- Key and session redundancy (values are written to N mirrors)
- Improved error reporting and failover handling

Installing the php memcache module is very simple and can be done either by using distribution repositories (the version we want to use 3.0.x will probably not be available) or by using pecl or manual compilation:

Using pecl:
pecl install memcache-3.0.4

or manually:
wget http://pecl.php.net/get/memcache-3.0.4.tgz
tar xvfz memcache-3.0.4.tgz
cd memcache-3.0.4
phpize
./configure
make
make install

Finally, we need to activate the module in php.ini. I normally prefer to create a new file for this memcache.ini inside the include directory of the php build (for ex. in debian this is under /etc/php5/conf.d/memcache.ini) like this:
extension=memcache.so
memcache.allow_failover = 1
memcache.redundancy = 1
memcache.session_redundancy = 2

To use memcached to store the php sessions we will have to edit php.ini and replace the default file handler settings with something like:
; Use memcache as a session handler
session.save_handler = memcache
; Use a comma separated list of server urls to use for storage:
session.save_path="udp://<memcache_server>:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

or if we don’t want to use this serverwide, we can just define it inside a php block like this:

$session_save_path = "tcp://<memcache_server1>:11211?persistent=1&weight=1&timeout=1&retry_interval=15, tcp://<memcache_server2>:11211";
ini_set('session.save_handler', 'memcache');
ini_set('session.save_path', $session_save_path);

Note: as you can see I used in the above example tcp and udp also. Please be sure that your memcached server has udp support enabled if you want to use that. Also ensure that the web server can connect to the memcache server/port (use proper firewall rules to allow this) in order for this to work.

After restarting apache, it will start using memcache to store the php sessions. If redundancy is needed (why not?) we will probably want to use the internal php memcache support to save the sessions to more servers, or if you prefer you can use an external solution to replicate the memcached server data – repcached.

Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):
git clone git://git.postgresql.org/git/~markwkm/iopp.git
cd iopp
cmake CMakeLists.txt
make

And install it:
make install DESTDIR=/usr
[100%] Built target iopp
Install the project...
-- Install configuration: ""
-- Installing /usr/bin/iopp
-- Installing /usr/share/man/man8/iopp.8
after this iopp will be installed into /usr/bin.

Let’s see what are the program parameters:
iopp -h
usage: iopp -h|--help
usage: iopp [-ci] [-k|-m] [delay [count]]
-c, --command display full command line
-h, --help display help
-i, --idle hides idle processes
-k, --kilobytes display data in kilobytes
-m, --megabytes display data in megabytes
-u, --human-readable display data in kilo-, mega-, or giga-bytes

And finally let’s test it:
iopp -i -u 5
pid rchar wchar    syscr    syscw reads writes cwrites command
2464    0B    0B        0        0    0B  8192B      0B kjournald
3364  515K    0B     1336        0    0B     0B      0B mysqld
4664    0B  134B        0        4    0B     0B      0B apache2
4685  803K  114K      454       80    0B     0B      0B collectd
20758   82K   82K      329      329    0B    84K      0B cronolog
26236   67K 3754B       59       27    0B     0B      0B apache2
...

The manual page explains each output field:

• pid: The process id.
• rchar: The number of bytes which this task has caused to be read from storage.
• wchar: The number of bytes which this task has caused, or shall cause to be written to disk.
• syscr: Count of the number of read I/O operations.
• syscw: Count of the number of write I/O operations.
• rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.
• wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.
• cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.
• command: Filename of the executable.

Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.

1. Create a new RAID array

Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
or using the compact notation:
mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1

2. /etc/mdadm.conf

/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
mdadm --detail --scan >> /etc/mdadm.conf
or on debian
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

3. Remove a disk from an array

We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
mdadm --fail /dev/md0 /dev/sda1
and now we can remove it:
mdadm --remove /dev/md0 /dev/sda1

This can be done in a single step using:
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

4. Add a disk to an existing array

We can add a new disk to an array (replacing a failed one probably):
mdadm --add /dev/md0 /dev/sdb1

5. Verifying the status of the RAID arrays

We can check the status of the arrays on the system with:
cat /proc/mdstat
or
mdadm --detail /dev/md0

The output of this command will look like:

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]

here we can see both drives are used and working fine – U. A failed drive will show as F, while a degraded array will miss the second disk -

Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
watch cat /proc/mdstat

6. Stop and delete a RAID array

If we want to completely remove a raid array we have to stop if first and then remove it:
mdadm --stop /dev/md0
mdadm --remove /dev/md0
and finally we can even delete the superblock from the individual drives:
mdadm --zero-superblock /dev/sda

Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
sfdisk -d /dev/sda | sfdisk /dev/sdb

(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).

There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm –help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.

It is quite frequent to use separate IPs for various services on the same machine, and move those IPs to another server if needed. These are sometimes called portable IPs that can be migrated to any server in a particular colo/lan. This is done normally to minimized downtime and keep maintenance of such operations minimal (and to not rely on dns changes). Still arp caching on various network devices can cause big problems. Let’s assume we moved the IP from one server to another one in the same LAN to move away some service from our main web server. Taking down the IP from the existing server and bringing it up on the new server will complete our direct work if we don’t have access on the switches/routers in front of us. Again if you have control on all devices just connect to them and delete the arp cache for this ip to allow it to be re-cached on the new machine.

So after we have the IP moved on the new machine and now have to wait… The arp cache depends on the actual devices and can be anything from 5 minutes to not expire. Let’s assume for this example that the ip is 192.168.0.101 and after we run ifup and ifdown we have the IP correctly showing on the new server, srv02. If we don’t want to wait helpless for this to happen automatically, the solution is to broadcast from our new machine the arp with the source of the IP. Hopefully this will make the remote device to verify and invalidate its existing cache entry. For this we can use arping; installation is simple as it should be in most modern linux distributions by default. On debian you would install arping just by running:
aptitude install arping
finally we use a command like:
arping -S <our_IP> -B
that will broadcast our source IP and direct it to the broadcast address (255.255.255.255) . If your arp command uses different parameters notations, you should looks for something similar (to set the source and ping the broadcast). In our example with the IP 192.168.0.101 we would use:
srv02:~# arping -S 192.168.0.101 -B
ARPING 255.255.255.255
--- 255.255.255.255 statistics ---
16 packets transmitted, 0 packets received, 100% unanswered
(stop it with CTRL-C once it is working).

Normally after this, all should be ok and the remote device should cache the new arp entry, invalidating the existing cached one. If this is not the case, then call your datacenter to minimize the downtime . I would always suggest testing this first and seeing what downtime to expect and if you can minimize it like this with arping, first try with a non-production test IP. Don’t do this with live, production IP/service until you know what to expect. I hope this post will help you if you will have to deal with a similar situation. If this doesn’t work as expected in your case, please let us know what devices you found problematic, and if you were able to use a different workaround.

The SkipFiles parameter is by default empty, and in order to use it we have to add the appropriate config in our awstats.conf file.  For example to ignore a file called somefile.php we will add “/somefile.php“:
SkipFiles="/somefile.php"
while if we want to ignore all the pages in a folder called somefolder we will have to use a regex value like: “REGEX[^\/somefolder]“
We can add several rules separated by spaces:
SkipFiles="/somefile.php REGEX[^\/somefolder]"

Note: this will be effective only for new updates; meaning existing data will not be affected by this (if really needed you will have to regenerate your stats to get rid of those pages in old stats).

Let’s start by assuming we want to use 8 mail queues. First thing is to create the actual directories as sendmail will not do this by default:
mkdir /var/spool/mqueue/q{1,2,3,4,5,6,7,8}
And fix the permissions to the ones of the original folder /var/spool/mqueue. For ex. this might look like:
chown -R smmta:smmsp /var/spool/mqueue/q*
using a default sendmail install running on debian. Fix the users to the specific ones found on your system (ls -al /var/spool/mqueue if you are uncertain of this).

Next, we need to enable the multiple queues in the sendmail configuration. For this we will edit sendmail.mc (normally found under /etc/mail) and append one line:
define(`QUEUE_DIR', `/var/spool/mqueue/q*')dnl
and now regenerate sendmail.cf; this is done normally running:
m4 sendmail.mc > /etc/mail/sendmail.cf
(fix your paths appropriately), or if you are using debian sendmail you can just run make all in /etc/mail.

After restarting sendmail, it will start using the multiple queues we defined. Running mailq will output each of the queues:
mailq
/var/spool/mqueue/q6 is empty
/var/spool/mqueue/q4 is empty
/var/spool/mqueue/q3 is empty
/var/spool/mqueue/q2 is empty
/var/spool/mqueue/q5 is empty
/var/spool/mqueue/q1 is empty
/var/spool/mqueue/q7 is empty
/var/spool/mqueue/q8 is empty
Total requests: 0

Note: if you want to add more folders to the configuration all you have to do is to create the respective folders, set the appropriate permissions and restart sendmail.

If you had existing mails in the queue (most likely if you were looking for this solution), if you want them still processed, move them from /var/spool/mqueue in one of the newly created queues (q1 for ex).

Individual queue directories can be symbolic links to other partitions to spreads load among multiple disks. Queue IDs are unique across queues so you can move the items among queues if you have to.

Reading from Tim Verhoeven’s explanation:

“For some background information, why does it take 3,5 weeks ? First we need to remove all the logos and trademarks of Upstream. Secondly we need to build everything from source and this for both i386 and x86_64. Then everything that gets build goes past the QA team that verify that everything works as it should. From all the build packages install media will be created and these also need to be tested by the QA team. For each release a set of release notes are created and these are translated in different languages (12 for 5.1). Finally all the packages and media need to be uploaded in distributed to the mirror network so you can download it.”

we learn that we should expect Centos 5.2 released sometimes around June 14th 2008 (sooner or later).

• Virtualization
• Laptop and Desktop improvements
• Encryption and Security
• Cluster & Storage Enhancements
• Networking & IPv6 Enablement
• Serviceability

“Update brings broad refresh of hardware support and improved quality, combined with new features and enhancements in areas such as virtualization, desktop, networking, storage & clustering and security”

For full details check out the redhat press release.