Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

The pecl memcache php extension has supported for a long time the memcache session.save_handler, but with the release 3.0.x (still in beta at this time) this brings in a set of interesting features for us:
- UDP support
- Binary protocol support
- Non-blocking IO using select()
- Key and session redundancy (values are written to N mirrors)
- Improved error reporting and failover handling

Installing the php memcache module is very simple and can be done either by using distribution repositories (the version we want to use 3.0.x will probably not be available) or by using pecl or manual compilation:

Using pecl:
pecl install memcache-3.0.4

or manually:
wget http://pecl.php.net/get/memcache-3.0.4.tgz
tar xvfz memcache-3.0.4.tgz
cd memcache-3.0.4
phpize
./configure
make
make install

Finally, we need to activate the module in php.ini. I normally prefer to create a new file for this memcache.ini inside the include directory of the php build (for ex. in debian this is under /etc/php5/conf.d/memcache.ini) like this:
extension=memcache.so
memcache.allow_failover = 1
memcache.redundancy = 1
memcache.session_redundancy = 2

To use memcached to store the php sessions we will have to edit php.ini and replace the default file handler settings with something like:
; Use memcache as a session handler
session.save_handler = memcache
; Use a comma separated list of server urls to use for storage:
session.save_path="udp://<memcache_server>:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

or if we don’t want to use this serverwide, we can just define it inside a php block like this:

$session_save_path = "tcp://<memcache_server1>:11211?persistent=1&weight=1&timeout=1&retry_interval=15, tcp://<memcache_server2>:11211";
ini_set('session.save_handler', 'memcache');
ini_set('session.save_path', $session_save_path);

Note: as you can see I used in the above example tcp and udp also. Please be sure that your memcached server has udp support enabled if you want to use that. Also ensure that the web server can connect to the memcache server/port (use proper firewall rules to allow this) in order for this to work.

After restarting apache, it will start using memcache to store the php sessions. If redundancy is needed (why not?) we will probably want to use the internal php memcache support to save the sessions to more servers, or if you prefer you can use an external solution to replicate the memcached server data – repcached.

In my opinion here are the advantages of the newly released official Ubuntu images:

• officially support by Canonical (Eric has done a great job in patching and updating his images, but I am sure he has better things to do and let the Ubuntu team do this).
• custom kernels: for Intrepid 2.6.27 and Hardy 2.6.24 by having Amazon support in doing this (while alestic images were using the default Amazon Fedora kernel 2.6.21 image).
• apt mirrors in the ec2 cloud provided by Ubuntu: us.ec2.archive.ubuntu.com and eu.ec2.archive.ubuntu.com
• RightScale support for advanced integration with the RightScale platform for RightScale users.

Starting a default small instance Intrepid US image (check for the current AMI Ids):
ec2-run-instances ami-5059be39 -k my-keypair
ec2-describe-instances
ssh -i .ec2/id_rsa-my-keypair ubuntu@ec2-x-x-x-x.compute-1.amazonaws.com
sudo su -
Note: you have to login as ubuntu user and sudo as root.

Don’t forget to shutdown your instances when you are done, to avoid unneeded charges .

For more details: http://www.ubuntu.com/ec2

Now this has been working fine for a long time now, using the original geoip module and patch-o-matic-ng method of installation (similar to what is very well described here). Still, this is unmaintained, and starting with kernel 2.6.22 it is no longer working. There is a patch that will make it work with a newer kernel, but if you run iptables 1.4.x this will again fail and even if there are some manual walkarounds this is still not the best solution.

The solution is called Xtables-addons. Xtables-addons is the successor to patch-o-matic-ng. Likewise, it contains extensions that were not, or are not yet, accepted in the main kernel/iptables packages. Xtables-addons is different from patch-o-matic in that you do not have to patch or recompile the kernel, sometimes recompiling iptables is also not needed.
The latest version 1.12 supports: iptables >= 1.4.1 and kernel-source >= 2.6.17.

The installation is very simple and requires only the following steps exemplified on a debian lenny machine (kernel 2.6.26 and iptables 1.4.2):

1. Install the needed dependencies: kernel headers and iptables dev:
aptitude install linux-headers-2.6.26-1-amd64 iptables-dev
libtext-csv-xs-perl will be also needed if you plan to update the database (normally you will want this to be able to update the db from time to time):
aptitude install libtext-csv-xs-perl

2. Download the xtables-addons package and the supplied database (or the sources to build your own):
wget http://switch.dl.sourceforge.net/sourceforge/xtables-addons/xtables-addons-1.12.tar.bz2
wget http://jengelh.medozas.de/files/geoip/geoip_iv0_database-20090201.tar.bz2

3. Configure and compile the package. There are several iptables modules included; you can leave them all enabled or choose to compile and install only the ones needed. For this edit the mconfig file and leave only the ones you want:
build_CHAOS=m
build_DELUDE=m
build_DHCPADDR=m
build_ECHO=
build_IPMARK=m
build_LOGMARK=m
build_SYSRQ=m
build_TARPIT=m
build_TEE=m
build_condition=m
build_fuzzy=m
build_geoip=m
build_ipp2p=m
build_ipset=m
build_length2=m
build_lscan=m
build_quota2=m

Compile and install:
./configure --with-xtlibdir=/lib/xtables
make
make install

this will add the iptables extension /lib/xtables/libxt_geoip.so and the kernel module in /lib/modules/<kernel>/extra/xt_geoip.ko

4. Now we have to put the geoip database files under the expected location (/var/geoip); this is hardcoded in the code, but you can change it if really needed and recompile. I would like to add that even if this uses the same geoip source (the free GeoLite Country database) as the original geoip iptables module, but the format has changed. You can either get the database from the source, or build your own with the supplied script. Once you have that copy the files to /var/geoip

wget http://www.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
unzip GeoIPCountryCSV.zip
./runme.sh
cp -R var/geoip/ /var/

That’s it! All you have to do is use the module based on your needs. The syntax is the same as the original geoip iptables module:
[!] –src-cc, –source-country country[,country...] = Match packet coming from (one of) the specified country(ies)
[!] –dst-cc, –destination-country country[,country...] = Match packet going to (one of) the specified country(ies)
NOTE:  The country is inputed by its ISO3166 code.

We use something like this to mark and send each type of traffic to its own destination:
iptables -t mangle -A PREROUTING -p tcp -m geoip --src-cc US -d <IP> --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp -m geoip ! --src-cc US -d <IP> --dport 80 -j MARK --set-mark 2

I hope you found this article useful, and as me, are grateful that Xtables-addons project took over the patch-o-matic-ng broken modules and made them available on current distributions. Xtables-addons was also accepted in debian repository (in testing) and this will make it even simpler to install and use in the future.

iotop can be downloaded either as source package or a rpm package. Starting with lenny, debian includes iotop in the main repository and it can be installed just as simple as running:
aptitude install iotopThis is very cool indeed and kudos to the debian team to include iotop in lenny

For other distributions, we can install it from source using the following simple steps (be sure to have python2.5 before trying this); I am using git and this will download the latest trunk version 3, if this is not what you want download the tar.gz (ver 0.2.1) and use the stable version instead.
git clone git://repo.or.cz/iotop.git
cd iotop
./setup.py install
after this iotop will be installed under /usr/bin.

Let’s see what are the program parameters:

iotop -h
Usage: /usr/bin/iotop [OPTIONS]

DISK READ and DISK WRITE are the block I/O bandwidth used during the sampling
period. SWAPIN and IO are the percentages of time the thread spent respectively
while swapping in and waiting on I/O more generally. PRIO is the I/O priority at
which the thread is running (set using the ionice command).
Controls: left and right arrows to change the sorting column, r to invert the
sorting order, o to toggle the --only option, p to toggle the --processes
option, q to quit, any other key to force a refresh.

Options:
--version             show program's version number and exit
-h, --help            show this help message and exit
-o, --only            only show processes or threads actually doing I/O
-b, --batch           non-interactive mode
-n NUM, --iter=NUM    number of iterations before ending [infinite]
-d SEC, --delay=SEC   delay between iterations [1 second]
-p PID, --pid=PID     processes/threads to monitor [all]
-u USER, --user=USER  users to monitor [all]
-P, --processes       only show processes, not all threads

We can run it interactively, just like top, by simply typing: iotop, or add some parameters like
iotop -o -d 2
We can also run it in batch mode using -b:
iotop -b -o -d 5
...
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.49 M/s
TID PRIO USER      DISK READ  DISK WRITE   SWAPIN    IO    COMMAND
19215 be/1 mysql       0.00 B/s    1.24 M/s  0.00 %  2.58 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11670 be/1 mysql       0.00 B/s  152.28 K/s  0.00 %  0.64 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
11656 be/1 mysql       0.00 B/s    1.11 M/s  0.00 %  0.00 % mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

Other similar tools are iopp (presented in a previous post) and pidstat from newer sysstat packages (>= Sysstat 8.0.0). Stay tuned for a future post on here is how you can run iotop on debian etch (that uses by default python2.4).

Now this sounds interesting, and I am sure anyone that has dealt with i/o issues in the past will probably find this very useful. Let’s see how we can install it and what kind of reporting we get. We will install this from source and here are some quick steps to do this (you will need git and cmake for this):
git clone git://git.postgresql.org/git/~markwkm/iopp.git
cd iopp
cmake CMakeLists.txt
make

And install it:
make install DESTDIR=/usr
[100%] Built target iopp
Install the project...
-- Install configuration: ""
-- Installing /usr/bin/iopp
-- Installing /usr/share/man/man8/iopp.8
after this iopp will be installed into /usr/bin.

Let’s see what are the program parameters:
iopp -h
usage: iopp -h|--help
usage: iopp [-ci] [-k|-m] [delay [count]]
-c, --command display full command line
-h, --help display help
-i, --idle hides idle processes
-k, --kilobytes display data in kilobytes
-m, --megabytes display data in megabytes
-u, --human-readable display data in kilo-, mega-, or giga-bytes

And finally let’s test it:
iopp -i -u 5
pid rchar wchar    syscr    syscw reads writes cwrites command
2464    0B    0B        0        0    0B  8192B      0B kjournald
3364  515K    0B     1336        0    0B     0B      0B mysqld
4664    0B  134B        0        4    0B     0B      0B apache2
4685  803K  114K      454       80    0B     0B      0B collectd
20758   82K   82K      329      329    0B    84K      0B cronolog
26236   67K 3754B       59       27    0B     0B      0B apache2
...

The manual page explains each output field:

• pid: The process id.
• rchar: The number of bytes which this task has caused to be read from storage.
• wchar: The number of bytes which this task has caused, or shall cause to be written to disk.
• syscr: Count of the number of read I/O operations.
• syscw: Count of the number of write I/O operations.
• rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer.
• wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer.
• cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache.
• command: Filename of the executable.

Other similar tools are iotop and pidstat from newer sysstat packages and I will describe them in a future post.

1. Create a new RAID array

Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
or using the compact notation:
mdadm -Cv /dev/md0 -l1 -n2 /dev/sd[ab]1

2. /etc/mdadm.conf

/etc/mdadm.conf or /etc/mdadm/mdadm.conf (on debian) is the main configuration file for mdadm. After we create our RAID arrays we add them to this file using:
mdadm --detail --scan >> /etc/mdadm.conf
or on debian
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

3. Remove a disk from an array

We can’t remove a disk directly from the array, unless it is failed, so we first have to fail it (if the drive it is failed this is normally already in failed state and this step is not needed):
mdadm --fail /dev/md0 /dev/sda1
and now we can remove it:
mdadm --remove /dev/md0 /dev/sda1

This can be done in a single step using:
mdadm /dev/md0 --fail /dev/sda1 --remove /dev/sda1

4. Add a disk to an existing array

We can add a new disk to an array (replacing a failed one probably):
mdadm --add /dev/md0 /dev/sdb1

5. Verifying the status of the RAID arrays

We can check the status of the arrays on the system with:
cat /proc/mdstat
or
mdadm --detail /dev/md0

The output of this command will look like:

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
104320 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
19542976 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
223504192 blocks [2/2] [UU]

here we can see both drives are used and working fine – U. A failed drive will show as F, while a degraded array will miss the second disk -

Note: while monitoring the status of a RAID rebuild operation using watch can be useful:
watch cat /proc/mdstat

6. Stop and delete a RAID array

If we want to completely remove a raid array we have to stop if first and then remove it:
mdadm --stop /dev/md0
mdadm --remove /dev/md0
and finally we can even delete the superblock from the individual drives:
mdadm --zero-superblock /dev/sda

Finally in using RAID1 arrays, where we create identical partitions on both drives this can be useful to copy the partitions from sda to sdb:
sfdisk -d /dev/sda | sfdisk /dev/sdb

(this will dump the partition table of sda, removing completely the existing partitions on sdb, so be sure you want this before running this command, as it will not warn you at all).

There are many other usages of mdadm particular for each type of RAID level, and I would recommend to use the manual page (man mdadm) or the help (mdadm –help) if you need more details on its usage. Hopefully these quick examples will put you on the fast track with how mdadm works.

It is quite frequent to use separate IPs for various services on the same machine, and move those IPs to another server if needed. These are sometimes called portable IPs that can be migrated to any server in a particular colo/lan. This is done normally to minimized downtime and keep maintenance of such operations minimal (and to not rely on dns changes). Still arp caching on various network devices can cause big problems. Let’s assume we moved the IP from one server to another one in the same LAN to move away some service from our main web server. Taking down the IP from the existing server and bringing it up on the new server will complete our direct work if we don’t have access on the switches/routers in front of us. Again if you have control on all devices just connect to them and delete the arp cache for this ip to allow it to be re-cached on the new machine.

So after we have the IP moved on the new machine and now have to wait… The arp cache depends on the actual devices and can be anything from 5 minutes to not expire. Let’s assume for this example that the ip is 192.168.0.101 and after we run ifup and ifdown we have the IP correctly showing on the new server, srv02. If we don’t want to wait helpless for this to happen automatically, the solution is to broadcast from our new machine the arp with the source of the IP. Hopefully this will make the remote device to verify and invalidate its existing cache entry. For this we can use arping; installation is simple as it should be in most modern linux distributions by default. On debian you would install arping just by running:
aptitude install arping
finally we use a command like:
arping -S <our_IP> -B
that will broadcast our source IP and direct it to the broadcast address (255.255.255.255) . If your arp command uses different parameters notations, you should looks for something similar (to set the source and ping the broadcast). In our example with the IP 192.168.0.101 we would use:
srv02:~# arping -S 192.168.0.101 -B
ARPING 255.255.255.255
--- 255.255.255.255 statistics ---
16 packets transmitted, 0 packets received, 100% unanswered
(stop it with CTRL-C once it is working).

Normally after this, all should be ok and the remote device should cache the new arp entry, invalidating the existing cached one. If this is not the case, then call your datacenter to minimize the downtime . I would always suggest testing this first and seeing what downtime to expect and if you can minimize it like this with arping, first try with a non-production test IP. Don’t do this with live, production IP/service until you know what to expect. I hope this post will help you if you will have to deal with a similar situation. If this doesn’t work as expected in your case, please let us know what devices you found problematic, and if you were able to use a different workaround.

Bcfg2 is a debian friendly project, meaning they provide inside the source package all what is needed to build a debian package very easy. We will use for this a debian etch system, but this should work on any debian based system.

1. Download

First we need to download the source package:
wget ftp://ftp.mcs.anl.gov/pub/bcfg/bcfg2-0.9.6.tar.gz
tar -zxvf bcfg2-0.9.6.tar.gz
cd bcfg2-0.9.6

2. Install dependencies

Next we need to install the dependencies necessary to build the package. Depending on the state of your system this might look different from the output bellow. You will need to have basic debian building tools; in case you don’t have them already install them now:
apt-get install devscripts build-essential fakeroot cdbs pbuilder

Since the package is prepared for multiple distributions types and versions of python, we will have to select that we want to use it for debian etch pycentral:
cd debian
./buildsys-select.sh pycentral
(the available choices are 2.3, 2.4 and pycentral)

Now we can move on and install the rest of the dependencies (python-dev, etc.):
cd ..
/usr/lib/pbuilder/pbuilder-satisfydepends

3. Build and deploy the package

Finally we can now rebuild the package:
debuild -us -uc
cd ..
ls *.deb
bcfg2-server_0.9.6-0.1_all.deb
bcfg2_0.9.6-0.1_all.deb
Probably you will want to put the resulted packages in a local debian repository to perform the upgrade easily on several systems, maybe even using bcfg2 itself .

The SkipFiles parameter is by default empty, and in order to use it we have to add the appropriate config in our awstats.conf file.  For example to ignore a file called somefile.php we will add “/somefile.php“:
SkipFiles="/somefile.php"
while if we want to ignore all the pages in a folder called somefolder we will have to use a regex value like: “REGEX[^\/somefolder]“
We can add several rules separated by spaces:
SkipFiles="/somefile.php REGEX[^\/somefolder]"

Note: this will be effective only for new updates; meaning existing data will not be affected by this (if really needed you will have to regenerate your stats to get rid of those pages in old stats).