A while ago one of the major problems people faced to use Amazon EC2 into production environments was the dynamic state of the instances IPs. Every time one instance was started it was getting a new, dynamic IP. This has been addressed with the introduction of Amazon Elastic IP Addresses, but even when using this, the private IPs are still dynamic and most of the time people will want to communicate between several instances on the private allocated IPs and not on the public ones. This article will show how you can easily automate the process to update DNS hostnames for your EC2 instances, by adding to the AMI’s the logic for this. I will use for this a master DNS server running bind9, but this can be adapted to any other DNS server. Read the rest of this entry »

The geoip iptables extension allows you to filter, nat or mangle packets based on the country’s source or destination. This does exactly what the geoip apache module does, or the regular geoip binary, but at the iptables level. I would not go into the details why you would want to use that, but there are many ‘positive’ ways it can be useful… For example myself I use it in a project where we want to serve customized content for different countries. Since this is a high traffic site running on many web servers behind a loadbalanced setup, we prefer to split this at the loadbalancer level and not at apache level, to simplify our setup. We serve customized content to the US based visitors, while for the other countries we serve another international site.

Now this has been working fine for a long time now, using the original geoip module and patch-o-matic-ng method of installation (similar to what is very well described here). Still, this is unmaintained, and starting with kernel 2.6.22 it is no longer working. There is a patch that will make it work with a newer kernel, but if you run iptables 1.4.x this will again fail and even if there are some manual walkarounds this is still not the best solution.

The solution is called Xtables-addons. Xtables-addons is the successor to patch-o-matic-ng. Likewise, it contains extensions that were not, or are not yet, accepted in the main kernel/iptables packages. Xtables-addons is different from patch-o-matic in that you do not have to patch or recompile the kernel, sometimes recompiling iptables is also not needed.
The latest version 1.12 supports: iptables >= 1.4.1 and kernel-source >= 2.6.17.

Read the rest of this entry »

Even though at some point it looked like debian lenny will not have full xen support (for the 2.6.26 amd64 kernel) in the end this was fixed and lenny supports fully Xen ever on amd64. Upgrading from 2.6.18 to 2.6.26 is very straightforward (though we were using xen-hypervisor 3.2-1 already) and the only problem noticed was that the console on the domU machines was no longer working: it was showing the output correctly, but you could not enter anything on the console.

This is caused by the ‘new Xen console’ (xen now uses hvc0 for its console) and to fix it you have to add to your virtual machine xen configuration file one line: extra = “console=hvc0 xencons=tty”, restart the vm and it should be fine. In /etc/xen/<myvm>.cfg add this line:
extra = "console=hvc0 xencons=tty"

Read the rest of this entry »

Here is a quick tip that will show how you can get a small sample dataset from a mysql database using mysqldump. We frequently need to get a small snapshot from a very big production database to import it into a development or staging database that will not need all the original data; let’s say we need 1,000,000 records from all the tables in the database; we will just use the option –where=”true LIMIT X”, with X the number of records we want mysqldump to stop after.

Simply we will run something like (add whatever other options you need to mysqldump):

mysqldump --opt --where="true LIMIT 1000000" mydb > mydb1M.sql
Read the rest of this entry »

Mdadm is the modern tool most Linux distributions use these days to manage software RAID arrays; in the past raidtools was the tool we have used for this. This cheat sheet will show the most common usages of mdadm to manage software raid arrays; it assumes you have a good understanding of software RAID and Linux in general, and it will just explain the commands line usage of mdadm. The examples bellow use RAID1, but they can be adapted for any RAID level the Linux kernel driver supports.

1. Create a new RAID array

Create (mdadm –create) is used to create a new array:
mdadm --create --verbose /dev/md0 --level=1 /dev/sda1 /dev/sdb2
Read the rest of this entry »

The Address Resolution Protocol (ARP) is the method for finding a host’s link layer (hardware) address when only its Internet Layer (IP) or some other Network Layer address is known. ARP is a Link Layer protocol (Layer 2) because it only operates on the local area network or point-to-point link that a host is connected to. When we migrate one IP from a machine to another one, we might have problems caused by ‘arp caching‘. Various devices will cache the arp information for a specified amount of time and even after we moved the IP this will not be seen by some devices that will still use the cached information. I am talking about directly connected switches or routers, that we might have control or maybe not. If we have control on all the external devices, normally we just connect to the router or switch and remove the arp entry, forcing the device to query again for the information. This post will try to help in the situation where we don’t have direct control on the external devices (we are collocated or use rented servers in a remote datacenter, etc.), to minimize the downtime associated with this type of IP migration.

It is quite frequent to use separate IPs for various services on the same machine, and move those IPs to another server if needed. These are sometimes called portable IPs that can be migrated to any server in a particular colo/lan. This is done normally to minimized downtime and keep maintenance of such operations minimal (and to not rely on dns changes). Still arp caching on various network devices can cause big problems. Let’s assume we moved the IP from one server to another one in the same LAN to move away some service from our main web server. Taking down the IP from the existing server and bringing it up on the new server will complete our direct work if we don’t have access on the switches/routers in front of us. Again if you have control on all devices just connect to them and delete the arp cache for this ip to allow it to be re-cached on the new machine.

Read the rest of this entry »

Here is a quick bash tip that might be useful if you need to use inside a bash script a check to see if a wildcard expression of files/folders exists or not. For example:

if [ -e /tmp/*.cache ]
then
echo "Cache files exist: do something with them"
else
echo "No cache files..."
fi
This is using -e (existing file check) that is working fine on individual files. Also the above code might seem to work fine if the result of the expression if one file; but if you have more files returned by the expression this will fail with the following error:
line x: [: too many arguments

Read the rest of this entry »

While working on setting up a monitoring solution for a big wordpress installation, I realized that the server-status url was not working as expected even if mod_status was configured correctly:

ExtendedStatus On
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from some_ips
</Location>

The .htaccess wordpress rules were taking over this and the server-status url was returning a page not found error from within wordpress. This was happening because of the way how the rewrite rules are setup to handle all the permalinks on the site, and for any non-existing file send it to index.php: Read the rest of this entry »

Trac uses an internal database to store its data, meaning some extra care needs to be taken when backing up a trac project. This means that we can’t just copy/tar/gz/etc the folder and we need to make sure the database is properly closed and no one is writing to it at that particular time. Of course this is useful for big installation, while for a small trac usage install, you can probably just tar gz the trac folder and be just fine. But even in that case, I would choose the safe method and do it right because it is so easy: the trac-admin command provides the hotcopy switch just for this.

Basically all you have to do is:
trac-admin <trac_project_path> hotcopy <backup_path>
and of course if you are familiar with svn you will notice that it is similar to the way you use svnadmin hotcopy
Read the rest of this entry »

This is rather silly, but I lost some time with it, so I thought to put it in a blog post as it might be useful to others. Ok, a few days ago I took over an existing cacti installation; I was working to improve it and bring in some other stuff I always do in a cacti installation (note to self, I should really put some blog posts about cacti templates I use, as that will definitely save me some time in the future). One problem I noticed was that there was a traffic interface graph that was wrong when it was going over 114Mbps. I knew what the problem was and it should have been a quick fix.

All I had to do was to change from regular 32-bit counters to 64-bit counters and increase the maximum value to 1M. I have done that, but to my surprise the graph stopped working completely for the respective interface. So I had to dig into it, and debug the problem. I was able to see that the cacti poller snmp query was failing and returning:
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: IF-MIB::ifHCInOctets.10124

Read the rest of this entry »