This works in bash and uses the ‘suspend‘ key (CTRL+Z) and the bg – background and fg – foreground commands. Let’s say we were running an intensive rsync command, and are wanted to check if we still have the available space on the disk without opening a new ssh session (yes, I know):
rsync -ar server:/source/ /destination/
^Z
Stopped

Let it run in the background:
bg
[1] rsync -ar server:/source/ /destination/ &

Now we can run some other commands like du:
du -h

We can see the background process with ps or jobs:
jobs
[1] Running rsync -ar server:/source/ /destination/

And finally we can bring it back to foreground with fg:
fg

Note: this works only on the running ssh/bash session and it will be closed once you exit. Logout should warn about open/running jobs and that they will be lost if exit.

To see what versions of java we have installed on the system (from debian packages):
update-java-alternatives -l
java-1.5.0-sun 53 /usr/lib/jvm/java-1.5.0-sun
java-6-sun 63 /usr/lib/jvm/java-6-sun

We can see that the default version is 1.6 in my case (as it was the last installed):
java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing)

We can change the default version with: update-java-alternatives –jre -s <ver> , like:
update-java-alternatives --jre -s java-1.5.0-sun
and now the default is 1.5:
java -version
java version "1.5.0_22"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)
Java HotSpot(TM) Client VM (build 1.5.0_22-b03, mixed mode, sharing)

This is quite handy if you need to have multiple java versions installed, and need a quick way to change the default one (you can access any of them directly from their own path of course).

The Amazon documentation explains how to get this information, basically just by using simple http get requests on the ip: 169.254.169.254, like for ex (for the metadata index):
curl http://169.254.169.254/latest/meta-data/
or for the custom data:
curl http://169.254.169.254/latest/user-data

Eucalyptus supports this great feature (starting with v1.4), but we obviously need to target a different ip to retrieve this information (as the amazon ip has nothing to do with our internal cloud ). We need to use the cloud controller IP for the request and the port it is bound (by default 8773 if you have not changed it). This will look like this (you need to run it from inside the actual instance):
curl http://<CC_IP>:8773/latest/meta-data
block-device-mapping/
security-groups
ami-manifest-path
ancestor-ami-ids
public-keys/
reservation-id
ramdisk-id
public-keys/0/
ami-launch-index
kernel-id
instance-type
local-hostname
local-ipv4
hostname
product-codes
public-ipv4
instance-id
public-hostname
ami-id
placement/

For the user data we can start the eucalyptus instance with:
ec2-run-instances <EMI> -d "myhostname" ...other params...
and later we can then retrieve that information from inside the instance using:
curl http://<CC_IP>:8773/latest/user-data
myhostname

While the documentation for Eucalyptus is getting better with every new version there are still missing parts like this one, and hopefully people looking for this information will find this article useful.

Ok, for this particular upload I am sync’ing a few millions files in folders structured like this:
000/000/files..
000/001/files..
…
999/999/file…
the process was taking days with a single s3sync running, so I just put up a small script to run several toplevel folder s3sync’s at the same time. This reduced the time a lot and was a good walkaround for our problem. Here is the script used, in case it might help others:

#!/bin/bash

cd /source_top_folder

id=0
while [  $id -lt 999 ]; do
        sleep 10
        echo "."
        running=$(ps -ef | grep s3sync | grep ruby |wc -l)
        if [ $running -lt 20 ]; then
                lid=`printf "%03d" $id`
                echo "starting a new s3sync - $lid"
                /usr/bin/s3sync -p --no-md5 -v -s -r --progress --delete ./$lid/ my_bucket:$lid/ &
                let id=id+1
        fi
done

This will basically run 20 s3sync instances and start a new one everytime it is needed (if total running go bellow 20). I realize this is not perfect, but it has done its job for us for this particular project. Ideally s3sync would be able to run several parallel upload threads to be much faster, but until then you might use a similar solution if you have such a problem .

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).

Maybe this can be done with ls, but since I have not found a way to do this, I turned to find. Find allows me very simple to get the list of subdirectories with their full paths:
find /var -type d
/var
/var/backups
/var/lib
/var/lib/ucf
/var/lib/ucf/cache
/var/lib/vim
/var/lib/vim/addons
/var/lib/php5
/var/lib/iptraf
/var/lib/mysql-cluster
/var/lib/collectd
...
and to get also the owner/permissions we can get help from ls:
ls -dl `find /var -type d`
drwxr-xr-x 15 root     root      4096 2009-05-11 06:02 /var
drwxr-xr-x  2 root     root      4096 2009-06-05 06:25 /var/backups
drwxr-xr-x  8 root     root      4096 2009-05-11 06:02 /var/cache
drwxr-xr-x  3 www-data www-data  4096 2009-05-11 06:02 /var/cache/apache2
drwxr-xr-x  2 www-data www-data  4096 2008-09-08 05:08 /var/cache/apache2/mod_disk_cache
drwxr-xr-x  3 root     root      4096 2009-06-03 09:32 /var/cache/apt
drwxr-xr-x  3 root     root      4096 2009-06-03 09:32 /var/cache/apt/archives
drwxr-xr-x  2 root     root      4096 2009-06-03 09:32 /var/cache/apt/archives/partial
drwxr-xr-x  2 root     root      4096 2009-06-05 06:25 /var/cache/apt-show-versions
drwxr-xr-x  2 root     root      4096 2009-06-03 09:32 /var/cache/debconf
drwxr-xr-x  2 root     root      4096 2009-06-05 06:25 /var/cache/locate
drwxr-sr-x 42 man      root      4096 2009-06-05 06:25 /var/cache/man
...

And finally the oneliner that gives us all the folders that are owned by some other user as root:
ls -dl `find /var -type d` | grep -v root
drwxr-xr-x  3 www-data www-data  4096 2009-05-11 06:02 /var/cache/apache2
drwxr-xr-x  2 www-data www-data  4096 2008-09-08 05:08 /var/cache/apache2/mod_disk_cache
drwxr-s---  2 mysql    adm       4096 2009-06-05 06:25 /var/log/mysql
drwxr-sr-x  2 news     news      4096 2009-05-06 04:49 /var/log/news


time for awk now

Update: as per the comment bellow from chlovechek a much cleaner solution is:
find /var ! -user root -type d -ls

How to get the needed information (IPs, hostnames, etc.)

Amazon api provides us all the needed information. Any EC2 instance can get a lot of information about itself just by querying a web server using a REST-like API. Here is how we can get all the available metadata items:
curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups

We have a direct interest in the public-ipv4 and local-ipv4 variables, but as we can see Amazon is providing many other useful information that can be used inside the instance for various purposes. For more details checkout the amazon docs.

Public and private IPs

This means that for getting the public and private ips we only have to make two calls like this:
curl http://169.254.169.254/latest/meta-data/local-ipv4and
curl http://169.254.169.254/latest/meta-data/public-ipv4

Hostnames

We need a way to identify from inside the instance what hostname this should be configured. There are probably several ways to do this, but the most common is to specify this when the instance is started using the –user-data option of the ec2-run-instances command (or the short form -d). This will pass the custom data and make it available to the instance.

You will probably want to customize this based on your needs. Myself I assumed that I will use the same domain for all instances and I need to pass only the hostname. Since I don’t need any other parameters to the machine I can just do this:
ec2-run-instances <AMI> -d "myhostname" ...other params...
If you use more user data, then you will probably use it like “hostname=myhostname <other_variables>”. (in this case you will need to update the script bellow like this: HOSTNAME=`echo $USER_DATA | cut -f 1 -d , | cut -f 2 -d =`)

Now making a http request like this will give us the hostname this instance should have:
curl http://169.254.169.254/latest/user-data

DNS Server configuration

Now that we have available inside the EC2 instance all the needed information, we need a way to allow the instance to update the DNS server. As I mentioned before, I will use for this a bind9 server, but this can by any other DNS server that allows this action to be scripted somehow (either using some api calls, or any way to use dynamic DNS update). For bind9 we will use the nsupdate utility to update the DNS server securely using the dnssec key mechanism.

Personally I don’t use the full domain for this, but delegate two subdomains (to the same nameservers) like this:
; ec2 zones:
ec2.domain.com.      NS      ns1.domain.com.
ec2.domain.com.      NS      ns2.domain.com.
ec2-int.domain.com.  NS      ns1.domain.com.
ec2-int.domain.com.  NS      ns2.domain.com.
and allow update access only to those zones. Of course if you prefer that you can give direct access to your full domain zone.

Generate a key using the dnssec-keygen utility like this:
dnssec-keygen -a HMAC-MD5 -b 512 -n USER user.domain.com.
and this will create two files like this:
Kuser.domain.com.+157+47950.key
Kuser.domain.com.+157+47950.private

Using the information from the public key add to your dns server configuration the key:

key user.domain.com. {
algorithm HMAC-MD5;
secret "xAw7F/axmVSxsZ+V4LAZnkeYObjOaJjbVKf21Zl4WhxtRHdlhqWSeCdd fIVR6MhC8LSQoim7NfkWD2j7WT5AHw==";
};

where secret is the value from the public key, that in my example looks like this:

cat Kuser.domain.com.+157+47950.key
user.domain.com. IN KEY 0 3 157 xAw7F/axmVSxsZ+V4LAZnkeYObjOaJjbVKf21Zl4WhxtRHdlhqWSeCdd fIVR6MhC8LSQoim7NfkWD2j7WT5AHw==

Finally we need to allow update access for the key:

zone "ec2.domain.com"
{
type master;
file "/etc/bind/zone/ec2.domain.com";
allow-update { key user.domain.com.; };
allow-query { any; };
};

zone "ec2-int.domain.com"
{
type master;
file "/etc/bind/zone/ec2-int.domain.com";
allow-update { key user.domain.com.; };
allow-query { any; };
};

Bind will need to be restarted after making these changes.

Using nsupdate to update the hostname

Next we will need to upload the key we created on the EC2 image (later we will save it inside the AMI once all runs well) and test to see if it is working properly.
cat<<EOF | /usr/bin/nsupdate -k Kuser.domain.com.+157+47950.private -v
server ns1.domain.com
zone ec2.domain.com
update delete test.ec2.domain.com A
update add test.ec2.domain.com 60 A <some_IP>
show
send
EOF

If this is working properly we can move on and put all this toghether in a script that will be running at the instance start time. If not, go back and see in your dns server logs if there are any issues why this is not working.

Finally automation

Now we just have to put all the pieces together and using a simple script like this will do the job:
ec2-hostname.sh:

#!/bin/bash

#you will need to have the key available in the instance in the same dir as this script
DNS_KEY=Kuser.domain.com.+157+47950.private
DOMAIN=domain.com

USER_DATA=`/usr/bin/curl -s http://169.254.169.254/latest/user-data`
HOSTNAME=`echo $USER_DATA`
#set also the hostname to the running instance
hostname $HOSTNAME.$DOMAIN

PUBIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/public-ipv4`
cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server ns1.$DOMAIN
zone ec2.$DOMAIN
update delete $HOSTNAME.ec2.$DOMAIN A
update add $HOSTNAME.ec2.$DOMAIN 60 A $PUBIP
send
EOF

LOCIP=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
cat<<EOF | /usr/bin/nsupdate -k $DNS_KEY -v
server ns1.$DOMAIN
zone ec2-int.$DOMAIN
update delete $HOSTNAME.ec2-int.$DOMAIN A
update add $HOSTNAME.ec2-int.$DOMAIN 60 A $LOCIP
send
EOF

You will probably want to run this at boot time either from rc.local or creating an initscript for it. I put all these EC2 related stuff under /usr/local/ec2 and in this case I just call it from rc.local with a line like this:
/usr/local/ec2/ ; sh ec2-hostname.sh

If all runs as you wanted you will probably want to save your AMI to include the script that will automatically update the dns hostname at instance boot time.

Hopefully you found this article interesting and it will be a starting point to create your own dns update script based on your needs.

Now this has been working fine for a long time now, using the original geoip module and patch-o-matic-ng method of installation (similar to what is very well described here). Still, this is unmaintained, and starting with kernel 2.6.22 it is no longer working. There is a patch that will make it work with a newer kernel, but if you run iptables 1.4.x this will again fail and even if there are some manual walkarounds this is still not the best solution.

The solution is called Xtables-addons. Xtables-addons is the successor to patch-o-matic-ng. Likewise, it contains extensions that were not, or are not yet, accepted in the main kernel/iptables packages. Xtables-addons is different from patch-o-matic in that you do not have to patch or recompile the kernel, sometimes recompiling iptables is also not needed.
The latest version 1.12 supports: iptables >= 1.4.1 and kernel-source >= 2.6.17.

The installation is very simple and requires only the following steps exemplified on a debian lenny machine (kernel 2.6.26 and iptables 1.4.2):

1. Install the needed dependencies: kernel headers and iptables dev:
aptitude install linux-headers-2.6.26-1-amd64 iptables-dev
libtext-csv-xs-perl will be also needed if you plan to update the database (normally you will want this to be able to update the db from time to time):
aptitude install libtext-csv-xs-perl

2. Download the xtables-addons package and the supplied database (or the sources to build your own):
wget http://switch.dl.sourceforge.net/sourceforge/xtables-addons/xtables-addons-1.12.tar.bz2
wget http://jengelh.medozas.de/files/geoip/geoip_iv0_database-20090201.tar.bz2

3. Configure and compile the package. There are several iptables modules included; you can leave them all enabled or choose to compile and install only the ones needed. For this edit the mconfig file and leave only the ones you want:
build_CHAOS=m
build_DELUDE=m
build_DHCPADDR=m
build_ECHO=
build_IPMARK=m
build_LOGMARK=m
build_SYSRQ=m
build_TARPIT=m
build_TEE=m
build_condition=m
build_fuzzy=m
build_geoip=m
build_ipp2p=m
build_ipset=m
build_length2=m
build_lscan=m
build_quota2=m

Compile and install:
./configure --with-xtlibdir=/lib/xtables
make
make install

this will add the iptables extension /lib/xtables/libxt_geoip.so and the kernel module in /lib/modules/<kernel>/extra/xt_geoip.ko

4. Now we have to put the geoip database files under the expected location (/var/geoip); this is hardcoded in the code, but you can change it if really needed and recompile. I would like to add that even if this uses the same geoip source (the free GeoLite Country database) as the original geoip iptables module, but the format has changed. You can either get the database from the source, or build your own with the supplied script. Once you have that copy the files to /var/geoip

wget http://www.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
unzip GeoIPCountryCSV.zip
./runme.sh
cp -R var/geoip/ /var/

That’s it! All you have to do is use the module based on your needs. The syntax is the same as the original geoip iptables module:
[!] –src-cc, –source-country country[,country...] = Match packet coming from (one of) the specified country(ies)
[!] –dst-cc, –destination-country country[,country...] = Match packet going to (one of) the specified country(ies)
NOTE:  The country is inputed by its ISO3166 code.

We use something like this to mark and send each type of traffic to its own destination:
iptables -t mangle -A PREROUTING -p tcp -m geoip --src-cc US -d <IP> --dport 80 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp -m geoip ! --src-cc US -d <IP> --dport 80 -j MARK --set-mark 2

I hope you found this article useful, and as me, are grateful that Xtables-addons project took over the patch-o-matic-ng broken modules and made them available on current distributions. Xtables-addons was also accepted in debian repository (in testing) and this will make it even simpler to install and use in the future.

This is caused by the ‘new Xen console’ (xen now uses hvc0 for its console) and to fix it you have to add to your virtual machine xen configuration file one line: extra = “console=hvc0 xencons=tty”, restart the vm and it should be fine. In /etc/xen/<myvm>.cfg add this line:
extra = "console=hvc0 xencons=tty"

If you are using pygrub then in grub.conf on the domU add: console=hvc0 xencons=tty to the kernel line, like:
kernel /boot/vmlinuz-2.6.26-1-xen-amd64 ro root=/dev/sda1 console=hvc0 xencons=tty

Here are the xen packages lenny uses:

ii  libxenstore3.0                          3.2.1-2                    Xenstore communications library for Xen
ii  linux-image-2.6.26-1-xen-amd64          2.6.26-13                  Linux 2.6.26 image on AMD64, oldstyle Xen su
ii  linux-modules-2.6.26-1-xen-amd64        2.6.26-13                  Linux 2.6.26 modules on AMD64
ii  xen-hypervisor-3.2-1-amd64              3.2.1-2                    The Xen Hypervisor on AMD64
ii  xen-tools                               3.9-4                      Tools to manage Debian XEN virtual servers
ii  xen-utils-3.2-1                         3.2.1-2                    XEN administrative tools
ii  xen-utils-common                        3.2.0-2                    XEN administrative tools - common files
ii  xenstore-utils                          3.2.1-2                    Xenstore utilities for Xen

Simply we will run something like (add whatever other options you need to mysqldump):

mysqldump --opt --where="true LIMIT 1000000" mydb > mydb1M.sql
and this will get 1M records from each of the tables in the database. If you want this for a single table you would use something like this:

mysqldump --opt --where="true LIMIT 1000000" mydb mytable > mydb_mytable_1M.sql

To restore this, you would use the same as on a regular dump:

mysql -p mydb_stage < mydb1M.sql

This will give you a small number of records that you can use for development, testing, etc. whatever you would need.