Amazon just announced the limited beta of Amazon Virtual Private Cloud (Amazon VPC), a secure and seamless bridge between existing IT infrastructures and the AWS cloud. Amazon VPC enables us to connect our existing infrastructure to a set of isolated AWS compute resources via a Virtual Private Network (VPN) connection.

“Amazon VPC enables you to use your own isolated resources within the AWS cloud, and then connect those resources directly to your own datacenter using industry-standard encrypted IPsec VPN connections. With Amazon VPC, you can:

• Create a Virtual Private Cloud on AWS’s scalable infrastructure, and specify its private IP address range from any block you choose.
• Divide your VPC’s private IP address range into one or more subnets in a manner convenient for managing applications and services you run in your VPC.
• Bridge together your VPC and your IT infrastructure via an encrypted VPN connection.
• Add AWS resources, such as Amazon EC2 instances, to your VPC.
• Route traffic between your VPC and the Internet over the VPN connection so that it can be examined by your existing security and networking assets before heading to the public Internet.
• Extend your existing security and management policies within your IT infrastructure to your VPC as if they were running within your infrastructure.”

Besides the regular ec2 prices, we will have to pay for the VPN connection ($0.05 per VPN Connection-hour) and for the data transfer using the VPN tunnel ($0.10 per GB IN – and starting with $0.17 per GB OUT).

This is a great new service from Amazon that takes the cloud computing offerings to a new level. And this just when major competitors thought they were gaining on Amazon with their offerings, Amazon continues to innovate and launch great new services like this one .

I’ve just finished reading “Learning Nagios 3.0″ by Wojciech Kocjan and published by Packt Publishing, and this is a great book for anyone interested in nagios. This is a beginner level book that introduces nagios to new users interested in monitoring their infrastructure, but it will also present advanced features that even more experienced sysadmins can benefit from. All these in a pretty compact book, at 301 pages.

The topics are as follows:

• Introduction
• Installation and Configuration
• Using the Nagios Web Interface
• Overview of Nagios Plugins
• Advanced Configuration
• Notifications and Events
• Passive Checks and NSCA
• Monitoring Remote Hosts
• SNMP
• Advanced Monitoring
• Extending Nagios

Read the rest of this entry »

One of the great features of Amazon EC2 is the possibility to dynamically query and use instance specific metadata, or even custom data. This can be useful for various reasons, and the greatest advantage I’ve personally seen into this, is the possibility to allow the instance to have some information on how to configure itself when first booting (using chef or puppet, or some other configuration management tool).

The Amazon documentation explains how to get this information, basically just by using simple http get requests on the ip: 169.254.169.254, like for ex (for the metadata index):
curl http://169.254.169.254/latest/meta-data/
or for the custom data:
curl http://169.254.169.254/latest/user-data

Eucalyptus supports this great feature (starting with v1.4), but we obviously need to target a different ip to retrieve this information (as the amazon ip has nothing to do with our internal cloud ). We need to use the cloud controller IP for the request and the port it is bound (by default 8773 if you have not changed it). This will look like this (you need to run it from inside the actual instance): Read the rest of this entry »

s3sync is a great tool to synchronize local data with Amazon S3 for backups, or whatever other reasons you might want to put your data on S3. It is very simple to install (gem install s3sync) and use (s3sync -v -s -r –progress <source_dir> s3_bucket:<dir>); it runs very well and it can be easily scripted to do regular backups or even synchronize live data with S3. The only problem I found while using s3sync was that it can be very slow when uploading a lot of data (millions of files) to S3; this because the process is slow but also because it runs a single file at a time, and it doesn’t do several uploads in parallel. I would have loved for s3sync to do this out of the box, but unfortunately it doesn’t, but for my particular need I was able to do this by running more s3sync commands a the same time. It will not apply to your data (unless it is structured the same way as here; very unlikely), but it might give an idea on how you could do this your own data if it is structured in a feasible way.

Read the rest of this entry »

Many startups these days are using Amazon S3 to serve directly their static assets. S3 is being used as a simple CDN instead of more professional (and expensive) solutions (including Amazon’s own CloudFront) because it is very simple and cheap to use. Still if you have a high traffic site, this will no longer be so cheap since you will be paying for all those requests and the bandwidth. In such cases if you still want to use S3 for the storage advantage (like storing millions of files and see it as an unlimited storage space) but not have your bill go up like crazy, you can use a reverse proxy or web accelerator to cache your assets locally and reduce the number of direct hits on S3. We could use Squid or Varnish for this, and in this article I will show how we can configure Varnish for this. We are using varnish with S3 on various projects and it works very well, simplifying the setup and saving a lot of money in the Amazon S3 bill.

Varnish is a state-of-the-art, high-performance HTTP accelerator. It uses the advanced features in Linux 2.6, FreeBSD 6/7 and Solaris 10 to achieve its high performance. I will not go over the installation of varnish here, but I would highly recommend to use the latest version available at this time 2.0.4 as older versions have various issues.

We could try to use something simple like this in a varnish vcl:

backend s3 {
   set backend.host = "my_bucket.s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.backend = s3;
     lookup;
   }
}

Read the rest of this entry »

Many times you might want to fine tune the default permissions of the files created on a linux system. This is very simple and usually if you are using bash all you have to do is to define somewhere in the bash startup files (/etc/profile is a good place for this) a new value for umask like this:
umask 002
(this will allow by default group write permissions on the newly created files)

Normally on modern linux distributions this is by default set to 022 and you can easily find out what it is on your system by running the umask command:
umask

Contrary to what you might think, this is not enough to have this working for all applications and daemons on the system. This works fine for any files created from a shell session, but the files created by other processes, like the web server for example, will still use the default, unless otherwise configured. In order to have apache use a different umask we can define this inside /etc/apache2/envvars (debian, and ubuntu systems) or /etc/sysconfig/httpd (rhel,centos systems) like this:
umask 002
and restart apache to enable it.

Other daemons will have different locations where you can define this to overwrite the default setting for umask (check their documentation if you are unsure).