Varnish is a state-of-the-art, high-performance HTTP accelerator. It uses the advanced features in Linux 2.6, FreeBSD 6/7 and Solaris 10 to achieve its high performance. I will not go over the installation of varnish here, but I would highly recommend to use the latest version available at this time 2.0.4 as older versions have various issues.

We could try to use something simple like this in a varnish vcl:

backend s3 {
   set backend.host = "my_bucket.s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.backend = s3;
     lookup;
   }
}

but unfortunately this will not work. The Amazon S3 servers will look into the hostname passed by the request and this will most likely be different than the amazon bucket (something like static.mydomain.com) and hence will return 403 on any such request.

There are several solutions to make this work correctly, and the first one I will present is going to insert the bucket name in the actual url passed to the S3 backed. This looks like:

backend s3 {
   set backend.host = "s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.url = regsub(req.url, "^", "/my_bucket");
     set req.http.host = "localhost";
     set req.backend = s3;
     lookup;
   }
}

this will work fine, inserting the bucket name in the actual url passed to the backend. Still I don’t like this solution very much as it changes the consistency between the urls (direct one and the forwarded one) so here is a much better solution:

backend s3 {
   set backend.host = "s3.amazonaws.com";
   set backend.port = "80";
}

sub vcl_recv {
   if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
     set req.http.host = "my_bucket.s3.amazonaws.com";
     set req.backend = s3;
     lookup;
   }
}

As we can see, we are setting the http host the the one Amazon S3 servers would expect for our bucket. So we can keep the same url and don’t mess with the actual link we are passing.

A complete varnish vcl configuration to use with the Amazon S3 backend might look like this:

backend s3 {
.host = "s3.amazonaws.com";
.port = "80";
}

sub vcl_recv {
    if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") {
        unset req.http.cookie;
        unset req.http.cache-control;
        unset req.http.pragma;
        unset req.http.expires;
        unset req.http.etag;
        unset req.http.X-Forwarded-For;

        set req.backend = s3;
        set req.http.host = "my_bucket.s3.amazonaws.com";

        lookup;
    }
}

sub vcl_fetch {
    unset obj.http.X-Amz-Id-2;
    unset obj.http.X-Amz-Meta-Group;
    unset obj.http.X-Amz-Meta-Owner;
    unset obj.http.X-Amz-Meta-Permissions;
    unset obj.http.X-Amz-Request-Id;

    set obj.ttl = 1w;
    set obj.grace = 30s;
} 

If you found this post interesting, stay tuned for future posts on varnish and how to use it in more complex setups .

The pecl memcache php extension has supported for a long time the memcache session.save_handler, but with the release 3.0.x (still in beta at this time) this brings in a set of interesting features for us:
- UDP support
- Binary protocol support
- Non-blocking IO using select()
- Key and session redundancy (values are written to N mirrors)
- Improved error reporting and failover handling

Installing the php memcache module is very simple and can be done either by using distribution repositories (the version we want to use 3.0.x will probably not be available) or by using pecl or manual compilation:

Using pecl:
pecl install memcache-3.0.4

or manually:
wget http://pecl.php.net/get/memcache-3.0.4.tgz
tar xvfz memcache-3.0.4.tgz
cd memcache-3.0.4
phpize
./configure
make
make install

Finally, we need to activate the module in php.ini. I normally prefer to create a new file for this memcache.ini inside the include directory of the php build (for ex. in debian this is under /etc/php5/conf.d/memcache.ini) like this:
extension=memcache.so
memcache.allow_failover = 1
memcache.redundancy = 1
memcache.session_redundancy = 2

To use memcached to store the php sessions we will have to edit php.ini and replace the default file handler settings with something like:
; Use memcache as a session handler
session.save_handler = memcache
; Use a comma separated list of server urls to use for storage:
session.save_path="udp://<memcache_server>:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

or if we don’t want to use this serverwide, we can just define it inside a php block like this:

$session_save_path = "tcp://<memcache_server1>:11211?persistent=1&weight=1&timeout=1&retry_interval=15, tcp://<memcache_server2>:11211";
ini_set('session.save_handler', 'memcache');
ini_set('session.save_path', $session_save_path);

Note: as you can see I used in the above example tcp and udp also. Please be sure that your memcached server has udp support enabled if you want to use that. Also ensure that the web server can connect to the memcache server/port (use proper firewall rules to allow this) in order for this to work.

After restarting apache, it will start using memcache to store the php sessions. If redundancy is needed (why not?) we will probably want to use the internal php memcache support to save the sessions to more servers, or if you prefer you can use an external solution to replicate the memcached server data – repcached.

• Enhanced scalability: the Google SMP enhancement for synchronization
• More efficient memory allocation: ability to use platform allocator tuned for multi-core systems
• Improved out-of-the-box scalability: unlimited concurrent thread execution by default
• Dynamic tuning: at run-time, enable or disable insert buffering and adaptive hash indexing

wow… now this is indeed some great news for innodb users… I am writting this, and still I can’t believe that they’ve included the Google SMP patch in their official release. I can only assume that alternative projects as XtraDB, Drizzle, Percona patches, Google patches, etc. made Oracle to look back and try to do something with innodb besides the regular bug fixes. Even if we already use several of the great ‘unofficial alternatives’ this is good news for everyone.
Way to go Oracle! and looking forward for future performance improvements in the official innodb plugin; including existing patches that are out there already for sometime is a good start, but internal improvements from the innodb team would be also great .

Here are some performance results based on their own tests:
http://www.innodb.com/innodb_plugin/plugin-performance/

CloudFront is the answer to all users’ requests about using S3 as a CDN, delivering the content using a global network of 14 edge locations. CloudFront uses S3 to store the original file, and caches copies of the content close to end users locations, lowering latency when they download the objects.

Amazon CloudFront uses the following edge locations:
United States
* Ashburn, VA
* Dallas/Fort Worth, TX
* Los Angeles, CA
* Miami, FL
* Newark, NJ
* Palo Alto, CA
* Seattle, WA
* St. Louis, MO
Europe
* Amsterdam
* Dublin
* Frankfurt
* London
Asia
* Hong Kong
* Tokyo

CloudFront advantages:

• simple to implement; uses S3 as a ‘backend’;
• cost effective – pay only for what you use; priced very well just as S3 with prices starting at $0.170 per GB for content delivered in the US and Europe, and $0.210 per GB for content delivered in Asia;
• reliable - even though this is launched as beta and there is no SLA, we can expect to have a very reliable service from Amazon built on the experiences of s3 and ec2.

CloudFront disadvantages:

• this is a http only service; if you will need https for ex. you will not be able to do that.
• no control over caching; CloudFront will cache the file from your S3 bucket and serve it based on the closest dns location; this cache can expire in case of infrequent used files.
• no stats (besides the aws bill of course ).
• this is not trying to compete with the big CDN solutions out there, as it will be hard to match their features, but to provide a simple and cost effective solution that everybody can use.

In conclusion, this is great news from Amazon, and I am sure that even as I am writing this, many users that are serving their content from S3 have just finished switching over to CloudFront. For more details about CloudFront check out the AWS CloudFront page.

Normally if this would have been a small ISP I am sure people would have ignored their users and the users would have complained back to the ISP that they can’t reach some big sites, and in the end the ISP would have found a friendlier solution for this. Since this is AOL and they have a huge base of clients, we can’t really ignore them and we have to find a solution ourselves.

The best solution on the long term is to rewrite your application so it will no longer be persistent dependent (require that each user request to be processed by the same real server). Unfortunately this is not always possible, or it can take a long time to complete, meaning you need a work-around in the meantime.

As shown in my previous post we can increase the network mask in order to try to keep a bigger range of IPs on the same real server assuming that the remote user IPs will be at least on the same range. This might help for smaller proxy networks, but for AOL this will not be very useful as they will come from various completely different ips. Without being able to make decisions about the content of the packet LVS will not be able to solve this issue. The solution presented here is a work-around and to my knowledge it is the only possible solution when using LVS. We will mark all AOL proxy IPs traffic with iptables and then create for them a separate VIP using FWMARK. Obviously this will break any balancing and will send all the AOL users to a single server.
Note: If your AOL traffic is too big to handle for one server, then this solution will not work for you and you will need a higher level proxy (L7) to make the persistence work for AOL users.

AOL publishes a list of their proxies ips: http://webmaster.info.aol.com/proxyinfo.html
Here is a sample bash script that will mark all the packets coming from AOL proxies and mark them to have them handled by one real server (I just aggregated a few ranges to keep the list shorter ):

#!/bin/bash

iptables -t mangle -F

AOLPROXYS="64.12.0.0/16 149.174.160.0/20 \
152.163.0.0/16 195.93.0.0/18 195.93.64.0/18 198.81.0.0/16 \
202.67.64.128/25 205.188.0.0/16 207.200.112.0/21"

for aolproxys in $AOLPROXYS
do
iptables -t mangle -A PREROUTING -p tcp -s $aolproxys -d VIP/32 \
--dport 80 -j MARK --set-mark 1
done

And the ipvs FWMARK1 service definition:
ipvsadm -A -f 1 -s wrr -p 3600
ipvsadm -a -f 1 -r RS1 -g

resulting in all AOL traffic going to RealServer1. If you are using ldirectord you will probably want to setup also another server as a backup in case RS1 goes down. You still have to mark the traffic using iptables as shown above; the ldirectord.cf service definition should look like this:
virtual=1
real=RS1:80 gate
fallback=RS2:80 gate
persistent=3600
protocol=fwm
...

I am aware that this is not the nicest solution, but it was the only work-around I found if you have to use LVS and deal with AOL users on a persistent web application. If you find this useful don’t forget to check on the AOL proxy IP list and update your configs if needed in the future.

For more details you can check the LVS FAQ.

IPVS is an advanced IP based load balancing application implemented inside the linux kernel. Working at IP level LVS can’t make a decisions based on the content of the packet. Still, it can perform a basic IP affinity, by keeping all connections from the same source IP directed to the same real server for a configurable amount of time. This is achieved with the -p ipvsadm command parameter and takes as a parameter the time in seconds to keep the connections in the persistence table. From the ipvsadm manual page:

• -p, –persistent [timeout] = Specify that a virtual service is persistent. If this option is specified, multiple requests from a client are redirected to the same real server selected for the first request. Optionally, the timeout of persistent sessions may be specified given in seconds, otherwise the default of 300 seconds will be used. This option may be used in conjunction with protocols such as SSL or FTP where it is important that clients consistently connect with the same real server.

Here is a simple ipvsadm example in a setup with one web server load balanced on 2 real servers using LVS-DR:
ipvsadm -A -t VIP:80 -p 3600 -s -wrr
ipvsadm -a -t VIP:80 -R RS1 -g -w 1
ipvsadm -a -t VIP:80 -R RS2 -g -w 1
by using -p 3600 we request to keep the users coming from the same ip on the same real server as their first connection was handled, for a period of 60 minutes. After this timeout expires any new connection from the same user ip will be routed based on the scheduling policy (Weighted Round-Robin in this case) and can be handled by any of the active servers.

The output of the ipvsadm command will show that the service is persistent:
ipvsadm -L -n
...
TCP VIP:80 wrr persistent 3600
-> RS2:80 Route 1 0 0
-> RS1:80 Route 1 0 0

I am assuming that not many people are using plain ipvsadm commands and will probably use a frontend like ldirector that will handle health checks, automatic reconfiguration and many other features. The same effect is achieved by using the ldirectord persistent config keyword. The same sample in ldirectord.cf terms will look like:
virtual=VIP:80
real=RS1:80 gate 1
real=RS2:80 gate 1
persistent=3600
...

We can increase the network range used in the persistent match, using -M netmask. From the manual page:

• -M, –netmask netmask = Specify the granularity with which clients are grouped for persistent virtual services. The source address of the request is masked with this netmask to direct all clients from a network to the same real server. The default is 255.255.255.255, that is, the persistence granularity is per client host. Less specific netmasks may be used to resolve problems with non-persistent cache clusters on the client side.

In our sample setup this will look like:
ipvsadm -A -t VIP:80 -p 3600 -s -wrr -M 255.255.255.0
ipvsadm -a -t VIP:80 -R RS1 -g -w 1
ipvsadm -a -t VIP:80 -R RS2 -g -w 1

While for ldirectord this will require the usage of the netmask keyword:
virtual=VIP:80
real=RS1:80 gate 1
real=RS2:80 gate 1
netmask=255.255.255.0
persistent=3600
...

Note: obviously persistence will break the scheduling policy resulting in no longer having exact balancing on the real servers. The bigger the timeout and/or network masks for the LVS persistence, the bigger the effect.

For more details you check the LVS FAQ.