Comments on: Using Varnish in front of your Amazon S3 static content

By: hassan

hassan — Mon, 09 Jan 2012 15:30:36 +0000

thanks to your post, i’ve been successfully using varnish infront of amazon s3 storage. However, there is a catch. Varnish tends to cache images from s3 in memory. Since there’z limited RAM on my image server, what i need is another layer.

In short, what i need is a 3 tier structure ..
image request => varnish => local-filesystem => s3

i have tons of space available on the server.

Any ideas?

By: Adam Lindsay

Adam Lindsay — Thu, 13 Oct 2011 21:47:40 +0000

In newer versions of Varnish in vcl_fetch use beresp instead of obj.

By: François Deppierraz

François Deppierraz — Mon, 14 Sep 2009 08:46:52 +0000

@Marius: The description I wrote was not entirely correct, it’s actually
307 temporary redirects which are sometimes returned by S3. According to
their documentation, this could happen in case of routing changes in
their infrastructure.

http://docs.amazonwebservices.com/AmazonS3/2006-03-01/Redirects.html

It’s probably not a big deal if you only use Varnish to provide faster
access to your files stored on S3. But, in my case, it’s also used to
ensure that incoming requests have a valid Referrer header to prevent
hotlinking.

By: - Marius -

- Marius - — Thu, 10 Sep 2009 07:14:31 +0000

@François: what do you mean by “different cases S3 will return 302 Redirect responses”. what particular cases are you talking about?

By: François Deppierraz

François Deppierraz — Wed, 09 Sep 2009 15:47:55 +0000

Are you sure that this setup really cache files coming from S3 ?

In different cases S3 will return 302 Redirect responses which, instead
of being handled by varnish, are sent directly to the client. This
results in the client bypassing varnish for that particular query.

If you lookup “Handling Dynamic Backend Redirects” on google, you’ll
find a mail from Poul-Henning Kamp, varnish’s main author, which
basically says that varnish cannot understand such redirects.

Thanks for your comments on this issue.

By: Weekly Digest for September 5th | The WebZappr

Weekly Digest for September 5th | The WebZappr — Sat, 05 Sep 2009 07:06:01 +0000

[...] Shared Using Varnish in front of your Amazon S3 static content | MDLog:/sysadmin [...]

By: Using Varnish with Amazon’s S3 to server static content | AdminStories.com

Thu, 06 Aug 2009 17:46:39 +0000

[...] your interested in using Amazon’s S3 to server static content, check out “Using Varnish in front of your Amazon S3 static content” by Marius over at [...]