This week Opscode hosted its inaugural user conference here in San Francisco, and it was an awesome event enjoyed by all chef fans. Even if this was the first one (they are already planing for the future ones), this was by no means a small event, with more than 400 people attending and the workshops that ran on Tuesday sold out.

Even if I have not attended any workshop (they had 2 flavors, one targeted towards a sysadmin workflow and one for developers) the general feeling from people I talked with and attended them was that it was a very good experience, with a lot of hands-on practical examples. Tuesday afternoon, myself I attended the “ChefConf Pre-event Hackday: TEST ALL THE THINGS!!!“ organized by Bryan Berry and it was great, and showed how many people are interested in testing their infrastructure as code; it was focused on cookbook testing (unit testing and integration testing), continuous integration with jenkins, and other things like that

The first full day of ChefConf was Wednesday. The conference was structured with main presentations during the mornings and breakout sessions in the afternoon (with 2 main tracks and also a vendor one). From the beginning you could tell that this will be a very well run conference, and even if this was the first one, people like Jesse Robbins have a lot of experience running such events. Not surprisingly ChefConf kicked off with Adam Jacob‘s “State of the Union Part 1: Chef, Past and Present” (video) ; Jesse Robbins talked about the community around chef and how this is a key part of Opscode strategy and their efforts to take this to the next level. He showed this very nice visualization of the commits to the chef github repo.

Read the rest of this entry »

I just started working on a new project and as you would expect one of the first things I did was to download its git repository from github. These were just some scripts and should have been very small ~5M, but the clone from gitbhub took about one hour as the full repo folder was 1.5G… (with the biggest size under .git/objects/pack) Crazy… What was in the git repository history that would cause something like this? I assumed that at some point in time the repository was much bigger (probably from some file/s that don’t exist anymore), but how could I find out what were those files? And more important howto remove them from history? Well if you came here from a google search on “how to remove a file from git history” then you probably know there are plenty of docs and howtos on how to achieve this but from my experience none of them really worked. This is why I decided to document the steps needed to identify the file from the git repo history that is using all that space and to have it removed fully and bring the repository to a manageable size.

First we need to identify the file that is causing this issue; and for this we will verify all the packed objects and look for the biggest ones:
git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -5
(and grab the revisions with the biggest files). Then find the name of the files in those revisions:
git rev-list --objects --all | grep <revision_id>

Next, remove the file from all revisions:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>'
rm -rf .git/refs/original/

Edit .git/packed-refs and remove/comment any external pack-refs. Without this the cleanup might not work. I my case I had refs/remotes/origin/master and some others branches.
vim .git/packed-refs

Finally repack and cleanup and remove those objects:
git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune

Hopefully these steps will help you completely remove those un-wanted files from your git history. Let me know if you have any problems after following these simple steps.

Read the rest of this entry »

I’m packing for Boston and will be there next week for LISA11. This will be my second year as part of the LISA blogging team, and after how much I enjoyed LISA last year in San Jose I wouldn’t miss this one even if it is on the other side of the country. I’ve tried to finish as much work as possible to be able to focus on the conference but for various reasons of course this was not quite possible, and actually during the first days I will even be on call… In anycase, I’m sure this is going to be a great week full of awesomeness. I will be blogging for the USENIX blog every day, so be sure to follow that for fresh articles from me and the other memebers of our team (Ben, Rikki and Matt).

If you are going to LISA11 in Boston next week, we should definitely meetup. Contact me on twitter or email.

—–

The Limoncelli Test, was a very interesting presentation by Tom Limoncelli based on a blog post he wrote earlier this year. If you haven’t done it already I would strongly recommend to take the test and see how does your sysadmin team rank on “The Limoncelli Test”.

Recovering From Linux Hard Drive Disasters is Theodore Ts’o signature training material on what to do if you have any sort of hard drive failure and covers in depth details on how to recover from such disasters caused by software or hardware failures.

GameDay: Creating Resiliency Through Destruction (slides): I enjoyed very much Jesse Robbins presentation, where he draws parallels between two of his greatest passions: firefighting and operations. Watch the video.

SRE@Google: Thousands of DevOps Since 2004: Tom Limoncelli, describes the technologies and policies that Google uses to do what is (now) called DevOps. Watch the video.

One of the advantages of being a member of the LISA11 Blog Team is that I was able to talk and interview this year program co-chairs: Tom Limoncelli and Doug Hughes. This was a great honor for me especially since I’ve been a big fan of Tom’s work for many years. The full article is available on the USENIX blog: “Tom Limoncelli and Doug Hughes Interview”

Also my colleagues from the LISA11 blogging team (Ben, Rikki and Matt) have done some very interesting interviews with some key people from LISA11 to get you prepared for the event. Check out the USENIX blog for more from us in the next week.

Here is also a quick intro of our team: “LISA11 Next Week – Meet your blog team!“

Building packages is a task that every system administrator will end up doing. Most of the time this is not a very interesting task but someone has to do it, right? Normally you will end up modifying and tweaking based on your own needs an existing package that was built by the maintainers of the Linux distribution that you are using. In time you might even become familiar with the packaging system you are using (rpm, deb, etc.) and you will be able to write a spec file and start from scratch and build a new package if you need to. Still, this process is complicated and requires a lot of work.

Luckily, Jordan Sissel has built a tool called FPM (Effing Package Management), exactly for this: to ease the pain of building new packages; packages that you will use for your own infrastructure and you want them customized based on your own needs; and you don’t care about upstream rules and standards and other limitations when building such packages. This can be very useful for people deploying their own applications as rpms (or debs) and can simplify a lot of the process of building those packages.

FPM can be easily installed on your build system using rubygems:
gem install fpm

Once installed you can use fpm to build packages (targets):

• deb
• rpm
• solaris

from any of the following sources:

• directory (of compiled source of some application)
• gem
• python eggs
• rpm
• node npm packages

Read the rest of this entry »

Yesterday Opscode, the company behind Chef, announced the first ever chef cookbook contest. In order to participate in the contest you will need to write a new cookbook and submit it by the end of September; this is going to be a little tricky as there are many cookbooks already available on the community site. So this is a great idea and it will take care of the few applications that don’t already have chef cookbooks. The cookbooks which shows off the awesome Chef features will have better chances to win. The prizes are also interesting: iPad, gift cards, etc. Here are the full details and rules of the contest: http://www.opscode.com/blog/2011/08/22/cookbook-contest/

So if you have an idea for a chef cookbook, now it’s the time to start working on it. I’m offering my help for free for all my blog readers: I will help you write a cookbook by implementing your ideas; help reviewing it or suggest improvements, or whatever else you might need help with. Use the contact form to email me (or DM me on twitter) and let me know how I can help.

If you don’t have time to write a new cookbook but you have a great idea for a cookbook that is missing from the opscode community site, please post it bellow in the comments section and I’m sure some of my blog readers will help create it.

Again this is a brilliant idea from Opscode and it creates a win-win situation for everyone. I’m just curious, is this the first idea from their new community manager? If this is the case, great job Jesse .

If you used vagrant (great tool, right?) you have probably downloaded a basebox from some remote location to get you started. This is a great quick start, and there are many good boxes out there that you can use; vagrantbox.es does a great job in listing various public vagrant boxes. But if you are like me, you probably will want to customize the boxes you are using; you might want to install them from scratch based on your own little/or/big customizations. Well if you are like that, then you will be happy to hear that Patrick Debois had exactly the same problem when he decided to write veewee. And veewee is exactly that missing part of vagrant that allows you to easily build your own vagrant boxes from scratch.

So let’s see how we can use veewee. I’m assuming you already have vagrant installed (and virtualbox), but if you don’t please install them first. To install veewee we just have to install the veewee gem:
gem install veewee
once you installed veewee you can see a new task added to vagrant: basebox.

Read the rest of this entry »

Yesterday evening I presented at the SF Bay Area Large-Scale Production Engineering meetup group at Yahoo HQ a talk about “Monitoring with Icinga”. This was an introductory talk intended to bring awareness about icinga (there were only 3-4 people from the audience of about 75 that heard of it before), and I think it reached its goal very well; afterwards there were many people interested to try it out and had various questions about it at the end. I was also very happy to have Matthew Brooks one of the icinga core developers in the audience and backing me up to some of the more difficult questions people had. Thanks again Matthew for coming! Here are the slides from my presentation:

@LSPEMeetup made available the video on justin.tv; unfortunately the quality of the video/sound is not the best; you can find it here.

Recently I’ve worked on a project where we deployed a bunch KVM instances. Immediately we noticed horrible IO performance on all the guests instances. In this particular case the hosts and the guests were all Ubuntu 10.04 Lucid and were created with vmbuilder without any special settings using the ubuntu defaults. Here is a sample command similar to what we used to build the kvm images:

vmbuilder kvm ubuntu --suite=lucid --flavour=virtual --arch=amd64 --mirror=http://en.archive.ubuntu.com/ubuntu -o --libvirt=qemu:///system --ip=10.0.0.11 --gw=10.0.0.1 --part=vmbuilder.partition --templates=mytemplates --user=username --pass=password --firstboot=/var/vms/vm1/boot.sh --mem=1024 --hostname=myhost --bridge=br0

Now even if we haven’t tuned anything I would have expected it to perform at least the same level or even better compared with a Xen instance. Still, this was not the case, and the performance was really horrible and any kind of IO bound tasks would effectively lock the instance. Looking into this and trying to understand what was the problem I was able to isolate this issue happening only on instances that had ext4 as the filesystem (the default for lucid), but strangely enough this didn’t happen for an older instance that was build with ext3 (actually a debian lenny instance). All the images build with the above command will use qcow2 sparse format as the default format for the disk.

Read the rest of this entry »

A few days ago Opscode released a security fix for chef server 0.10.0 and 0.9.16 and this post will show how upgrade to chef-server 0.10.2. First start by backing up your data. Seriously. In the past I’ve had serious problems when performing similar upgrades (even a minor one like this that looks harmless), and even if now opscode are much better with this process it never hurts to be precautions. Since I use a rubygem install the next steps will focus on this type of installation; if you are using distribution or opscode packages this will not be very helpful as probably packages are not yet available for this upgrade; once they will replace the gem upgrade part with the deb/rpm upgrade and you should be set.

1. Stop all the chef related services

Here is a handy command that will stop all the possible chef server related services:
for svc in server server-webui solr expander
do
sudo /etc/init.d/chef-${svc} stop
done

Read the rest of this entry »