Chef 11 was released earlier in February and it is awesome! Like most people, I love the new features like partial search, chef-apply and knife-essentials inclusions, awesome formatted output, etc. Of course the open source chef 11 server was rewritten completely in erlang with postgresql/mysql support replacing the ruby/couchdb backend stack. solr and rabbitmq are still there ;)… There are many breaking changes meaning you will want to make sure that you fix your cookbooks before upgrading.

When you are ready to upgrade, you will notice that unfortunately there is no official migration path. This howto will document what I’ve used myself for such migrations and hopefully will help you too if you are trying to perform a similar upgrade.

Opscode has done an amazing job with the omnibus installers and starting with Chef 11, the chef server has support for this also. Meaning you can install a new chef server simply by installing the rpm or deb for your platform and everything should be installed for you (ruby/gems, chef, rabbitmq, solr, erlang, postgresql, nginx). Just head over to http://www.opscode.com/chef/install/ and from the chef-server tab download the version for your OS.

In order to migrate to a new chef server we need to migrate from the old server:

• clients
• nodes
• roles
• environments
• data bags
• cookbooks (with all the versions used in each environment)

It is important to have all the clients with their proper public keys because if not we would have to re-register each one of them.

Personally, I’ve migrated using this process several servers from open source chef 0.10.x to chef 11, but theoretically this should work from any chef server implementation (hosted, private, etc.) because we are downloading and uploading the assets using the api calls.

While working on migrating a chef server from 0.10.x to version 11, I ended up extending the BackupExport and BackupRestore plugins written by Steven Danna and Joshua Timberman and added support for cookbooks and clients. Currently knife-backup has support for the following objects:

• clients
• nodes
• roles
• environments
• data bags
• cookbooks and all their versions.

knife-backup will backup all cookbook versions available on the chef server. Cookbooks are normally available in a repository and should be easy to upload like that, but if you are using various cookbook versions in each environment then it might not be so trivial to find and upload them back to the server; downloading them and having them available to upload like that is simple and clean. If you have too many cookbook versions then you might want to cleanup them first using something like knife-cleanup.

I’m working on many projects where we have a process that will make sure that every change we introduce in the cookbooks enters as a new version and where we use extensively environments to select what versions of cookbooks we want to use in each environment. This sounds like a great idea, and a workflow that I would highly recommend to anyone for sure. Still, after a while, the result is that you will end up with hundreds maybe even thousands of cookbook versions and most of them are unused (besides the few ones that you are referencing in each environment and maybe the latest ones). Normally I would not care about this and as long as it is not causing performance issues you should not care about it either. Still you must admit that when debugging any problems, it will make it more complex with all those versions everywhere; see bellow an example.

It’s been a little more than a year since I stepped up and became one of the organizers of the Bay Area Chef user group, trying to help my good friend Rob Berger as he was getting swamped with work and could not dedicate as much time to this, as he used to in the past. This post is meant to be a quick review on what happened during this time, what worked well and of course some ideas on how we can improve this in the future. I’m also hoping to get feedback from our members on what we can do differently in the future to better serve them and make this an even better group.

One of the first things we’ve done last year was to introduce the Chef Cafes. These are small events (we have a max limit of 10 people set for them) done consistently at the same time (1st and 3rd Thursday of the month) at the best coffee in Mountain View (Red Rock Coffee) with the intent to facilitate the interaction between people, give them a place where they can regularly meet and discuss about chef, ask questions and also try to help other members in the spirit of the open source community. The first Chef Cafe was on March 1st 2012 and it was just me and Rob (we had a good time preparing the future events and just catching up). But after that, we had 16 Chef Cafe’s all year long and many of them had 10 or even more people, and each one of them was unique and special in its own way. We had some, where we had new chef users that had various questions on how to use chef and we tried to help them and resolve their blocks in understanding and getting up to speed with chef. On the other hand we had other cafes where we had really advanced uses that brainstormed about various unresolved problems and what was their take on things like cookbook testing, workflow or orchestration. Overall, I think it was a great success and allowed us to be more connected with members, and also more open and helpful to new chef users.

In 2013 we look forward to your suggestions on how we can improve the Chef Cafes and we will try to keep these going. We hope to be able to move one in San Francisco and keep the other one in the South Bay as we had various requests for that. So if you are in the City and you want to get involved with this please ping me.

For a while now, I wanted to migrate my blog from Wordpress to Octopress, but for some reason I kept putting it on the shelf and not doing it. (let’s say because of all those client related projects…). Finally last weekend I’ve completed the migration and I’m really excited to get back to blogging after this. This post is meant to capture some of the issues I’ve encountered during the migration and how to fix them. This is not a full how to migrate post, as there are many such great articles available already.

Migrate old blog posts.

Believe it or not, I had 364 blog posts when I started the migration. Meaning a lot of energy was spent in importing those old articles. I’ve used exitwp to convert the wordpress-xml export of the blog posts; and this produced a reasonably good result. Still I had to run some fixes…

• for code blocks:

1

 perl -pi -e 's/([^\`]|^)(\`)([^\`]|$)/$1\n\`\`\`\n$3/g' *

• to enable comments (as ‘comments: true’ was missing from all posts)

1
2

find source/_posts/ -type f -print0 | xargs -0 -I file sed -i '' '2 i \
  comments: true' file

Categories/Tags/URLs

Enabled the octopress category list plugin and tags plugin, that you can see in the sidebar. Since I had already tags and categories on all posts it was very important to keep the same urls and not break them. Same thing for regular posts urls. Here are the relevant settings form the octopress config file:

1
2
3
4
5
6

root: /
permalink: /:year/:month/:day/:title/

category_dir: category

tag_dir: "tag"

Just keep in mind that if you have many tags as I do, the generation of the pages will increase a lot after you enable the tags plugin. You’ve been warned!

After completing the migration of my blog from Wordpress to Octopress I had the surprise that Disqus comments were not showing up on the site. I’ve already migrated in advance to Disqus and the Wordpress blog was working just fine with the new format. However, once switched to Octopress there were no comments active on the site. Strangely, the total number of comments for each post on the index page was showing just fine, but once you clicked on any post there were no comments. I tested adding new comment and it did show up correctly in Disqus.

Trying to understand and debug this issue, I looked in source/_includes/disqus.html and found the code that is generating the javascript variable disqus_identifier for the posts:

This week Opscode hosted its inaugural user conference here in San Francisco, and it was an awesome event enjoyed by all chef fans. Even if this was the first one (they are already planing for the future ones), this was by no means a small event, with more than 400 people attending and the workshops that ran on Tuesday sold out.

Even if I have not attended any workshop (they had 2 flavors, one targeted towards a sysadmin workflow and one for developers) the general feeling from people I talked with and attended them was that it was a very good experience, with a lot of hands-on practical examples. Tuesday afternoon, myself I attended the ”ChefConf Pre-event Hackday: TEST ALL THE THINGS!!!“ organized by Bryan Berry and it was great, and showed how many people are interested in testing their infrastructure as code; it was focused on cookbook testing (unit testing and integration testing), continuous integration with jenkins, and other things like that ;)

The first full day of ChefConf was Wednesday. The conference was structured with main presentations during the mornings and breakout sessions in the afternoon (with 2 main tracks and also a vendor one). From the beginning you could tell that this will be a very well run conference, and even if this was the first one, people like Jesse Robbins have a lot of experience running such events. Not surprisingly ChefConf kicked off with Adam Jacob’s ”State of the Union Part 1: Chef, Past and Present” (video) ; Jesse Robbins talked about the community around chef and how this is a key part of Opscode strategy and their efforts to take this to the next level. He showed this very nice visualization of the commits to the chef github repo.

I just started working on a new project and as you would expect one of the first things I did was to download its git repository from github. These were just some scripts and should have been very small ~5M, but the clone from gitbhub took about one hour as the full repo folder was 1.5G… (with the biggest size under .git/objects/pack) Crazy… What was in the git repository history that would cause something like this? I assumed that at some point in time the repository was much bigger (probably from some file/s that don’t exist anymore), but how could I find out what were those files? And more important howto remove them from history? Well if you came here from a google search on “how to remove a file from git history” then you probably know there are plenty of docs and howtos on how to achieve this but from my experience none of them really worked. This is why I decided to document the steps needed to identify the file from the git repo history that is using all that space and to have it removed fully and bring the repository to a manageable size.

First we need to identify the file that is causing this issue; and for this we will verify all the packed objects and look for the biggest ones:

1

git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -5

(and grab the revisions with the biggest files). Then find the name of the files in those revisions:

1

git rev-list --objects --all | grep <revision_id>

Next, remove the file from all revisions:

1
2

git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>'
rm -rf .git/refs/original/

Edit .git/packed-refs and remove/comment any external pack-refs. Without this the cleanup might not work. I my case I had refs/remotes/origin/master and some others branches.

1

vim .git/packed-refs

Finally repack and cleanup and remove those objects:

1
2
3

git reflog expire --all --expire-unreachable=0
git repack -A -d
git prune

Hopefully these steps will help you completely remove those un-wanted files from your git history. Let me know if you have any problems after following these simple steps.

I’m packing for Boston and will be there next week for LISA11. This will be my second year as part of the LISA blogging team, and after how much I enjoyed LISA last year in San Jose I wouldn’t miss this one even if it is on the other side of the country. I’ve tried to finish as much work as possible to be able to focus on the conference ;) but for various reasons of course this was not quite possible, and actually during the first days I will even be on call… In anycase, I’m sure this is going to be a great week full of awesomeness. I will be blogging for the USENIX blog every day, so be sure to follow that for fresh articles from me and the other memebers of our team (Ben, Rikki and Matt).

If you are going to LISA11 in Boston next week, we should definitely meetup. Contact me on twitter or email.

The Limoncelli Test, was a very interesting presentation by Tom Limoncelli based on a blog post he wrote earlier this year. If you haven’t done it already I would strongly recommend to take the test and see how does your sysadmin team rank on “The Limoncelli Test”.

Recovering From Linux Hard Drive Disasters is Theodore Ts’o signature training material on what to do if you have any sort of hard drive failure and covers in depth details on how to recover from such disasters caused by software or hardware failures.

GameDay: Creating Resiliency Through Destruction (slides): I enjoyed very much Jesse Robbins presentation, where he draws parallels between two of his greatest passions: firefighting and operations. Watch the video.

SRE@Google: Thousands of DevOps Since 2004: Tom Limoncelli, describes the technologies and policies that Google uses to do what is (now) called DevOps. Watch the video.

One of the advantages of being a member of the LISA11 Blog Team is that I was able to talk and interview this year program co-chairs: Tom Limoncelli and Doug Hughes. This was a great honor for me especially since I’ve been a big fan of Tom’s work for many years. The full article is available on the USENIX blog: ”Tom Limoncelli and Doug Hughes Interview”

Also my colleagues from the LISA11 blogging team (Ben, Rikki and Matt) have done some very interesting interviews with some key people from LISA11 to get you prepared for the event. Check out the USENIX blog for more from us in the next week.

Here is also a quick intro of our team: ”LISA11 Next Week – Meet your blog team!”