@LSPEMeetup made available the video on justin.tv; unfortunately the quality of the video/sound is not the best; you can find it here.

If you are in the area and want to meetup ping me on twitter or email.

Real-World Configuration Management Workshop: Sunday I’ve attended all day the CM workshop; this was an interesting workshop, where different people shared their experiences and pains in configuration management.

Time Management for System Administrators: Monday I attended  Tom Limoncelli’s tutorial on time management for system administrators. Very educational and inspiring. I will definitely revisit his book as its been a while since I’ve read it. As takeaways, I have at least 2-3 ideas that I’m really looking forward to implement and see how I can use them to improve my productivity.

OpsCamp: Tuesday I went to OpsCamp Silicon Valley - San Jose co-located with LISA10, and even if it was sponsored by LISA this was a standalone event. It was definitely a smaller event than expected (because of this we had only one round of sessions with everyone in the same room), but it was definitely interesting and I’ve met some very smart people with a lot of experience in operations and building high performance infrastructures.

Postfix: Past, Present, and Future: Wednesday started with the opening remarks by the program chair Rudi van Drunen, followed by the opening keynote by Tony Cass from CERN. Amazing stuff; if you have the chance to see the video do that . Afterwards, in the afternoon I went to the invited talks by Dinah McNutt, Google: “The 10 Commandments of Release Engineering” and Wietse Venema on “Postfix: Past, Present, and Future”. It was amazing to be able to meet in person and chat with the author of postfix, a program that I’ve been using for many years.

Last week I was in Baltimore for the inaugural edition of Surge, a conference organized by OmniTI. Ever since I signed up for Surge2010, I sow this as a conference of Velocity quality, only without the frontend track and focusing mostly on backend topics (what I was interested anyway), and in a much more distant location than Santa Clara . This sounded interesting enough to make me want to go, as the first conference I will go in 2010 outside Silicon Valley. Also I could not pass the opportunity to hang out with my friend Andy and sync’up with what we’ve been up to during all this time.

The first day started with John Allspaw‘s keynote (I’m a big fan of John’s) where he made a parallel between systems engineering and web operations. Next we had a second keynote, and I had the chance to see Bryan Cantrill (VP of Engineering at Joyent) for the first time speaking, and I was really impressed by his high energy. I don’t think I’ve seen any other speaker radiating so much energy, and it was hard not to follow him on his presentation about DIRT = “data-intensive real-time”. Still, I would say that his slides were way too overloaded and hard to read, but if you are such an energetic speaker this doesn’t matter much. Next, we attended Theo Schlossnagle‘s talk “Scalable Design Patterns”; I’m a big fan of Theo’s, but given the fact that OmniTI was hosting the event (his company) honestly, I was expecting more than the regular ‘velocity’ talk he gives. But in anycase, it was a good talk, and Andy liked his style a lot (he never sow him talk before).

After seeing two db scalability talks: “The most common MySQL scalability mistakes, and how to avoid them” and “Database Scalability Patterns”, I sow the most interesting talk of the day by Artur Bergman: “Scaling and Loadbalancing Wikia Across The World”. Takeaways from this talk were that you can use google analytics for something else than the regular stuff; I am already using it in some places to track 404s and other errors, but I never thought you can use it with some simple js to track the load time of the page. That was quite cool. Wikia’s central piece of software (besides mediawiki of course) is varnish. They use varnish for some of the strangest possible things, and this opened my eyes that varnish can be used for something else than what I’ve used it on some projects (plain reverse proxy): like cookie manipulations, geoip, custom C code, etc. If you have varnish questions then Artur is your man as they seem to have abused varnish in all the possible ways. Andy was in Tom Cook’s talk (Facebook) and was quite impressed, but since I sow it at Velocity 2010 I anticipated it will be the same. Strange Twitter was not here… hmm…

Another interesting talk was Rasmus Lerdorf‘s “PHP Performance Checklist”. This was basically a free course for how to speed up Drupal, though I’m sure there were no Drupal developers in the room . Rasmus demonstrated that PHP is quite fast and usually the code people complain is slow, is the problem. He also had many examples and comparisons for various projects like drupal, wordpress, etc. running with facebook’s hiphop. My suggestion to Aquia/Drupal devs: listen to Rasmus talk and get some awesome free advice on how to make your software faster. .

Finally the day closed with the SQL vs NoSQL panel; nothing spectacular and most people agreed that NoSQL is a good solution for various problems, but will not replace SQL; people will use a combination of both based on their own needs. Seemed that everyone was looking forward to the afterparty and the drinks .

The second day started slowly, and we almost lost the first round of talks (after a long night at the Irish pub across the corner); it was a tough decision and I went to Benjamin Black talk where he presented some of the things he will be offering with his startup fastip. Even if he didn’t give much details this sounded like a very interesting product, that will offer ip flow based exports and analysis.

Next, I sow John Allspaw‘s talk “Go or No-Go”; very interesting, explaining how Etsy is doing new features deploys, and how they have these meetings. I really appreciated the fact that John has presented new things and didn’t just reuse his Velocity talk. I was really looking forward to Christopher Brown talk “Design for Scale – Patterns, Anti-Patterns, Successes and Failures” to see what challenges they had building the Opscode Platform. Unfortunately, he focused mostly on his EC2 experience and didn’t even touched chef or the opscode platform. Too bad, as there were many chef users in the room that would have liked that.

At this point the inevitable happened and I had to deal with an emergency and could not follow up much of Tom Daly talk about “Anycast Routing”. By the time I finished, the break was there and actually we had to leave to the airport to catch our planes back home. Too bad I could not see Joe Williams talk “Availability, the Cloud and Everything” but I will definitely check it out when the video will be posted online.

Jason assured me that all the talks were recorded, and will be posted with the slides online (probably in 2-3 weeks). I’m really looking forward to see some of the talks I could not attend because I was in the other room. This should be also very interesting for anyone that was not able to make it this year, but wants to see the talks and have an idea on what it was like to be there. Highly recommended.

Also, my colleges from the LISA’10 blogging team, have done some very interesting interviews with Anne Dickison about marketing LISA’10 (by Ben Cotton) and Alan Clegg about his DNSSEC tutorial (by Matt Simmons).

Everyone uses Google Analytics (GA), right? It’s a great product and even better it’s free. It is a win/win situation where there is no point in anyone running their own analytics unless they have something custom and specific that is not covered by GA.

Still as other google product its documentation is, let’s say not the best. A while ago I started working on a project and there was no web analytics in place. I asked them why is that? They said they have too big traffic to be accepted in GA. Hmm… I looked into it and I must admit I could not find much information that we were interested. Finally looked over the terms of services and there I found:
“2. FEES AND SERVICES . Subject to Section 15 herein, the Service is provided without charge to You for up to 5 million pageviews per month per account, and if You have an active Adwords campaign in good standing, the Service is provided without charge to You without a pageview limitation.”

So there is a limit, a tiny one I would say of 5mil pageviews per month (per account, not even per site). Our site was making about 45Mil pageviews at that time. Per day! So what if we wanted to use GA? We searched everywhere but could not find any commercial offering of GA or any other information. We asked the @googleanalytics on twitter but we were completely ignored.

What to do? Well, we just gave it a try and added the site and started tracking it in GA as any other site. Surprisingly, it worked just fine for a few months. Yesterday though, we received an email from the google analytics team (or should I say GA “robot”?) telling us that they have detected we have a high traffic, much higher than the allowed limit of 5mil pageviews per month, and from now on we are no longer going to have live reports but only daily updated reports. This is a limitation we can live with, but it would have been great if they would have given us some option to pay for some extra services. My client would have been happy to pay in the first place, but I assume this is something google doesn’t care at all and they just want to offer it as a free services. There is a great opportunity for such a product that could handle high traffic analytics and can do real-time and other goodies; we would be definitely interested. In the meantime if you have a site that makes more than 5mil pageviews per month (not so uncommon) you can definitely use GA; in the worst case they will restrict your updates to keep up with your traffic. For our site we tracked 1,608,074,379 Pageviews last month in GA  and it works just fine.

Seems like lately I’ve moved everyone I could onto google apps; friends, clients, or even strangers I could easily convince them on how great it is to not worry about your email server and put this into the hands of someone like google; and all this for free. Then why did it take so long for me to move? Well, email is very important to my business and this is why a long time ago (too many years to remember) I’ve made the decision to serve it on my own dedicated server, instead of a cheap vps. This was the main reason I rented the server in a good hosting facility (started with ThePlanet and then moved to SoftLayer about 3 years ago) and was happy to pay for it to know that I have a reliable service and my email will be reliable also, and be sure that if I get an email from a client or some nagios alert that something is not working I will be getting it immediately as expected. I’ve been a big fan of imap and used that all the time so I can check in the emails from different locations and have a central place where the files are and can be easily backed up. As any sysadmin I ended up with a big .procmailrc file with many rules, where some of them are most certainly no longer needed (projects completed, etc.) and with a huge Maildir, as I like to save anything that might be useful in the future. Don’t get me wrong I hit delete probably 80% of the time, but over time this grew to something like 1.2G quite easy. I’m sure many people have much bigger mailboxes than this, but anyway…

With google apps being the best solution for any new startup or basically anyone that is ‘small’ enough to not need anything special (you loose some control of course when you put this on someone else’s infrastructure), I wanted to move my email there for sometime. But it seemed too complicated to move it easily, without downtime and keeping all the existing emails in the same way how I like to see them in my Thunderbird, and my procmail rules still working; I don’t like the labels stuff, and still live in a folder world, and even if occasionally I would go to the web interface, I prefer to use the imap client with my Thunderbird. Naively I tried to move the emails by creating the new account in thunderbird and dragging and dropping the emails to the new location. This works just fine for a small inbox (like my wife’s that had about 500 emails) but for mine it was always timing out and needed to be restarted; this was impossible to be done like that as you always had to see where it stopped and with many emails/folders after 2-3 tries I realized this was not the way to go . Time to get more serious about this…

I sow the light when I found the script called imapsync. This is a script that basically does exactly what I needed and resume if it had any issue and it was obviously more reliable that any thunderbird copy. I installed this locally on the server, and it is basically just a perl script that syncronizes two imap accounts (not google apps specific). Once you download and uncompress it you will probably need some perl modules based on your system setup. In my case this was just Mail::IMAPClient: (you can see this by running perl -c imapsync):
perl -c imapsync
and I installed it from the debian packages with:
aptitude install libmail-imapclient-perl
after this imapsync was happy and I could install it with:
make install
(your setup might be different and you might need other modules like: Digest::MD5, Term::ReadKey, IO:Socket:SSL, Date::Manip, etc. see INSTALL for full details; you can install them using cpan or your system packages whatever you prefer).

Now that I had imapsync installed I was ready to play with this and test it out. I had to tune the command a little but in the end it looked like this:
imapsync --syncinternaldates --host1 <local_server> --port1 993 --ssl1 --user1 <user> --password1 <pass> --host2 imap.gmail.com --port2 993 --ssl2 --user2 <user@domain.com> --password2 <pass> --useheader 'Message-Id' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1
and a similar one for the sent folder (that needs a special rewrite rule for the different folder name):
imapsync --syncinternaldates --host1 <local_server> --port1 993 --ssl1 --user1 <user> --password1 <pass> --host2 imap.gmail.com --port2 993 --ssl2 --user2 <user@domain.com> --password2 <pass> --prefix2 '[Gmail]/' --folder 'INBOX.Sent' --regextrans2 's/Sent/Sent Mail/' --skipsize --noauthmd5 --reconnectretry1 1 --reconnectretry2 1

Depending on your mailbox size this will take a while… For me, the initial sync took about 10 hours (about 60,000 emails ~ 1.2GB):
Time : 37329 sec
Messages transferred : 60893
Messages skipped : 142
Messages deleted on host1: 0
Messages deleted on host2: 0
Total bytes transferred : 985592338
Total bytes skipped : 876862
Total bytes error : 0
Average bandwith rate : 25.8 Ko/s

After this I happily changed my MX record from a single mail.ducea.com line to something that looks more redundant like:

@                       MX      10 ASPMX.L.GOOGLE.COM.
@                       MX      20 ALT1.ASPMX.L.GOOGLE.COM.
@                       MX      20 ALT2.ASPMX.L.GOOGLE.COM.
@                       MX      30 ASPMX2.GOOGLEMAIL.COM.
@                       MX      30 ASPMX3.GOOGLEMAIL.COM.
@                       MX      30 ASPMX4.GOOGLEMAIL.COM.
@                       MX      30 ASPMX5.GOOGLEMAIL.COM.

and everything is looking good.

Still, I want to be able to use backups to save my emails somewhere else (just in case) as I was doing until now with a simple rsync/tar/gz script. For this, I’ve implemented a solution based also on imapsync that copies back the emails in the reverse direction now from google apps to my own server where I can then backup them up as I want. I hope that this post will be useful to other people that have similar concerns as I had while moving their email to google apps. There is really no reason at this time to run your own email server, deal with spams, etc. when there is a great free solution like google apps.

Matt’s full announcement on the USENIX blog: Introducing the 2010 LISA Blogging Team

If you don’t know what a barcamp is, this is a an “international network of user generated conferences — open, participatory workshop-events, whose content is provided by participants. The day consists of sessions proposed by attendees and the schedule is created on site the morning of the event. BarCamp is an ad-hoc gathering born from the desire for people to share and learn from each other in an open environment. It is an intense event with discussions, demos and interaction from participants.”

This was my first barcamp, or unconference type of event I’ve attended and I must say it has been by far the most interesting conference I’ve ever participated. So much better, engaging and with great conversation than a the usual conference where someone on the stage presents his slides. I’ve learned many things and this post is to outline my takeaways from this event. Here are just the most important ones:

Takeaway 1: meeting with other freelances.
Designers, photographers, writers, programers, or lawyers, all were freelancers. Even if working in different fields and with different experiences it was great to see that others have the same problems, and experience the same joy to work as freelancers. It is great to meet other people that are the same as yourself, and have the same values.

Takeaway 2: working from home… or not…
During the session about how people are working from home and maintain their sanity, I realized that for each person that works from home the situation is different. Some like it to work from home, some tried various solutions like working from a caffe shop, a library, a store, or even while driving in a train. My takeaway is to look for what works best for each individual and the particular type of work.

Takeaway 3: coworking
I’ve heard before about coworking and thought it is a very interesting concept and wanted to give it a try. Still from what I knew before I thought it was just a place to work, where you pay a free and have your desk, power and internet connection to do your thing. Now I realized that this is much more than that, and normally in such places the most important thing is the community of people that work there, that help and support each other. I would love to find a close by coworking place in the south bay area, but unfortunately as nextspace, most of them are in the city to far for a daily commute.

Takeaway 4: volunteer work; helping the community.
I was very proud to see that many freelances are involved in volunteer work and community projects. Even if we are so busy with projects there is always time for such great actions where everyone can contribute with their experience and expertise for their community. I know for sure that I will dedicate more time for this in the future.

Takeaway 5: networking, connections, promote your services.
You need to have an elevator pitch for your yourself, being able to explain who you are, what you do and why you are good the job in less that 30 sec. A freelancer needs to always market his services and not stop doing this even when he has a full plate of work; workflow can change in the future, and it will be much harder then to do this when pressured by the need to find new works quickly. I must admit that I’ve done this myself, being much to busy with regular work to have the time to market myself, and even dropped some potential clients because I didn’t had time. I will definitely have to change this approach.

Takeaway 6: productivity technique – pomodoro
Chris Burbridge mentioned about this great method he has been using to increase his productivity called pomodoro. It is a time boxing technique where you focus on only one task for 25 mins and then take a 5min brake; this is called a pomodoro after the name in italian of a kitchen timer in the form of a tomato. After 4 pomodoro’s you take a longer brake. For more details check out the online docs at http://www.pomodorotechnique.com/ It makes sense to me, and I will try it out to see if it’s working for my workflow that is rather heretic with many interruptions as you would expect for a busy sysadmin

In conclusion this was an awesome event and I will definitely try to attend the next one in Santa Cruz – October, 2nd 2010.