Loading ...
Rotating Linux Log Files
Logs… Any Linux system will generate many log files by default, containing various information about the operation of the system (normal actions, debugging information, security/authorization messages, web/email events, etc). If no rotation would occur on the various log files, then they will just grow bigger and bigger, filling up the space (on high traffic sites) but most importantly making it very difficult to find any information that we might be looking for in those log files. Fortunately this is handled in most Linux distributions by default and we don’t have anything special to do to set it up… if will function out of the box, rotating the log files by default.
This little how-to will show you how the default log rotation works, based on syslog and logrotate. There are other ways to achieve this, like using syslog-ng instead of syslog that I will cover in a future article.
Understanding how the default setup works, will help you have an idea of what will happen and what logs will be rotated, when will this occur, and how long will they be kept. Also it will show you the places where you can make changes in case you need to do that (if you want to save some log for a longer time, or if you want to rotate it differently from the default). The examples I will present are taken from a Debian system, so if you are running a different system, they might differ a little, but not drastically (like you might have the cron setup to run at a different time, or keep a different number of log iterations by default).
I have split this article in two parts based on the way how the log rotation is handled:
- system log files: most of the system log files are rotated by syslog itself and not using logrotate. You will see here what are these log files, and how they are rotated.
- application log files: logrotate is the default choice to rotate all the other log files. It can rotate the logs based on various parameters: daily, weekly, monthly, based on the size of the log, it can compress the logs to save space, etc.
Go to:
Rotating Linux Log Files - Part 1: syslog
Rotating Linux Log Files - Part 2: logrotate
Tags: Linux, logrotate, log_rotation, syslog
Share This
7th June 2006, 22:06
I’m glad that he mentioned that this was based on a Debian systems, since the RedHat & SuSE systems that I work with do not have the syslog “Feature” that is described (must be part of syslogNG which I’m not familiar with). Both of these major commercial distributions manage the syslog files via logrotate.
Actually I’m just starting to work with SuSE Enterprise, and I’m noticing the differences in the way RedHat & SuSE implement scheduling Hourly, Daily, Weekly, & Monthly rotations.
A feature of of LogRotate 3.7.x that I found, (and was looking for) is the ‘dateext’ which appends the rotation date to the end instead of a sequential number. The problem I have with the sequential number, is that logrotate moves the older log’s up the sequence, which causes those file to be backed-up multiple times with differential backups.
One item that you didn’t cover, is how logrotate knows when it is the proper time to rotate a daily, weekly, or monthly file. Part of that is controlled by the status file (/var/lib/logrotate.status on most Linux system) which tracks when a particular file was last rotated. The other part is that Monthly rotations are done with the first run of logrotate that month, and Weekly’s are run if the last run was later in the previous week, or the last rotation was over 7 days prior (Sunday=0 Saturday=6). So if you are running logrotate daily, then it works itself out to the first day of the month, and Sunday’s for the weeklies.
7th June 2006, 23:05
Chris,
Thank you very much for your comment. I too find that the dateext option quite cool, especially for rotating apache log files daily (at least here I have needed it). If you will save the logs for an extended period of time then this is quite a great option to use.
The way RedHat based systems (RHEL, Fedora, Centos, etc) handle the rotation of ’system log files’ is a little different than in Debian based systems: this is handled by logrotate itself, and actually I will post an update to the original post, to not create confusions about this. I don’t have a Suse system available to look at it but I assume that it will follow the RedHat path, but please confirm that if you can. Thanks.
8th June 2006, 14:07
Marius
This is a great primer, wish I would have found it a year back when I was trying to learn this stuff, the docs confused me (Google for examples came to the rescue then) but I needed to learn since my hardrive (and /var in particular) was smaller thna I wanted. Now I’m using newsyslog, sort of an enhanced version of syslog; same basic principals apply, but is gaining popularity among the *BSDs (but can be compiled/installed for Linux easily too, prob in debs). I write HOWTOs on my site as well, but most aren’t this detailed, as this could serve the community better than the project’s current doc.
fak3r
8th June 2006, 15:02
fak3r ,
Thank you very much for your nice words. It really helps to know that other peoples find useful what I am writing. This is really the greatest reward I can expect when doing this…
Regarding your comment about syslog… there is an option in syslog that could have been useful to you:
olddir “directory”
= old logs are moved into “directory” for rotation
But this has an important limitation: olddir must be on the same physical drive as the log file being rotated. Now depending from your case this could have been useful (if the partition was the problem and you had enough space on another partition), or not (if you needed the logs saved on another drive - still you could have solved this by placing the logs on that drive in the first place, and solve it like that). Anyway if you are happy with the solution you found, then this is all that matters. This is the beauty of the Linux world that you can find so many different solutions to achieve the same purpose.
Cheers
- Marius -
29th June 2008, 08:54
[...] Further information about logrotate:- http://linuxcommand.org/man_pages/logrotate8.html http://www.ducea.com/2006/06/06/rotating-linux-log-files/ [...]