Loading ...
Rotate Apache logs using Awstats
Whatever web log analyzer you would use, at one time or the other you will end up with one problem: how to deal with weblogs rotation, to not have a gap in your statistics? This has various solutions starting with running the weblog analyzer at the prerotate step of logrotate, or maybe using some manual script.
This post will show how you can rotate the apache logs using awstats right after it has processed the logs. This can be beneficial for situations where you have quite big logs and using this method will keep them small all the time, and also where restarting apache just for logrotating is not such a good idea. Obviously for this to make sense, you need to be already using awstats for your log processing 
.
The awstats config option PurgeLogFile will purge the log file, after analyzing it. The default is 0 and this means no purge is done, and will assume some external tool is used for log rotation (like logrotate, or even apache internal mechanisms like rotatelogs). If this is set to 1, awstats will purge of the log file everytime it is run:
PurgeLogFile=1
By default the ‘archive’ file will be saved under: DirData/awstats_archive.configname.log , meaning this will become in time a huge file that needs to be also rotated, compressed, archived, etc. Another nice option of awstats is the possibility to rotate this file. ArchiveLogRecords can be used for this, and again this is also set by default on 0.
- for ex. if we would want this file rotated daily you would use: %YYYY%MM%DD
ArchiveLogRecords=%YYYY%MM%DD
- or monthly: %YYYY%MM
ArchiveLogRecords=%YYYY%MM
For example using:
PurgeLogFile=1
ArchiveLogRecords=%YYYY%MM
will create the following monthly archive files:
awstats_archive.configname.200804.log
awstats_archive.configname.200805.log
...
and from here you can compress them, archive them, or even pipe them to another analyzer software if you need that .
Depending on your needs, this might be a great (and very simple) solution to rotate your apache logs, right after processing and this without even restarting apache. For more details, check the awstats docs.
Tags: apache, awstats, logrotate
Share This
11th June 2008, 21:35
I have found that mod_log_sql is my answer to apache log manipulation. Basically it simply writes your apache logs to a MySQL database rather than a flat file.
Not only can you avoid gaps in your stats you can log to the same table from multiple web servers. This allows for sites that are run on multiple web servers to have aggregated stats or the ability to aggregate stats across all sites that you are responsible for (network-wide stats).
A simple perl script can create a temporary logfile from the database to run your stats program against and a tiny table can be used to track the time stamp of where you left off.
I have been doing this with great success for over 1000 sites that I manage.
17th June 2008, 14:21
@Jay: do you know any tools, that analyses such logs which was wrote to a mysql db??
18th June 2008, 15:26
@Erik: I don’t know of any tools that query the database directly, however the fields in the table are the same fields that you would see in the apache logs.
What I have been doing is running a perl script I wrote on a cron that does several things. First, it dumps the contents of the database into a temporary file that is in the apache log format. It then calls awstats which is set up to use that file as its log file. Lastly, it records in another table the timestamp of the last record it looked at. This way, it knows where to begin the log the next time around.