AWstats with Vlogger Rotation

Vlogger is a logging tool that you can pipe your Apache logs to. The logs will be broken down by virtual hosts and days. To do this, you need to put just one CustomLog directive into your global Apache configuration. I had two downsides to using Vlogger. One was that I could not use logrotate to trigger AWstats. The other is that old log files are not compressed.

Vlogger by default creates log files with names like these:

06202008-access.log
06192008-access.log
06182008-access.log

AWstats is flexible enough to allow for this naming convention (MM/DD/YYYY). In your site’s configuration file for AWstats you’ll need to set the LogFile value as such:

LogFile=”/var/log/apache2/your.domain.com/%MM-24%DD-24%YYYY-24-access.log”

This is simply the month minus 24 hours, the day minus 24 hours and the year minus 24 hours. Now that we have AWstats set to work with the log files generated by Vlogger we need to create a shell script that can manage the task for us via cron. Here is one I created that seems to be working quite nicely:

#!/bin/sh

logdir=/var/log/apache2
yesterdaysdate=`/bin/date -d "1 day ago" +%m%d%Y`
lastweeksdate=`/bin/date -d "7 days ago" +%m%d%Y`

cd ${logdir}

debug_out()
{
  if [ "$1" = '-debug' ] || [ "$1" = '-d' ];
  then
    echo "$2"
  fi
}

for directory in *
do
  # does the site (ServerName) directory exist and is there a matching awstats config file?
  awstatsconf="/etc/awstats/awstats.${directory}.conf"
  yesterdayslog="${logdir}/${directory}/${yesterdaysdate}-access.log"
  lastweekslog="${logdir}/${directory}/${lastweeksdate}-access.log"
  lastweekscompressedlog="${logdir}/${directory}/${lastweeksdate}-access.log.gz"
  if [ -d ${directory} ]
  then
    linebreak=0
    if [ -f ${awstatsconf} ]
    then
      # update statistics
      linebreak=1
      debug_message="executing:   /usr/lib/cgi-bin/awstats.pl -config=${directory} -update > /dev/null"
      debug_out "$1" "${debug_message}"
      /usr/lib/cgi-bin/awstats.pl -config=${directory} -update > /dev/null
    fi
    # is there a log for yesterday? If so then compress it.
    if [ -f ${yesterdayslog} ]
    then
      linebreak=1
      debug_message="executing:   gzip -q ${yesterdayslog}"
      debug_out "$1" "${debug_message}"
      gzip -q ${yesterdayslog}
    fi
    # is there a log for this day last week? If so remove it.
    if [ -f ${lastweekslog} ]
    then
      linebreak=1
      debug_message="executing:   rm -f ${lastweekslog}"
      debug_out "$1" "${debug_message}"
      rm -f ${lastweekslog}
    fi
    # is there a compressed log for this day last week? If so remove it.
    if [ -f ${lastweekscompressedlog} ]
    then
      linebreak=1
      debug_message="executing:   rm -f ${lastweekscompressedlog}"
      debug_out "$1" "${debug_message}"
      rm -f ${lastweekscompressedlog}
    fi
    if [ ${linebreak} = 1 ]
    then
      debug_out "$1" ""
    fi
  fi
done

exit 0

The above script can be called manually with -debug to see the steps being taken. The script will look for directories in Apache’s log directory and use their names when running AWstats. After AWstats has ran the file is then compressed. Files older than 7 days are removed but this can be adjusted as needed. Place this script in an area where it can be executed on your server such as /usr/local/sbin/awstats and make it executable (chmod 755). Then you’ll need a cron job to run this script daily like this:

0 4 * * * /usr/local/sbin/awstats &> /dev/null

And there you have it.

Feedback

  Comments: 4


  1. easy way is disable vlogger rotation
    -n param


  2. Thanks for the tip! I wasn’t trying to disable it though. I just wanted the logs compressed after having them parsed with AWStats.


  3. Hello Dave,
    Starting from your idea, I wrote a fully automated script creating a conf file for each of the vlogger directories and executing awstats for each site found.
    I dont use the vlogger -n feature.

    Here’s the code:
    #!/bin/sh
    #
    # this program will scan the logs created by vlogger,
    # create an awstat configuration file for each of them
    # and then execute an update of the awstats statistics
    #
    # it assumes that vlogger rotation is disabled (-n option)
    # a good way is to rotate the logs with logrotate and run
    # this program in the prerotate directive
    # this shold be done if you dont want to lose any record in the rotation
    #
    # you can run this program as often as you need as a cron job
    #
    # Apache log facility setting:
    # LogFormat “%v %h %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”” combined
    # CustomLog “| /usr/sbin/vlogger -n -s access.log -t access.log /var/log/apache2” combined
    #
    # This program is based on the idea from dave
    # http://davelozier.com/2008/06/20/awstats-vlogger/

    logdir=/var/log/apache2
    confdir=/etc/awstats
    datadir=/var/lib/awstats
    cgidir=/cgi-bin
    logfile=access.log
    awCommonSetupFile=/etc/awstats/awstats.conf.local
    apacheUser=www-data
    apacheGroup=www-data

    # check if we are the only running instance of the program
    LOCK=”/tmp/${0##*/}.lock”
    [ -e “${LOCK}” ] && exit 0
    touch “${LOCK}”

    # if a log directory exists in /var/log/apache and this site has no awstat conf file
    # create it awstat.sitename.conf
    cd $logdir
    for directory in *
    do
    # does the site (ServerName) directory exist and is there a matching awstats config file?
    awstatsconf=”$confdir/awstats.$directory.conf”
    if [ -d ${directory} ]
    then
    if [ ! -f ${awstatsconf} ]
    then
    # create configuration file
    printf “Include “%s”nLogFile=%snSiteDomain=%snDirData=%snDirCgi=%sn”
    $awCommonSetupFile
    “$logdir/$directory/$logfile”
    $directory
    $datadir/$directory
    $cgidir/$directory > $awstatsconf
    #create logging directory
    mkdir $datadir/$directory
    fi
    fi
    done

    # execute all updates found in $confdir
    # this allow to execute manually added stats configuration files for sites not logged in the $logdir directory (smtp, ftp for instance)
    cd $confdir
    for conffile in awstats.*.conf
    do
    # extract sitename from $conffile
    sitename=`echo $conffile | awk ‘BEGIN {FS=”.”} {i=2; while (i < NF) {if (i<(NF-1)) printf(“%s.”,$i); else printf(“%s”,$i); i++; }}’`
    # update stats
    /usr/lib/cgi-bin/awstats.pl -config=$sitename -update
    done

    #change ownership of stats data in order to allow the direct update from web interface (parameter AllowToUpdateStatsFromBrowser)
    chown -R $apacheUser.$apacheGroup $datadir

    #remove lockfile
    rm “${LOCK}”

    exit 0


  4. Hey Guy! Thank you for this script.

    I’ve got some work to do this coming weekend with which this will make much easier.

    Cheers 🙂