Log viewer setup

From Smith family
Jump to: navigation, search
Server setup
← Previous Next →
Samba Backup

I use AWStats to keep track of the usage of my web sites and mail server. Setting it up is quite simple

Installation

  • Install AWStats
root@desktop:~# apt-get install awstats libnet-ip-perl libgeo-ipfree-perl
  • Because I have the cgi-bin directory in a non-standard location, copy the awstats script to that location:
root@desktop:~# cp /usr/lib/cgi-bin/awstats.pl /var/www/cgi.site.domain.tld/
  • Create a new user for awstats, make it part of the www-data and adm groups. Give it permissions for the awstats data files.
root@desktop:~# useradd -g www-data -G adm awstats
root@desktop:~# chown -R awstats:www-data /var/lib/awstats
  • Make the cron jobs run as this user. Adjust /etc/cron.d/awstats to change the www-data user to awstats
MAILTO=root

*/10 * * * *  awstats [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh

# Generate static reports:
10 03 * * * awstats [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh

AWStats configuration

SiteDomain

AWStats doesn't like not knowing a default domain, even though the domain is specified in each site-specific configuration below.

Modify /etc/awstats/awstats.conf to include a default domain, such as the one that serves the AWStats pages.

SiteDomain="site.domain.tld"

Websites

  • Each site you want to generate a log view for needs its own configuration. For the site site.domain.tld. copy the basic config file /etc/awstats/awstats.conf to /etc/awstats/awstats.site.domain.tld.conf. Make the following changes
LogFile="/var/log/apache2/site.domain.tld.access.log"

SiteDomain="site.domain.tld"
HostAliases="localhost 127.0.0.1 site.domain.tld"

LogFormat=1

LoadPlugin="tooltips"
LoadPlugin="geoipfree"
  • Create the statistics databases
root@desktop:~# /var/www/cgi.site.domain.tld/awstats.pl -config=site2.domain.tld -update
root@desktop:~# mkdir /var/cache/awstats/site2.domain.tld
root@desktop:~# chown awstats:www-data /var/cache/awstats/site2.domain.tld
Repeat for each site you have
  • The default cron job for AwStats, /etc/cron.d/awstats,works: it updates the stats every ten minutes.
MAILTO=root

*/10 * * * * www-data [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh

# Generate static reports:
10 03 * * * www-data [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh
Note that the cron jobs need to run as root, not www-data.

Mail servers

This is very similar to the setup for Web log statistics, but the config file is a bit different. This assumes you're using Postfix as the mail server.

  • Create the config file /etc/awstats/awstats.mail.conf with the following changes:
# MAIN SETUP SECTION (Required to make AWStats work)
LogFile="perl /usr/share/doc/awstats/examples/maillogconvert.pl standard < /var/log/mail.log |"
LogType=M 
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" 
SiteDomain="mail.domain.tld"
HostAliases="localhost 127.0.0.1 mail.domain.tld smtp.domain.tld"

LoadPlugin="tooltips"
LoadPlugin="geoipfree"

# OPTIONAL ACCURACY SETUP SECTION (Not required but increase AWStats features)
LevelForBrowsersDetection=0 
LevelForOSDetection=0 
LevelForRefererAnalyze=0 
LevelForRobotsDetection=0 
LevelForWormsDetection=0 
LevelForSearchEnginesDetection=0 
LevelForFileTypesDetection=0 

# OPTIONAL ACCURACY SETUP SECTION (Not required but increase AWStats features)
ShowMenu=1 
ShowSummary=HB 
ShowMonthStats=HB 
ShowDaysOfMonthStats=HB 
ShowDaysOfWeekStats=HB 
ShowHoursStats=HB 
ShowDomainsStats=0 
ShowHostsStats=HBL 
ShowAuthenticatedUsers=0 
ShowRobotsStats=0 
ShowEMailSenders=HBML 
ShowEMailReceivers=HBML 
ShowSessionsStats=0 
ShowPagesStats=0 
ShowFileTypesStats=0 
ShowFileSizesStats=0 
ShowBrowsersStats=0 
ShowOSStats=0 
ShowOriginStats=0 
ShowKeyphrasesStats=0 
ShowKeywordsStats=0 
ShowMiscStats=0 
ShowHTTPErrorsStats=0 
ShowSMTPErrorsStats=1

Remove default config

If the default awstats.conf remains in /etc/awstats, it will generate errors as SiteDomain isn't set. Therefore, rename it.

root@desktop:~# mv /etc/awstats/awstats.conf /etc/awstats/awstats.conf.original

Apache configuration

  • Allow whatever virtual host you're using to view the statistics. Update site.domain.tld to include these lines inside the VirtualHost block
Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
ScriptAlias /statistics/ /var/www/cgi-bin.domain.tld/
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

<Files "awstats.pl">
  AuthType Basic
  AuthName "AWStats"
  require valid-user
  AuthUserFile /etc/apache2/awstats.passwd
</Files>
  • The final stanza places the statistics behind a password (though only one password is needed to view the statistics of any of the sites). Add passwords for viewing the statistics with hte command
root@desktop:~# htpasswd -c /etc/apache2/awstats.passwd user_name
where user_name is the relevant user and then give the password.
  • Reload Apache config
root@desktop:~# systemctl reload apache2.service

View stats

You can now view the statistics by browsing to http://site.domain.tld/statistics/awstats.pl?config=site2.domain.tld, where site2.domain.tld is the name of one of the AWStats config files created earlier.

Cataloguing old stats and logs

If you want to keep historic stats from an old machine when moving to a new one, you should be able to move the files in DirData in the config files, typically /var/lib/awstats. However, that didn't work for me, so I recreated the stats from the old log files.

  • Disable automatic updates while you're doing this. Edit /etc/cron.d/awstats and comment out the two commands there.
  • Merge all the logs you want into one file:
root@desktop:~# perl /usr/share/awstats/tools/logresolvemerge.pl /var/log/apache2/site.access.log* > site.all.log
  • Mail logs need to be manually strung together in date order, converted, and bundled together.
root@desktop:~# rm mail.converted.log
root@desktop:~# for f in `ls -tr /var/log/mail.log*` ; do /usr/share/doc/awstats/examples/maillogconvert.pl standard < $f >> mail.converted.log; done
  • Process them (this will take a long time):
root@desktop:~#  /var/www/cgi-bin.scripts.domain.tld/awstats.pl -update -config=domain.tld -LogFile=site.all.log

See also