Log viewer setup

From Smith family
Server setup
← Previous Next →
Samba Backup

I use AWStats and Munin to keep track of the usage of my web sites and mail server. Setting them up is quite simple.

Installation

  • Install AWStats
root@desktop:~# apt-get install awstats libnet-ip-perl libgeo-ipfree-perl
  • Because I have the cgi-bin directory in a non-standard location, copy the awstats script to that location:
root@desktop:~# cp /usr/lib/cgi-bin/awstats.pl /var/www/cgi.site.domain.tld/
  • Create a new user for awstats, make it part of the www-data and adm groups. Give it permissions for the awstats data files.
root@desktop:~# useradd -g www-data -G adm awstats
root@desktop:~# chown -R awstats:www-data /var/lib/awstats
  • Make the cron jobs run as this user. Adjust /etc/cron.d/awstats to change the www-data user to awstats
MAILTO=root

*/10 * * * *  awstats [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh

# Generate static reports:
10 03 * * * awstats [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh
  • Install Munin and Munin-node
On each machine being monitored, install Munin node and sensors.
root@desktop:~# apt install munin-node munin-plugins-extra
On the central machine that does the reporting, install Munin (and probably munin-node as well).
root@desktop:~# apt install munin

AWStats configuration

SiteDomain

AWStats doesn't like not knowing a default domain, even though the domain is specified in each site-specific configuration below.

Modify /etc/awstats/awstats.conf to include a default domain, such as the one that serves the AWStats pages. 

SiteDomain="site.domain.tld"

Websites

  • Each site you want to generate a log view for needs its own configuration. For the site site.domain.tld. copy the basic config file /etc/awstats/awstats.conf to /etc/awstats/awstats.site.domain.tld.conf. Make the following changes
LogFile="/var/log/apache2/site.domain.tld.access.log"

SiteDomain="site.domain.tld"
HostAliases="localhost 127.0.0.1 site.domain.tld"

LogFormat=1

LoadPlugin="tooltips"
LoadPlugin="geoipfree"
  • Create the statistics databases
root@desktop:~# /var/www/cgi.site.domain.tld/awstats.pl -config=site2.domain.tld -update
root@desktop:~# mkdir /var/cache/awstats/site2.domain.tld
root@desktop:~# chown awstats:www-data /var/cache/awstats/site2.domain.tld
Repeat for each site you have
  • The default cron job for AwStats, /etc/cron.d/awstats,works: it updates the stats every ten minutes.
MAILTO=root

*/10 * * * * www-data [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh

# Generate static reports:
10 03 * * * www-data [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh
Note that the cron jobs need to run as root, not www-data.

Mail servers

This is very similar to the setup for Web log statistics, but the config file is a bit different. This assumes you're using Postfix as the mail server.

  • Create the config file /etc/awstats/awstats.mail.conf with the following changes:
# MAIN SETUP SECTION (Required to make AWStats work)
LogFile="perl /usr/share/doc/awstats/examples/maillogconvert.pl standard < /var/log/mail.log |"
LogType=M 
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" 
SiteDomain="mail.domain.tld"
HostAliases="localhost 127.0.0.1 mail.domain.tld smtp.domain.tld"

LoadPlugin="tooltips"
LoadPlugin="geoipfree"

# OPTIONAL ACCURACY SETUP SECTION (Not required but increase AWStats features)
LevelForBrowsersDetection=0 
LevelForOSDetection=0 
LevelForRefererAnalyze=0 
LevelForRobotsDetection=0 
LevelForWormsDetection=0 
LevelForSearchEnginesDetection=0 
LevelForFileTypesDetection=0 

# OPTIONAL ACCURACY SETUP SECTION (Not required but increase AWStats features)
ShowMenu=1 
ShowSummary=HB 
ShowMonthStats=HB 
ShowDaysOfMonthStats=HB 
ShowDaysOfWeekStats=HB 
ShowHoursStats=HB 
ShowDomainsStats=0 
ShowHostsStats=HBL 
ShowAuthenticatedUsers=0 
ShowRobotsStats=0 
ShowEMailSenders=HBML 
ShowEMailReceivers=HBML 
ShowSessionsStats=0 
ShowPagesStats=0 
ShowFileTypesStats=0 
ShowFileSizesStats=0 
ShowBrowsersStats=0 
ShowOSStats=0 
ShowOriginStats=0 
ShowKeyphrasesStats=0 
ShowKeywordsStats=0 
ShowMiscStats=0 
ShowHTTPErrorsStats=0 
ShowSMTPErrorsStats=1

Remove default config

If the default awstats.conf remains in /etc/awstats, it will generate errors as SiteDomain isn't set. Therefore, rename it. 

root@desktop:~# mv /etc/awstats/awstats.conf /etc/awstats/awstats.conf.original

Munin configuration

On each monitored machine, alter /etc/munin/munin-node.conf to include the addresses of the monitoring machine. Add these lines: 

allow ^192\.168\.1\.251$
allow ^1234:abcd:4321:dcba::2$

host 1234:abcd:4321:dcba::1
port 4949

On the machine does the reporting, modify /etc/munin/munin.conf to include sections for each monitored machine. 

[desktop.domain.tld]
   address [abcd:1234::1]
   use_node_name yes

[server.domain.tld]
   address [abcd:1234::2]
   use_node_name yes

Apache reporting for Munin

Ensure mod-status is enabled. 

root@server:~# a2enmod status

The default configuration file should be sufficient. 

<Location /server-status>
       SetHandler server-status
       Require local
</Location>

Create a stub virtual host file for localhost to show the server status. Create /etc/apache2/sites-available/localhost.conf containing: 

<VirtualHost *:80>
  ServerName localhost
</VirtualHost>

and enable it. 

root@server:~# a2ensite localhost
root@server:~# systemctl reload apache2.service

Symlink the Munin plugin for Apache on the node with the webserver. 

root@server:~# ln -s /usr/share/munin/plugins/apache_* /etc/munin/plugins/

Restart Munin. 

root@server:~# systemctl restart munin-node
root@server:~# systemctl restart munin

You should now have an "apache" section in the Munin monitoring page.

Apache configuration

AWStats

  • Allow whatever virtual host you're using to view the statistics. Update site.domain.tld to include these lines inside the VirtualHost block
Alias /awstatsclasses "/usr/share/awstats/lib/"
Alias /awstats-icon/ "/usr/share/awstats/icon/"
Alias /awstatscss "/usr/share/doc/awstats/examples/css"
ScriptAlias /statistics/ /var/www/cgi-bin.domain.tld/
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

<Files "awstats.pl">
  AuthType Basic
  AuthName "AWStats"
  require valid-user
  AuthUserFile /etc/apache2/awstats.passwd
</Files>
  • The final stanza places the statistics behind a password (though only one password is needed to view the statistics of any of the sites). Add passwords for viewing the statistics with the command
root@desktop:~# htpasswd -c /etc/apache2/awstats.passwd user_name
where user_name is the relevant user and then give the password.

Munin

I have the Munin reports available on a directory of a virtual host, rather than as a separate subdomain. That means I have to make some changes to the default web config. 

   # Rewrite rules to serve traffic from the root instead of /munin-cgi
   RewriteEngine On
   RewriteRule ^/munin/$ /munin-cgi/munin-cgi-html/index.html [PT]
   # Static files
   RewriteRule ^/munin/static/(.*) /etc/munin/static/$1          [L]
   # HTML
   RewriteRule ^/munin(/.*\.html)?$ /munin-cgi/munin-cgi-html/$1 [PT]
   # Images
   RewriteRule ^/munin/munin-cgi/munin-cgi-graph/(.*) /munin/$1
   RewriteCond %{REQUEST_URI} !^/munin/static
   RewriteRule ^/munin/(.*.png)$ /munin-cgi/munin-cgi-graph/$1 [L,PT]    

   Alias /munin /var/cache/munin/www
   ScriptAlias /munin-cgi/munin-cgi-graph /usr/lib/munin/cgi/munin-cgi-graph
   ScriptAlias /munin-cgi/munin-cgi-html /usr/lib/munin/cgi/munin-cgi-html

   <Directory /etc/munin/static>
       # Require all granted
     AuthType Basic
       AuthName "AWStats"
       require valid-user
       AuthUserFile /etc/apache2/awstats.passwd
   </Directory>

   <Directory /usr/lib/munin/cgi>
       # Require all granted
     AuthType Basic
       AuthName "AWStats"
       require valid-user
       AuthUserFile /etc/apache2/awstats.passwd
       <IfModule mod_fcgid.c>
           SetHandler fcgid-script
       </IfModule>
       <IfModule !mod_fcgid.c>
           SetHandler cgi-script
       </IfModule>
   </Directory>

As with AWStats, the final couple of stanzas put the Munin reports behind a password.

After making any changes

  • Reload Apache config
root@desktop:~# systemctl reload apache2.service

View stats

You can now view the statistics by browsing to http://site.domain.tld/statistics/awstats.pl?config=site2.domain.tld, where site2.domain.tld is the name of one of the AWStats config files created earlier.

Cataloguing old stats and logs

If you want to keep historic stats from an old machine when moving to a new one, you should be able to move the files in DirData in the config files, typically /var/lib/awstats. However, that didn't work for me, so I recreated the stats from the old log files.

  • Disable automatic updates while you're doing this. Edit /etc/cron.d/awstats and comment out the two commands there.
  • Merge all the logs you want into one file:
root@desktop:~# perl /usr/share/awstats/tools/logresolvemerge.pl /var/log/apache2/site.access.log* > site.all.log
  • Mail logs need to be manually strung together in date order, converted, and bundled together.
root@desktop:~# rm mail.converted.log
root@desktop:~# for f in `ls -tr /var/log/mail.log*` ; do /usr/share/doc/awstats/examples/maillogconvert.pl standard < $f >> mail.converted.log; done
  • Process them (this will take a long time):
root@desktop:~#  /var/www/cgi-bin.scripts.domain.tld/awstats.pl -update -config=domain.tld -LogFile=site.all.log

See also

Retrieved from "https://www.njae.me.uk/mediawiki/index.php?title=Log_viewer_setup&oldid=3053"