named-report.pl - summarize ISC BIND 9 log messages

Copyright (C) 2002,2003 John Kristoff

Copying and distribution of this file, with or without modification,
are permitted in any medium without royalty provided the copyright
notice and this notice are preserved.

Obtaining the Latest Release

  For the latest version of this software visit the following page:

    John Kristoff's Home Page, Northwestern University
    <http://aharp.ittns.northwestern.edu/software/>

Getting Started

  This program is a rudimentary ISC BIND 9 named log summarizing
  tool.  This program expects a BIND 9 named log file as a command
  line argument or as standard input (STDIN) to the program.

  Example:

    ./named-report.pl /path/to/named/logfile

     or

    ./named-report.pl < /path/to/named/logfile

    or

    STDOUT | ./named-report.pl

  In the last example, another process would generate the proper named
  log output, which would in turn be used as input through the '|' pipe
  character to this program.  A more advanced technique in using this
  program would be to email a daily summary to an administrator. The
  following could be run (on a Solaris 8 system, adjust as necessary):

    grep -h "^`TZ=EST29EDT date +%b' '%d`" /path/to/named/logfiles* | \ 
    /path/to/named-report.pl | mailx -s "### named log report ###" \
    user@example.com

  The previous command does the following:

    1. Sets the time zone (TZ) environment variable to the previous day.
       In this example, we are assuming a US Eastern time zone.  This is
       done by adding 24 (hours), plus the 5 (hours) offset for the EST
       from UTC.

    2. The "date +%b' '%d" portion of the command will return the three
       character month followed by the day, with a leading zero if it
       is necessary.  This date format follows the format found in the
       named log messages tested with this program.

    3. The previous two items constitute the regular expression that
       will be used by grep.  An anchor (^) precedes the expression to
       specify that a match will be found at the start of a line.

    4. The file pattern includes any named log file.  This is useful
       if there may be many revisions of named log files that contain
       log messages from the previous day.

    5.  The output of the grep search will be sent to named-report.pl
        via the pipe (|) character as STDIN to the program.   Note the
        use of the '-h' option to grep which prevents a filename from
        being prepended to each log message.  This is required else
        the reporting tool would not be able to properly parge each
        line.

    6.  The output from the report-named.pl script is then sent to the
        mailx program via the pipe (|) character as STDIN.

    7.  Not shown, but possibly configured on the system is a crontab
        entry for a user with access to the program and named log files.
        Cron could be used to run the named-report.pl program daily,
        pushing out a summary report of the previous day's activity to
        an administrator.  Note: for Solaris, it will be necessary to
        escape the percent (%) characters with a backslash (\) in the
        system crontab.

Requirements

  This program assumes a unix-based system.  It was tested on Solaris
  SunOS 8, but it should run on other platforms equally as well.  Version
  5.005_03 of the Perl programming language was used on the test platform
  and it is expected that similar or newer versions of Perl will also work.

  Log input is based on a 9.2.x version of an ISC BIND named log file.
  Similar or newer versions of BIND may also work.

Reporting Details

  This program analyzes named log messages on a line by line basis.
  Each line is parsed and categorized according to its log message
  detail.  At this time, the following log message types are known
  and summarized by this program:

    server status    - BIND and related low level daemon messages
    invalid commands - invalid or authorized rndc commands
    client queries   - client queries for resource records
    client denies    - client queries denied  
    server fetches   - name server initiated requests to other DNS servers
    zone xfers       - zones transferred to/from the server
    xfer failures    - zone transfers to this server that have failed
    xfer denies      - zone transfer requests denied
    zone status      - zone loads, fetches, refreshes, updates and so on
    lame servers     - lame delegations as seen by the server
    version requests - client requests and denies for BIND version info
    broken clients   - broken client queries for invalid class views
    quota limits     - recursive and TCP client quotas reaches
    format errors    - client format errors
    notify error     - receive notify authoritative errors
    insecure updates - insecure dynamic DNS updates

  For further details on how the log messages are summarized examine
  the program code.  Log messages not understood by this program are
  placed in an unknown category and can be displayed if DEBUG mode is
  enabled.  DEBUG mode may be useful to help build new parsing routines
  for unknown messages or to assist in the troubleshooting process by
  displaying unusual log messages in full.

Configuration

  This program should generally work without any customized options
  for a particular environment.  However, the output generated by the
  program can be somewhat controlled by a small number of variables
  that are defined within the beginning portion of the code.  The
  following options are available:

    DEBUG             Generates additional output to aid troubleshooting.
		      This option is particularly useful for displaying
		      log messages the program does not currently
		      understand.

		      Setting: [0|1]  (0 = disabled, 1 = enabled)
		      Default: disabled 

    PRIVACY           This option controls whether the last octet is
		      displayed in the client queries report.  This
		      option is useful to provide some basic level of
		      client privacy.  Knowing records a particular
		      client requests may reveal more than one wants to
		      know about particular host or user activity.

		      Setting: [0|1]  (0 = disabled, 1 = enabled)
		      Default: enabled

    RESOLVE           Controls whether the program attempts to display
		      the host names of IP addresses found in the log
		      messages and output in the reports.  Enabling this
		      option slows down the program considerably, because
		      execution time then depends on the program being
		      able to do DNS queries.  In some cases, IP addresses
		      may not have associated names, which will cause a
		      considerable delay in program execution time since
		      a timeout must occur before the program can continue.
		      The program will output '[lookup failed]' in place of
		      a host name when an IP address cannot be resolved.

		      Setting: [0|1]  (0 = disabled, 1 = enabled)
		      Default: enabled

     TOPx             Determines how many rows of summary to include in
                      the reports for each category of log message.

                      Setting: [5 to 25 recommended]
                      Default: 10

References

  This program was designed to provide a 'quick-n-dirty' summary view
  of log messages a particular DNS server was generating.  Other log
  reporting tools are available and may be more appropriate for more
  demanding requirements.  The following references may be useful for
  exploring the use of other tools and techniques:

    Lire - open source reporting and analysis software
    <http://www.logreport.org>
    GNU GPL software that manages many types of system log file formats
    including DNS, HTTP, FTP, firewalls, email and more.  A very complete
    system written mostly with the Perl programming language and using
    XML formats.

    Log Analysis Resources page - long list of tools and info 
    <http://www.loganalysis.org>
    A web page maintained by Tina Bird and Marcus Ranum that contains
    a large collection of links to log management information and tools.

    DNS Tools listing page - various DNS tools reference page
    <http://www.dns.net/dnsrd/tools.html>
    Many links and information for DNS tools, related DNS management,
    troubleshooting and analysis software.  Somewhat dated.
 
Caveats

  This program may not run properly on your systems.  While it is not
  expected to cause harm in its usage, correct operation and report
  output may not be possible depending on system configuration.  It
  has been verified by one other party in limited testing to perform as
  expected, but mileage may vary. It is recommended that precautions
  be taken, such as running this script first on a test system with a
  copy of typical named log data for the environment.

Frequently Asked Questions

  Q: When I run this tool, I only have totals for 'other (unknown)
     log messages'.  Why am I not getting reports for known categories?

  A: You may be performing a 'grep <files>' using a wildcard spec for
     the files to process, then piping the output to the program.  By
     default grep will prepend each matched line with the filename.
     The program does not expect the filename and will be unable to
     correctly parse the log messages.  Use the '-h' option with grep
     disable to the prepending of filenames the matched lines.

  Q: What are broken client queries?

  A: These are often Microsoft Windows clients specifying an invalid
     class view in queries.  In these cases, I do not currently know
     what specific client configuration causes these messages and I
     have had trouble finding out further details.  In these cases a
     standard answer in other forums has been "the client is broken,
     upgrade/fix the client".  If anyone has any additional details,
     please let me know.

  Q: When I run the tool, I get an error message like "Command not
     found".

  A: The Perl script assumes that you have Perl installed in the
     /usr/bin directory.  First ensure you have Perl installed and
     that it is in the excepted directory.  You can edit the first
     line of the code to change the path to the perl binary if
     necessary.

Acknowledgments

  O'Reilly -  for the books
  Google - for the answers to my search questions
  Rob Thomas - for initial testing and a link on his DNS page
  #perl on freenode IRC - for answers to my questions not found elsewhere
  loganalsysis.org - for the link to this tool and all the references
  Richard Gadsden - for patch suggestions
  Kenneth Porter - for feedback, sample logs and patch suggestions
  Brian Tower - for feedback and testing

Contact Information

  John Kristoff <jtk@northwestern.edu>
  Northwestern University
  2001 Sheridan Road 
  Evanston, IL USA

  Personal and work webpage: <http://aharp.ittns.northwestern.edu>

Revision History

  2002-12-05,jtk - Initial program release
  2003-01-21,jtk - Documentation updates for v1.1 and minor edits
  2003-04-01,jtk - Documentation updates for v1.2 and minor edits
  2003-07-22,jtk - Documentation updates for v1.3 and minor edits
  2003-12-01,jtk - Contact, Acknowledgement and FAQ updates
