One of the parts of managing linux instances is understanding the state of the machine so it can be troubleshooted. I really needed to output logs from my machine, so I set out to learn a very well known tool for log summarization called logwatch.

There are three parts to any logwatch ‘service’. I use this term in quotes because I haven’t defined it yet, but also because it is different than the unix concept of a service. Generally it encompasses the type of logs you wish to summarize.

  1. A logfile configuration(located in <logwatch_root_dir>/logfiles/mylog.conf)
  2. A service configuration(located in <logwatch_root_dir>/services/mylog.conf)
  3. A filter script(located in <logwatch_root_dir>/scripts/services/mylog)

This three pieces together are what forms a service. What really helped me understand was to go through an example of one of these logs. Let’s take for example, the http-error service.

Before we continue, a note about <logwatch_root_dir>. Logwatch internally looks for these configuration files based on several dirs. For Ubuntu, the lookup is /usr/share/logwatch/default.conf > /usr/share/logwatch/dist.conf > /etc/logwatch/. The idea being that each successive directory overrides parameters from the previous location. This is covered really well here.

Logfile Configuration
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# /usr/share/logwatch/default.conf/logfiles/http-error.conf
########################################################
#   Define log file group for httpd
########################################################

# What actual file?  Defaults to LogPath if not absolute path....
LogFile = httpd/*error_log
LogFile = apache/*error.log.1

[ ... truncated ]

# If the archives are searched, here is one or more line
# (optionally containing wildcards) that tell where they are...
#If you use a "-" in naming add that as well -mgt
Archive = archiv/httpd/*error_log.*
Archive = httpd/*error_log.*
Archive = apache/*error.log.*.gz

[ ... truncated ]

# Expand the repeats (actually just removes them now)
*ExpandRepeats

# Keep only the lines in the proper date range...
*ApplyhttpDate

Line 7-8 are basically file filters on which files from the log root will logwatch feed into your service. This is a pretty great idea, because you could potentially generate a custom log based on many different kinds of logs. For example, your custom log can incorporate the number of http access errors that were encountered by your server in a given time period. If absolute paths are not given, paths are relative to the default log root, /var/log/

Lines 14-15 show that you can also search archive files for the same log information.

Line 21 seems to be some left over unused code, but was meant to expand the logs when standard syslog files have the message “Last Message Repeated n Times”. As the comment indicates, it just removes repeats.

Line 24 is interesting. The * indicates to the logwatch perl script to apply a filter function to all lines of this file. At <logwatch_root_dir>/scripts/shared/applyhttpdate, we can see that this filters the dates in the logs, assuming a certain header format for the lines in the file. Logwatch provides a couple of standard filters with intuitive names like onlycontains, remove, etc.

Service Configuration

So now we know how logwatch finds the logs that might be of interest to us. What does it do with these files? For that, we have to look at the service configuration file:

1
2
3
4
5
6
7

# /usr/share/logwatch/default.conf/services/http-error.conf
Title = http errors

# Which logfile group...
LogFile = http-error

Detail = High

The directive on Line 2 is straightforward - what should this log be named? When the log output is generated, this is what goes in the headers.

Line 5, confusingly, tells logwatch which logfile “group” it’s interested in. This is simply the logfile configuration we looked at earlier, minus the .conf extension. However, just as the logfile configuration can filter logs with different extensions and names, the service configuration can incorporate multiple logfile groups.

Service Configuration

Finally, logwatch runs the output of all of the logs gathered by the configurations through a script with the same name as the service configuration, but in <logwatch_root_dir>/scripts/services/<servicename>. Most bundled scripts are perl scripts, but the great thing is that you can pretty much use any scripting language.

I won’t actually go through /usr/share/logwatch/scripts/services/http-error here, one because it’s pretty long, and two, I don’t understand perl and can’t explain it very well : ) However, the gist of it is that it takes all the output of all the logs and summarizes them, outputting the result in stdout.

My custom log watch doesn’t actually watch any logs, but I still need to write these three files. This was my final setup.

Logfile conf
1
2
3
4

# /etc/logwatch/conf/logfiles/customlogger.conf
# This is actually a hack - I ask for log files, but I never actually use them.
LogFile = *.log
*ExpandRepeats
Service conf
1
2
3

# /etc/logwatch/conf/services/customlogger.conf
Title = customlogger
LogFile = customlogger
Script
1
2
3
4
5
6

# /etc/logwatch/scripts/services/customlogger
#!/usr/bin/env bash

# I just need to know what the memory usage is like at this point in time.
top -o %MEM -n 1 -b | head -n 20
free -m

To test, I just ran:

$ logwatch --service customlogger

 --------------------- customlogger Begin ------------------------

 top - 21:36:06 up  4:16,  1 user,  load average: 0.12, 0.05, 0.01
 Tasks: 144 total,   1 running, 143 sleeping,   0 stopped,   0 zombie
 %Cpu(s):  2.4 us,  0.6 sy,  0.0 ni, 96.8 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
 KiB Mem :   497664 total,    43716 free,   194456 used,   259492 buff/cache
 KiB Swap:        0 total,        0 free,        0 used.   280112 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 13373 root      20   0  202988  44448  12796 S  0.0  8.9   0:00.76 uwsgi
 11868 root      20   0  762376  41940   8192 S  0.0  8.4   1:04.90 dockerd
 13389 root      20   0  202988  35224   3568 S  0.0  7.1   0:00.00 uwsgi
 13390 root      20   0  202988  35220   3564 S  0.0  7.1   0:00.00 uwsgi
 13305 root      20   0   47780  15720   6432 S  0.0  3.2   0:02.16 supervisord
  9024 root      20   0  290696  13876   3268 S  0.0  2.8   0:02.55 fail2ban-se+
 31127 ubuntu    20   0   36412  11060   4316 S  0.0  2.2   0:00.07 logwatch
 11872 root      20   0  229844   9460   2536 S  0.0  1.9   0:00.28 containerd
  1162 root      20   0  266524   7372    488 S  0.0  1.5   0:00.02 snapd
 27837 root      20   0  101808   6844   5868 S  0.0  1.4   0:00.00 sshd
   417 root      20   0   62048   6356   3732 S  0.0  1.3   0:01.52 systemd-jou+
     1 root      20   0   55208   5796   3936 S  0.0  1.2   0:04.99 systemd
 13430 root      20   0   67824   5636   4896 S  0.0  1.1   0:00.02 sshd
               total        used        free      shared  buff/cache   available
 Mem:            486         190          42           7         253         273
 Swap:             0           0           0

 ---------------------- customlogger End -------------------------


 ###################### Logwatch End #########################

So if you need this to run every other hour, the last thing to do is to set up a cron job to do it. Pretty nifty, I think.

0 */2 * * * /usr/sbin/logwatch --service customlogger --output mail --mailto <your-email>