Linux Log Monitoring and Watching – How can you monitor your Linux log files without having to spend hours writing a script for it or searching the internet endlessly for a Linux log monitoring solution?
Linux log monitoring and/or log watching is a duty that is in a different dimension all by itself. It is unique. The reason I say this is because log files in Linux often vary in format which makes it difficult for many Linux users to accomplish the task of monitoring their various logs.
Linux logs often fall under two categories: custom log (i.e. application logs) or system related (syslogs i.e. /var/log/messages file). Because of this variety, it is extremely difficult to find one good monitor that can be set up to watch them all, since not all log files are formatted in the same way.
But what exactly do I mean by “format”?
By format, I’m referring solely to the format of the dates that’s usually in the beginning of each line in a Linux/UNIX log file.
For instance, a typical /var/log/messages system log on a Linux server, will look something like this:
Linux System Log-File:
[nagios.kedy0:501] tail -6 /var/log/messages
Sep 18 08:23:51 nagios snmpd: Received SNMP packet(s) from UDP: [10.10.*.*]:47725
Sep 18 08:23:54 nagios snmpd: Connection from UDP: [10.10.*.*]:47725
Sep 18 08:24:11 nagios sshd: Authorized to root, krb5 principal dadmin/root@NETNET
Sep 18 08:24:11 nagios sshd: permit_root_login: PERMIT_GSSAPI_ONLY method: gssapi-with-mic
Sep 18 08:24:11 nagios sshd: GSSAPIII authenticated jbowman login accepted
Sep 18 08:24:11 nagios sshd: Accepted gssapi-with-mic for root from 10.10.*.* port 5345 ssh2
Notice the first three columns of each line of log in the above Linux system log. They represent the date and time.
Another log file, usually a custom application log (not a system log like the one above), can look like this:
[nagios.kedy0:516] tail -3 /prod/app.log
2011/01/20 14:26:35 UTC [SMTPProper,48088609,69.*.*.*] Receiving message for delivery: firstname.lastname@example.org to=[‘email@example.com’]
2011/01/20 14:26:35 UTC [-] Attempting ‘attach’ (promo is False) delivery for 17777551333
2011/01/20 14:26:35 UTC [-] Starting factory
Explanation of both Log-Files:
Again, focus on the columns; in the case of the application log, it is the first 2 columns that matter.
In the first output of the /var/log/messages file, the fields of the date are separated by spaces and are a combination of words and numbers.
In the second output of a custom application log, the day, month and year are separated with forward slashes and they are all numerical. Notice the order they’re in. The year comes first, then the month, then the day.
Other files may have the order reversed; with the day coming first, followed by the month, then followed by the year. Other logs may have the fields separated by hyphens instead of slashes. The scenarios here are endless.
Do you now see how trying to monitor a Linux log, without the proper tool, can be a hellish task?