Monitoring log files

Question

We have around 20 log files of varying formats, generated by several different applications. We would like to receive notifications (preferably text messages, but email can be made to work in lieu of that) should certain conditions be met in the contents of those files.

For example, for one log file we might like to be notified if a line containing the word "EXCEPTION" is logged. In another, we would like to be notified if anything is logged at all. And still other log files should be ignored altogether.

Has anyone a suggestion on a decent way of accomplishing this? We were really hoping to find a solid, existing solution rather than rolling our own monitoring scripts, but we're not afraid to get our hands dirty if that turns out to be the best answer.

Cheers!

Accepted Answer

1

If I were doing this, I'd opt for a simple script running for each log.

For example, if I wanted to be pinged when the word "EXCEPTION" showed up in my app's error log:

#!/bin/bash

LOGFILE=$HOME/logs/user/error_myapp.log
SEARCHFOR="EXCEPTION"
SUBJECT="Exception in my app"
MAILTO="me@domain.com"

tail -Fn0 $LOGFILE | \
while read line ; do
  echo "$line" | grep -q "$SEARCHFOR" && ( echo "$line" | /bin/mail -s "$SUBJECT" $MAILTO )
done

This will mail the log line containing EXCEPTION to me@domain.com whenever it occurs.

You can change the variables at the start of the script to adapt to whatever it is you're monitoring, then run the scripts continuously, perhaps backgrounded in a screen/tmux session.

Hope that helps!

permanent link

answered 17 Feb '16, 20:21

seanf
12.2k●4●18●36
accept rate: 37%

edited 17 Feb '16, 20:23

1

This is great. We will play around with it a bit and see if we can get it working the way we want and post the final solution back here.

There are a few other things we need to work out, for example, if we hit a rocky patch, it would be nice if we could queue up a few errors and send them all at once rather than sending one email for each error.

Also, we'll want these "listeners" running the the background so we'll need to monitor them for uptime using crontab or maybe supervisor.

We'll also need to figure out what happens to these "listeners" when log rollovers occur.

Anyhow, lots of stuff to work out (some of which involves us hashing out our requirements).

Thank you!

(18 Feb '16, 15:25) kirkb

1

So we ended up going a different way with this. We added the ability to view log files from our application. I'm going to go ahead and mark this as the accepted answer, though, in case it helps someone else in the future.

Thanks for your help!

(22 Feb '16, 23:20) kirkb

Monitoring log files

Follow this question

Related questions