watchdog script to keep process running with cron

Question

Does anyone have a simple but effective script which can be used with cron in order to maintain a process and keep it running, restarting it only if necessary?

If I simply tell the program itself to run on cron, I may get several copies running simultaneously. This program is not smart enough to test whether or not it is already active before firing up another copy of itself.

Accepted Answer

3

Yes. All you need to do is to create a simple bash script like this:

#!/bin/bash

mkdir -p "$HOME/tmp"
PIDFILE="$HOME/tmp/myprogram.pid"

if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |
                           grep -P "^\s*$(cat ${PIDFILE})$" &> /dev/null); then
  echo "Already running."
  exit 99
fi

/path/to/myprogram > $HOME/tmp/myprogram.log &

echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"

Then, run the program on a short cron job, like this:

*/5 * * * * $HOME/cron/watchdog.sh > $HOME/cron/watchdog.log 2>&1

In this example, I have named the script watchdog.sh and placed it in my $HOME/cron directory. We would also want to ensure that execute permissions are set on the script; so we would chmod 755 it.

The script itself also uses the $HOME/tmp directory, so you would want to create that as well. In general, keeping a $HOME/tmp directory is a better idea than relying on /tmp.

permanent link

answered 11 Nov '11, 18:46

ryans ♦♦
5.0k●10●39●60
accept rate: 43%

edited 17 Apr '14, 05:58

Thanks, but I'm actually trying to keep an SSH tunnel alive for a database connection, and the:

/path/to/myprogram & > $HOME/tmp/myprogram.log &

line doesn't seem to work out-of-the-box for ssh.

(11 Nov '11, 18:48) ryans ♦♦

The above is a general format applicable to all programs. Specifically for SSH, you should try something like this:

/usr/bin/ssh -N "-L 12671:127.0.0.1:3306" -oServerAliveInterval=60 -oServerAliveCountMax=3 -oBatchMode=yes -oConnectTimeout=10 "USERNAME@webNNN.webfaction.com" &

Where the following would be replaced by your information:

12671 - your local port on your server. The source of the tunnel.
3306 - the destination port of the tunnel. Would point to the database port. (use 3306 for mysql, 5432 for postgresql)
USERNAME - your username.
webNNN - the number of your webserver; for example, web200.webfaction.com

To be more aggressive in automatically handling migrations and/or IP changes by ignoring host credentials, add the -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no options to the above command as well.

Take a look at this community question for an example.

(11 Nov '11, 18:54) ryans ♦♦

I'm new to Bash scripting, and have a problem with the watchdog script. My user name is replicounts, and the program that needs an occasional restart is aff.py.

The crontab is working fine, and my watchdog.sh script (below) is being executed when it should. But the resulting watchdog.log is:

/home/replicounts/cron/watchdog.sh: line 7: python3.1: command not found

I manually run the program (in webapps/aff/htdocs) with:

nohup python3.1 aff.py &

The python3.1 is available interactively in all my directories. Also note:

[replicounts@web214 cron]$ echo $PATH
/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/replicounts/bin
[replicounts@web214 cron]$ ls -a /home/replicounts/bin
.  ..  .ph

FYI, my watchdog.sh script is:

[replicounts@web214 cron]$ cat watchdog.sh
#!/usr/bin/env bash
PIDFILE="$HOME/tmp/aff.pid"
if [ -e "${PIDFILE}" ] && (ps -u $USER -f | grep "[ ]$(cat ${PIDFILE})[ ]"); then
  echo "Already running."
  exit 99
fi
python3.1 $HOME/webapps/aff/htdocs/aff.py > $HOME/tmp/aff.log &
echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"

What needs to be changed?

John

(25 Apr '12, 14:03) replicounts

cron doesn't operate with the same PATH as your shell account, so try using the full path to everything in your script and in your cron jobs. For example, instead of python3.1, use /usr/local/bin/python3.1. You can find the full path for a particular tool with the which command, eg:

[root@web214 ~]# which python3.1
/usr/local/bin/python3.1

Or, you can explicitly set the PATH in your script.

Hope that helps!

(25 Apr '12, 14:21) seanf

Sorry but how can use this watchdog script with django and memcached? I think we need to restart django after starting memcached...

(02 Aug '13, 05:39) sistineburak

The post directly above is a way to test your scripts in the cron environment. The first post in this answer explains how to create a monitor script. You would have to create the script and test it using the process above and than schedule it in cron, you would have to start memcached first, but that is something you will have to test and make work in the script with debugging.

(02 Aug '13, 21:12) johns

I run this for supervisor but it keeps saying 'Already running.' any thought?

(28 Oct '13, 09:03) jonyk

That means that supervisor (or another process with the same ID that you have in $HOME/tmp/myprogram.pid) is already running.

(28 Oct '13, 11:09) seanf

Good thinking; I think it is working now, I set the pidfile on supervisord to a folder and it runs perfectly. Thank you for your enlightment, Seanf!

(28 Oct '13, 20:29) jonyk

Once you've got this all set up and running, how do you break the process if you make some change and want to restart it? I've got it working for my python script but I made a couple changes to the code and just want to restart it so those changes are in effect.

(14 Sep '14, 15:40) arajendran

Hay men, how are you? How should be a script for maintain ever in on an elastichsearch server? Already are installed the server, and works, but, i can't be all time in the ssh connection for maintain the server on.

Can anyone help me?

(24 Oct '14, 04:58) Yusuf Salah ...

@arajendran:

This command will show you your running processes:

ps jf -u $USER

Just kill the associated processes. The watchdog script should start them again, this time using the latest version.

(24 Oct '14, 05:29) ryans ♦♦

@yusuf-salah-addin:

Can you tell me how you normally start elasticsearch if it's not running? Which command do you use, and does it start the process in the foreground or in the background?

(24 Oct '14, 08:12) ryans ♦♦

Ok, normally i have init the elasticsearch file in bin folder in elasticsearch folder because we can't use sudo for the server comand. '''./bin/elasticsearch This is the command, and i have put the completly file's path, and with this i init the server. You can see all here: http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/running-elasticsearch.html

i think do something like this: https://xenforo.com/community/threads/how-to-basic-elasticsearch-installation-debian-ubuntu.26163/, but, i haven't permissions for create the file.

Thank you very much.

(24 Oct '14, 17:35) Yusuf Salah ...

You would put the file in /home/YOU/bin or anywhere else in your home directory. You can't place files within the standard paths since it is a managed server, but you can place them within your home directory and configure them to run from there.

(24 Oct '14, 23:44) johns

Yes, but, how? In my home is the elasticsearch folder, but, the rest, i don't know how do it; here more documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-service.html

(27 Oct '14, 16:16) Yusuf Salah ...

Just type:

readlink -e ./bin/elasticsearch

That will give you the full path to the elasticsearch binary. Use that path in your watchdog script.

(28 Oct '14, 01:29) ryans ♦♦

an my script should have the same instructions as your script's example?

(08 Nov '14, 20:59) Yusuf Salah ...

Yes, you can't run traditional services, so you have to create a watchdog script which does it, which should be mostly the same as the example above.

(08 Nov '14, 23:10) johns

I'm new to crontab jobs, but have some processes I need to keep running and have a few questions about the script at the top of this thread.

How do I create the "myprogram.pid" file? re: PIDFILE="$HOME/tmp/myprogram.pid"
I imagine the "whoami" variable is my Webfaction user id? re: if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |

Thanks!

(16 Nov '15, 16:44) istarion

How do I create the "myprogram.pid" file? re: PIDFILE="$HOME/tmp/myprogram.pid"

The script creates it.

I imagine the "whoami" variable is my Webfaction user id? re: if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |

The $(whoami) is not a variable. It is an expression that evaluates and substitutes the output of the whoami command at runtime. It will be equal to the user id of whoever runs it, so if you're running it, then yes, it will be your user name.

(16 Nov '15, 20:24) seanf

showing 5 of 21 show 16 more comments

Answer 2

For reference, here is cron's environment:

SHELL=/bin/sh
USER=<username>
PATH=/usr/bin:/bin
PWD=/home/<username>
SHLVL=1
HOME=/home/<username>
LOGNAME=<username>

Additionally, cron runs the commands directly from your $HOME directory. To test cron commands, I like to run something like this:

cd $HOME
unset $(env | awk -F= '/^\w/ {print $1}' | xargs)
export USER=$(/usr/bin/whoami)
export SHELL=/bin/sh
export PATH=/usr/bin:/bin
export PWD=/home/$USER
export SHLVL=1
export HOME=/home/$USER
export LOGNAME=$USER

And then run the command exactly as it appears in the crontab.

watchdog script to keep process running with cron

Follow this question

Related questions