Monitoring Apache with Bluepill

22 December 2009

I always used god to monitor my processes. god is working fine, has a lot of features and is very easy to use, but has some serious memory leaks. To handle this problem I usually set-up a cron to restart god every day.

Recently I read an article about a new process monitoring tool called bluepill based on EventMachine. The DSL looks simple and comes memory-leaks-less.

In this article I’ll cover how to install it and monitor apache.

First let’s grab it :

$ sudo gem install bluepill

Next, we need to configure the syslogger to log the local6 facility used by bluepill.

If your syslogger is syslog, add a line to /etc/syslog.conf :

local6.*          /var/log/bluepill.log

If it’s syslog-ng, add ths to /etc/syslog-ng.conf :

filter f_local6 { facility(local6); };
destination d_local6 { file("/var/log/bluepill.log"); };

log { source(src); filter(f_local6); destination(d_local6); };

You’ll also want to add /var/log/bluepill.log to /etc/logrotate.d/syslog so that it gets rotated.

Next restart your syslogger, for syslog-ng :

$ sudo /etc/rc.d/syslog-ng restart

And finally create the /var/bluepill directory for bluepill to store its pid and sock files.

Finally let’s create our bluepill script to monitor apache in apache-monitor.pill :

Bluepill.application("httpd") do |app|
	app.process("httpd") do |process|
		process.pid_file = "/var/run/httpd/httpd.pid"
		process.start_command = "apachectl start"
		process.stop_command = "apachectl stop"
		process.restart_command = "apachectl restart"
		process.start_grace_time = 10.seconds
		process.stop_grace_time = 10.seconds
		process.restart_grace_time = 10.seconds
	end
end

This simple script just checks if apache is up and if not reboot it.
You can try it by :

$ sudo bluepill load apache-monitor.pill

You can try to kill your httpd processes and you’ll see the magic happens, bluepill will restart them.

Bluepill comes with some great features to monitor your memory and cpu, so you can add to your script :

process.checks :cpu_usage, :every => 10.seconds, :below => 5, :times => 3
process.checks :mem_usage, :every => 10.seconds, :below => 100.megabytes, :times => [3,5]

The first line checks every 10 seconds to make sure the cpu usage of this process is below 5 percent; 3 failed checks results in a restart.
The second line checks every 10 seconds to make sure memory usage is below 100MB; 3 out of 5 failed attempts results in a restart.

If you want your bluepill daemon to run at boot, just add a line to /etc/rc.local :

$ sudo bluepill load /full/path/to/your/script/apache-monitor.pill