Munin is a great open-source monitoring solution for servers. I’ve used it in the past and was really happy with the results, but when I was using it my server was just a little VPS and running the server and node on the same box caused some system resource issues. But now having colo equipment in the rack at DimeNOC I decided to give it another go.

Installation was pretty much straightforward, the instructions on their website were sufficient to get the server up and running. There was some confusion, however, getting some plugins to play nicely. So, to pick up where the official installation instructions left off, I’ll go over some of the issues and solutions I was able to come up with.

What you’ll need to know from the installation

  • The location of the munin logfiles. You can set this in the munin.conf file. I used /var/log, seemed like the obvious choice. Your preferences may vary.
  • The location of your munin config files, and plugins.
    • Config files are typically placed in /etc/opt/munin/
    • Plugin files are typically in /etc/opt/munin/plugins
    • Plugin special configs are typically in /etc/opt/munin/plugin-conf.d
  • If you installed from source, you should have all that information in the installation scrollback. If you installed from an RPM, first try the paths above; if they’re not there then time to run the ‘ol trusty updatedb and do a locate.

Ok, so now that we have all that stuff taken care of, it’s time to dive right in. By default, there are no plugins enabled. In order to enable them we need to put them in the /etc/opt/munin/plugins directory. If you’d like some help determining what plugins to use, run the munin-node-configure script that came with the installation:

-bash-3.2$ /usr/sbin/munin-node-configure --suggest

Plugin | Used | Suggestions
------ | ---- | -----------
acpi | yes | yes
amavis | no | no
apache_accesses | yes | yes
apache_processes | yes | yes
apache_volume | yes | yes
apc_envunit_ | no | no [no units to monitor]
bonding_err_ | no | no [No /proc/net/bonding]
courier_mta_mailqueue | no | no [spooldir not found]
courier_mta_mailstats | no | no
courier_mta_mailvolume | no | no

This gives you a general idea of what plugins are compatible with your server, and why others may not run. It’s a great tool to begin with. If you enable *all* the plugins munin comes with you’re going to be left with a lot of empty graphs and log messages complaining about errors. Instead, let munin do some work for you, it can move the ‘suggested’ plugins into the correct place and enable them for you by running this command:
-bash-3.2$ /usr/sbin/munin-node-configure --shell | sh -x

Once that’s been done, look over the list of plugins that it didn’t suggest to see if there any additional ones you’d like to include. Usually the reason for excluding them is because a prerequisite wasn’t met, which most can be corrected.

The crontab

There’s a script that needs to run every five minutes to collect data from the node, /usr/bin/munin-cron This job must not be run as the root user, it should be in the munin users’ crontab. You can either su over to munin’s account or as root edit the users crontab with crontab -u munin -e. To have this job run every five minutes, simply insert this line:
*/5 * * * * /usr/bin/munin-cron >/dev/null 2>&1

The last part of that line keeps the cron daemon from sending the munin user an email each time the job runs, which will eventually fill his mailspool and use unnecessary disk space.

Now you’re ready to start up the munin-node. There’s a couple of ways to do it, the easiest is just to use the service command:
-bash-3.2$ sudo service munin-node start

Once that’s running, go ahead and (as munin) kick off the /usr/bin/munin-cron from the command line. This should begin the data collection process and log any errors (and there will probably be some) to the update log, in my case that’s located in /var/log/munin-update.log.

In addition to the error logs, you can run each munin plugin individually with the munin-run command-line tool, which provides some details on why something may have failed. Can be run:
/usr/sbin/munin-run [plugin_name]

The plugins that gave me the most trouble were Apache (thanks to some necessary mod_rewrite stuff on this site), sendmail_mailstats and mysql_. I’ll go over the problems with each of those below.

apache_ plugins

This really should be the easiest plugin to enable, all you need to do is enable server-status in the configuration file, httpd-info.conf ( I use Apache2):
SetHandler server-status
Order deny,allow
Deny from all
Allow from 127.0.0.1

I had diis.net listening on it’s IP address and localhost and mod_rewrite was sending everything to /front (where this blog lives) so that really messed with the server-status URL. A simple change in the httpd.conf file fixed that right up.

sendmail_mailstats

This was a permissions issue. The user munin could run the mailstats program, but the /var/log/mail/statistics file it reads was owned by root. So the simple solution was to chown the file to root:munin, which made it start collecting data right away. I don’t really see any harm in making that change, albeit I’m not a big fan of changing default system permissions.

mysql_

Ok, for this plugin to work you need to create a new unprivileged MySQL user account. You also need to put some special directives in the config file /etc/opt/munin/plugin-conf.d/munin-node so munin knows how to use the mysqladmin program to get the stats:
[mysql*]
user root
group root
env.mysqladmin /usr/bin/mysqladmin
env.mysqlopts -u MUNIN_USER_NAME --password=MUNIN_PASSWORD

This should have worked fine, but I believe there’s a limitation of what types of character you can have in the password since you’re passing the command from a config file into a variable and then as an exec to the shell. My password had some special characters in it, once it was simplified to numbers and letters the queries worked fine. There shouldn’t be any risk in a simple password for an unprivileged account. Especially if your server has a limited number of users that have shell access and you have MySQL’s port firewalled off from the internet. Note: That’s one dash in front of the ‘u’ variable and two in front of the ‘password’ variable.

And that’s pretty much it for now, I hope this helps someone stop pulling out their hair out. Got a tip or trick related to munin? Drop it in the comments below.