The Angry Dome

It's no surprise that puppet can be incredibly useful for managing your nagios config. One common approach is to use puppet's dsl to configure nagios checks using the included nagios_* types, which is a very powerful technique when combined with collected/exported resources; this allows you to export checks from each host you want to monitor, and then have those collected on the nagios server. In this manner you can have hosts automatically show up in nagios once their collection resource has been processed.

I like this approach, but I've been sold on using check_mk along with nagios for a while now, so I need to come up with something a bit different.

For those who are unaware, check_mk consists of an agent (which runs on systems to be monitored by nagios) and a poller (which runs on the nagios server). The idea is that you never need to touch the nagios config files directly; the poller autodetects services and creates a valid nagios config all on its own. In addition, you get a fancy replacement web UI that doesn't look like a time warp from 1995.

In order to work, though, check_mk needs a list of nodes to be monitored and (optionally) a list of tags that describe those nodes. The tags are used by check_mk to apply configurations to resources; for example, you might tag a "production" server with the "production" tag, and configure check_mk to enable 24x7 paging on all services that are so tagged.

So, you can do all this manually, but puppet has all the information you need already. Here's the plan: create a puppet module that all to-be-monitored clients will include, have that module export a config snippet that describes each node, and then have puppet collect those snippets on the nagios server.

I've created such a module, and you can find my puppet module that configures check_mk at my github; an explanation of this module follows below, for the curious.

The checkmk class

The checkmk::agent class

   @@file { "$mk_confdir/$fqdn.mk":

        content => template( "checkmk/collection.mk.erb"),

        notify => Exec["checkmk_inventory_$fqdn"],

        tag => "checkmk_conf",

    }

    @@exec { "checkmk_inventory_$fqdn":

        command => "/usr/bin/check_mk -I $fqdn",

        notify => Exec["checkmk_refresh"],

        refreshonly => true,

        tag => "checkmk_inventory",

    }

The important thing to understand is that the exported resources are created in the scope of the client, but not realized on the client; they are then available to the server which actually realizes them. In this case, each client calling checkmk::agent will have these resources defined in puppet's stored config backend, and the nagios server will later scoop them up and process them. Exported resources are cousins of virtual resources and the syntax in the DSL is similar; you simply precede the name of the resource with "@@".

You will notice that I'm both creating a file resource and an exec resource. In my initial version of this module, I did not have per-node exec resources, and whenever a node changed I triggered a re-inventory of all nodes. This proved to be a bit excessive; using per-node exec statements allows you to inventory only the nodes that change.

The tricks in the template require a little explaining too. I like to be able to add check_mk tags from variables assigned in the node classifier, and this template takes those variables and creates valid check_mk configuration strings. The scope.to_hash.keys stuff allows me to use reflection in order to identify any variables whose names contain the string "check_mk_tags," and I append their corresponding values to the list of tags. This is, again, a workaround for limitations of the dashboard classifier, where we want some tags coming from a higher scope but we want to append to the list, which forces us to use multiple variables.

So, for example, I might have check_mk_tags_webserver = webserver attached to my "webserver" group in the classifier, but also check_mk_tags_paging = "paging|critical" in my "critical nodes" group; I can then place a node in both groups, and this template will smush all of the tags in both variables together (note that you should delimit your tags with a pipe if you assign multiple tags to the same variable).

The other trick is to do a DNS lookup on the fqdn of the host, and if the DNS lookup fails, hard code facter's "ipaddress" as the IP of the host. This prevents check_mk from choking on hosts with broken DNS. In addition, it tags such hosts with "brokendns," and I like to assign a corresponding hostgroup in nagios so I may easily shame whoever created such badness into fixing his problem.

Oh, one last thing; there's nothing stopping you from using facts or any other puppet variable as a tag. Simply append your fact or variable to the 'mktags' array in the ERB template, and you're good to go!

The checkmk::server class

So, that's it! Happy check_mk'ing!