Zabbix Autodiscovery for KVM VM Images

A very short description about Zabbix:

Zabbix Agent: Executes Checks on a system that you want to monitor (for example checking the avialable memory).

Zabbix Server: Central part that receives values from the agents, serves the frontend with data, throws everything into a Database, sends active checks to agents.

Zabbix DB: This is a Postgres DB which stores monitoring values but also configuration data. If you run a huge setup you want to put it on a dedicated box.

Zabbix Web: This is the webinterfaces that connects to the DB to present you the values in a nice way. It also offers a JSON based API.

The monitoring itself is split up into multiple parts:
Checks: This is a thing that collects a specifc dataset on a node. for example the CPU temperature or memory usage.

Trigger: A trigger uses the observer pattern to start an action if one or multiple checks return a specific value (CPU temp higher than X since Y minutes and network traffic higher than Z).

notifications: zabbix can inform you by mail, jabber, sms and several other backends about trigger that matched. It is also possible to create escalation paths or repeat notifications in an interval if nobody fixed the issue or acknowledged it.

In a normal case, you define the specific checks for a server. One commen scenario is the disksize monitoring for virtualmachines. But you don’t want to hardcode the paths to images or the amount of them on the zabbix server/DB, you want to detect them dynamically. Zabbix offers low level discovery for this use case:

A local discovery script returns a JSON array with a macro => value notation, you can create check/trigger templates who uses these macros. Here is a short ruby snippet which detects KVM images and creates the JSON data for you (every image is saved as /var/lib/libvirt/images/VMNAME/image.qcow2, the name should always be an Integer):

#!/usr/bin/env ruby

##
# Written by Tim Meusel
# improved by jokke
# Discovers qcow2 disks + customer IDs
# Returns JSON, tested on Ruby 1.8.7
##

require 'rubygems'
require 'json'

data={:data => []}
ignored_vs = [1]
Dir.glob("/var/lib/libvirt/images/*/image.qcow2").each do |dir|
  vmname = Integer(dir.split(File::SEPARATOR)[5]) rescue next
  next if ignored_vs.include? vsname
  data[:data] << { '{#VMNAME}' => vmname, '{#VMPATH}' => dir}
end

puts JSON.pretty_generate(data)

this will return something like this:
{
"data": [
{
"{#VMNAME}": 2,
"{#VMPATH}": "/var/lib/libvirt/images/2/image.qcow2"
},
{
"{#VMNAME}": 3,
"{#VMPATH}": "/var/lib/libvirt/images/3/image.qcow2"
},
{
"{#VMNAME}": 4,
"{#VMPATH}": "/var/lib/libvirt/images/4/image.qcow2"
}
]
}

You can define this in your zabbix agent config as a new UserParameter (zabbix agent should never run as root. depending on your permissions it can’t read the directories with the images itself so you need to run it with sudo or update permissions):

# cat /etc/zabbix/zabbix_agentd.d/vmimages.conf 
UserParameter=discovery.vmimages,sudo /usr/bin/lld_vsdisks.rb

It is now possible to configure a discovery rule. Login to your Zabbix webfrontend -> Configuration -> Templates -> Select the template where you want to add the rule and checks -> Discovery rules -> Create discovery rule. As they key you know of to specify the name of the UserParameter, type is “Zabbix agent”. Now the Server knows that every host with this template should have a UserParameter which points to a script which returns JSON with macros. You can also add “Item prototypes” to this discovery rule. Here a screenshot of the rule itself and than for a check:

Zabbix has a build in check for getting a file size, vfs.file.size IIRC, but this also runs with the zabbix agents privileges, you can create another UserParameter for checking the size as root:

# cat /etc/zabbix/zabbix_agentd.d/vmimages.conf 
UserParameter=vm.disksize[*],sudo /usr/bin/get_size.sh "$1"
UserParameter=discovery.vmimages,sudo /usr/bin/lld_vsdisks.rb

And the script:

# cat /usr/bin/get_size.sh
#!/bin/bash
wc -c "$1" | awk '{print $1}'

You *could* do something like this in the vmimages.conf:

UserParameter=vm.disksize[*],sudo wc -c  "$1" | awk '{print $1}'

But you need to escape the second $1 which is really PITA….

You should now have a sweet solution to automatically discover new VM images and monitor their size. You can also add trigger prototypes to the template if you want or add additional checks.