Cronjob monitoring with Prometheus Pushgateway

Today I discovered that one of my backup servers could not boot for about 2 weeks because the BIOS battery was empty, and my settings were lost. I was really shocked because I thought my backup setup was quite good, but without monitoring, this must be verified manually, which I didn't do.

This server is connected to an IP plug and gets started by another server. When the backup is completed, it shuts itself down. On the next day, this process starts over again. I plan to write a blog post about this setup in the future.

My first thought for solving this problem was the Node Exporter textfile collector I wrote about in my last blog post. But the issue with this solution is that I don't have a Node Exporter running on this server because it only runs when a backup should be copied to it. So I needed to search for something else.

Prometheus Pushgateway

I am in the Prometheus ecosystem for a long time, so I check out the Pushgateway multiple times, but it was never the right solution for my problem. When I thought about a solution for this problem, I instantly remembered the Pushgateway and checked if it could be used for that. And yes this use case is exactly for what it was designed for. Prometheus is a Pull based system, but sometimes this is not possible.

If you want to use the Pushgateway, read the documentation because there are multiple things to consider or things that can make problems.

Setup

I could describe how to run it but everything you need is in the GitHub README. I run it as a Docker Container.

One important thing to do is to enable the persistence of the pushed metrics because without that, the Pushgateway will lose all metrics when it will be restarted. I use the Folder /var/local/pushgateway for it, so I specified --persistence.file=/var/local/pushgateway/metrics.

Example

Below is a short Example Bash Script that uses curl to update the job last run time at the end of the Script. I use "pushgateway" as the "JobName" below.

#!/bin/bash

# exit if a command returns a nonzero exit status or an undefined variable is used!
set -eu

# work like rsync, ...

cat << EOF | curl --data-binary @- http://[IP or Hostname]:9091/metrics/job/[Job Name]/instance/[Hostname]
# HELP cronjob_last_run_24h Cronjob last run, in unixtime.
# TYPE cronjob_last_run_24h gauge
cronjob_last_run_24h{name="backup"} $(date +%s)
EOF

If you are interested how I monitor jobs with such a setup, checkout my last blog post about the Node Exporter Textfile Collector.