What the watchdog is
The QFlex watchdog is a small, database-independent safety net. On a fixed schedule it verifies two things QFlex depends on to alert you at all: that the QFlex database is reachable, and that the disks are not nearly full.
If either check fails, it emails you an alert using SMTP settings it caches locally while healthy, so it can still reach you even when the database is down. It is the thing that still notifies you when the normal, database-driven alert path has gone silent, for example when the host has run out of disk.
Why it matters: conventional monitoring cannot report on the failure of the monitoring platform itself. The watchdog closes that gap.
How it works
Every cycle, 60 seconds by default after a short startup delay, the watchdog:
1. Probes the database with a cheap read. While healthy, it refreshes its cached SMTP settings from QFlex and writes them to a local file.
2. Probes disk usage across all filesystem roots plus the temp and Tomcat base directories, or a list you specify.
3. Decides. If the database is unreachable for several consecutive checks, it raises an alert. If any disk is at or above the critical threshold, it raises an alert. If a disk is at or above the warn threshold but not critical, it logs a loud early warning while everything still works.
4. De-duplicates. While degraded it re-sends at most once every configured interval.
5. Recovers. It sends a single monitoring recovered email once the database and disks are healthy again.
Because it caches SMTP settings to disk, a watchdog restarted during an outage still knows how to email you. At notification time it never touches the database.
Getting started with zero configuration
The watchdog is enabled by default and works out of the box using your existing QFlex SMTP configuration. No setup is required for basic protection.
To confirm it is running, check the monitor log after startup. You will see the watchdog cycle activity, and any disk warnings, under the monitor logger.
Configuration
To tune the watchdog, copy the sample file and edit it:
Copy conf/watchdog.properties.sample to conf/watchdog.properties under your Tomcat base directory.
If the live file does not exist, the watchdog creates it with defaults and appends the cached SMTP settings on the first healthy run. The file lives under the Tomcat base directory, so it is portable across the Windows and Unix installers with no absolute paths. All keys are optional. Anything omitted uses its default.
Settings:
watchdog.enabled (default true). Master switch. Set to false to disable the watchdog entirely.
watchdog.interval.sec (default 60). How often to run the health check, in seconds. Minimum 15.
watchdog.db.failures.threshold (default 3). Consecutive failed database probes before an alert is raised. With interval 60 and threshold 3, the database must be unreachable for about 3 minutes.
watchdog.disk.paths (default blank, meaning auto). Filesystems to check. Blank auto detects all roots plus temp and Tomcat base. To restrict or extend, list comma separated native paths.
watchdog.disk.warn.pct (default 85). Disk usage percentage that triggers a loud log warning. This is an early warning while everything still works.
watchdog.disk.crit.pct (default 95). Disk usage percentage that raises a watchdog alert email.
watchdog.resend.min (default 15). While degraded, re-send the alert at most once every this many minutes.
watchdog.recipients (default blank). Recipient override, comma or semicolon separated. Blank uses the admin email configured in QFlex under Administration then SMTP, falling back to the monitor SMTP email to address.
Auto managed SMTP keys:
The keys for SMTP server, from, to, username, password, and HTML flag are written automatically by the watchdog from the QFlex database on each healthy cycle, so it can still send during an outage. You normally do not set these by hand.
Security note: the cached SMTP password is stored in clear text, as it is in the database, so the watchdog creates watchdog.properties owner readable only. Keep it that way.
Example configurations
Faster detection, with 30 second checks and an alert after about 90 seconds of database downtime:
watchdog.interval.sec = 30
watchdog.db.failures.threshold = 3
Tighter disk thresholds on a busy host:
watchdog.disk.warn.pct = 80
watchdog.disk.crit.pct = 90
Restrict disk checks to specific volumes:
On Unix, watchdog.disk.paths = /,/opt/qflex,/var/log
On Windows, watchdog.disk.paths = C:\,D:\
Send watchdog alerts to a dedicated on call address:
watchdog.recipients = oncall@yourcompany.com;ops-team@yourcompany.com
What the alerts look like
Degraded: QFlex monitoring is degraded and monitors may not be firing, with the reason given, for example the database being unreachable for several consecutive checks or a disk being nearly full. The message notes that it was sent by the QFlex watchdog using cached settings, because the normal alert path may be unavailable.
Recovered: QFlex monitoring has recovered, the database is reachable again, and disk usage is reported as healthy.
Operational tips
Route watchdog alerts differently. Because a watchdog alert means the monitoring platform itself is in trouble, consider pointing watchdog.recipients at an on call channel separate from routine monitor alerts.
Mind the warn threshold. The 85 percent warn level is your chance to act before a disk hits critical and starts suppressing normal alerts. Watch the monitor log for these early warnings.
Leave it enabled. The watchdog is intentionally cheap and never throws, so there is little reason to disable it. If you must, set watchdog.enabled to false.
Restarts are safe during an outage. The cached SMTP file means a restart mid incident does not lose the ability to alert you.
Available in QFlex Standard 4.6.3 and later.