Monitoring answers “is it up?” Detection answers “is something wrong on purpose?” This is the catalogue of rules the lab actually evaluates — 18 over logs and 18 over metrics — each one shown with its real expression, the threshold that trips it, and the MITRE ATT&CK technique it maps to. Everything below is copied verbatim from the live config on the host (Prometheus alert rules and the Loki ruler); a rule that isn’t running doesn’t get a card. For how the sources, alert path and dashboard fit together, see the SOC.
Detection Engineering
Every alert rule the lab actually runs — the live LogQL and PromQL, what each one catches, and how it maps to MITRE ATT&CK. Copied from the config on the host, not a wish list.
53
live alert rules — 18 on logs, 18 on metrics
7
log sources feeding detection
16
MITRE ATT&CK techniques mapped
5m
the window most log rules evaluate over
ATT&CK coverage
Credential AccessPrivilege EscalationReconnaissanceCommand and ControlInitial AccessExecutionDefense EvasionPersistenceImpact
Security detections
These fire on adversary behaviour. Each is mapped to the MITRE ATT&CK technique it represents — click the badge for the technique's own page. 23 rules across 3 pages.
SSHBruteForce
critical
sum(count_over_time({job="auth"} |= "Failed password" [5m])) > 10tripsMore than 10 failed SSH passwords in a rolling 5 minutes. Fires after 0m.
Password guessing against SSH. Kept as the aggregate half of a pair: it still catches a distributed attempt spread across many source addresses that the per-source rule below would miss. Proven end to end on 2026-09-06 with two deliberately generated failed logins, after 30 days of matching nothing.
SSHBruteForceSource
critical
sum by (src_ip) (count_over_time({job="auth"} |= "Failed password" | regexp `from (?P<src_ip>[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)` [5m])) > 5tripsMore than 5 failed SSH passwords from one source address in 5 minutes. Fires after 0m.
The same behaviour as above, but attributed. The aggregate rule cannot name the host to block; only this one can. The source address is parsed at query time rather than indexed, so it costs nothing in cardinality. Real observed baseline across 28 days of raw logs: zero.
SudoAuthFailureBurst
warning
sum(count_over_time({job="auth"} |= "sudo" |~ "(authentication failure|auth could not identify password)" [5m])) > 15tripsMore than 15 rejected sudo passwords in 5 minutes. Fires after 0m.
Local privilege escalation attempts. Threshold derived from 28 days of rotated raw logs where the real maximum was 12, all of it one person fighting a password prompt. It first shipped at 5 against a baseline measured over 7 days that was believed to be 30, and would have fired on ordinary use.
SocAnalystSilent
warning
absent_over_time({job="soc-analyst"} [20m])tripsNo output at all from the AI SOC analyst for 20 minutes, sustained for 10.
The analyst layer failing quietly. It is an ENHANCEMENT — the SOC is unaffected if it stops — but a dead analyst that nobody notices leaves someone believing alerts are being triaged when they are not. This only became possible once the analyst emitted an idle heartbeat: before that, a healthy analyst with no alerts to work on produced nothing, so silence and death were indistinguishable and an absence rule would have fired constantly during normal operation.
Assurance
How it was built →
FirewallBlockSpike
warning
sum(count_over_time({job="pfsense", action="block"} | dst_ip != "" | dst_ip !~ ".*[.]255" | dst_ip !~ "2(2[4-9]|3[0-9])[.].*" | dst_ip !~ "ff0.*" [5m])) > 75tripsMore than 400 firewall blocks in 5 minutes, sustained for 2. Fires after 2m.
A burst of dropped packets at the perimeter, which is the shape of a port scan or recon sweep. Threshold derived from 46 days of the raw firewall log: median 75, p95 225, max 347.
FirewallBlockSourceFlood
warning
sum by (src_ip) (count_over_time({job="pfsense", action="block"} | dst_ip != "" | dst_ip !~ ".*[.]255" | dst_ip !~ "2(2[4-9]|3[0-9])[.].*" | dst_ip !~ "ff0.*" [5m])) > 40tripsMore than 250 blocks from a single source in 5 minutes, sustained for 2. Fires after 2m.
One noisy host hiding inside a normal-looking total. Observed per-source maximum over the same window was 191.
FirewallOutboundBlock
warning
sum(count_over_time({job="pfsense", action="block", direction="out"} [5m])) > 0tripsAny outbound block at all. Fires after 0m.
Every one of 410,982 parsed firewall events across 46 days was inbound. An outbound block therefore means something changed: a new egress rule, or a host on this network reaching for something it should not. Zero false positives by construction.
VpnInterfaceBlockSpike
critical
sum(count_over_time({job="pfsense", action="block", interface="tun_wg0"} [5m])) > 5tripsMore than 5 blocks on the WireGuard interface in 5 minutes. Fires after 0m.
Traffic being denied on the one interface that is supposed to carry only trusted, authenticated peers. Either a misconfigured client or something probing the tunnel.
AgentAuditCriticalRisk
critical
sum(count_over_time({job="agent-auditor", risk="critical"} [5m])) > 0tripsAny action the auditor scored as critical risk. Fires after 0m.
The AI agent with real access to this lab attempting something the audit plane considers dangerous. The agent builds; a separate plane judges.
AgentAuditPolicyViolation
critical
sum(count_over_time({job="agent-auditor", verdict="POLICY_VIOLATION"} [5m])) > 0tripsAny POLICY_VIOLATION verdict. Fires after 0m.
The agent crossed a written rule. Observe-only today, so this alert is the enforcement.
AgentAuditUnverifiable
warning
sum(count_over_time({job="agent-auditor", verdict="UNVERIFIABLE"} [10m])) > 0tripsAny UNVERIFIABLE verdict in 10 minutes. Fires after 0m.
The auditor could not prove what the agent claimed. Not proof of wrongdoing, and that is the point: an unverifiable claim is a gap in evidence, which is worth knowing about separately from a violation.
Assurance
How it was built →
AgentAuditReviewFailure
warning
sum(count_over_time({job="agent-auditor", stage="ReviewFailed"} [10m])) > 0tripsAny ReviewFailed stage in 10 minutes. Fires after 0m.
The review step itself broke. A detection plane that fails silently is worse than none, so its own failure is an alert.
Assurance
How it was built →
AuthentikLoginFailureBurst
critical
sum by (client_ip) (count_over_time({job="authentik-events", action="login_failed"} [5m])) > 12tripsMore than 12 failed identity-provider logins from one client address in 5 minutes. Fires after 0m.
Credential stuffing or password spraying against single sign-on, which fronts every service in the lab. These events never appear on the identity provider's stdout; they are lifted out of its database by a dedicated exporter with an allow-listed projection.
AuthentikSuspiciousRequest
warning
sum by (client_ip) (count_over_time({job="authentik-events", action="suspicious_request"} [5m])) > 0tripsAny request the identity provider itself flags as suspicious. Fires after 0m.
The identity provider's own judgement, surfaced rather than left in its database.
AuthentikAuthenticatorRemoved
critical
sum by (username) (count_over_time({job="authentik-events", action="custom_static_authenticator_disable"} [5m])) > 0tripsAny MFA authenticator being removed. Fires after 0m.
Removing a second factor is a normal admin action and a classic account-takeover step. It is rare enough here to alert on every occurrence rather than a threshold.
ContainerFatalError
critical
sum by (container_id) (count_over_time({job="containers"} |~ `(?i)(panic:|level=fatal|"level":"fatal"|\[fatal\]|traceback \(most recent)` [5m])) > 0tripsAny panic, fatal, or Python traceback in any container. Fires after 0m.
A service crashing rather than degrading. Matched with a top-level alternation and no leading wildcard, deliberately: the other shape silently matches nothing on this Loki version.
Health
How it was built →
ContainerErrorSpike
warning
sum(count_over_time({job="containers"} |~ `(?i)(level=error|"level":"error")` [5m])) > 25tripsMore than 25 container error lines in 5 minutes, sustained for 2. Fires after 2m.
Something is broken and nobody has opened a terminal. Derived over the full 7-day retention: 13 non-zero buckets, median 8, second-highest 17, max 65. The one bucket above 25 was a real fault, so the threshold is confirmed rather than raised. Seven days is the honest maximum here, because a container's log file is deleted when the container is recreated.
Health
How it was built →
ContainerLogIngestionStopped
critical
absent_over_time({job="containers"} [15m])tripsNo container logs at all for 15 minutes. Fires after 15m.
The detection stack going blind. Everything above depends on logs arriving, so the absence of logs is itself the signal. This is the rule that watches the watchers.
FirewallBroadcastStorm
warning
sum(count_over_time({job="pfsense", action="block"} | dst_ip =~ ".*[.]255" [5m])) > 2500tripsMore than 2,500 broadcast or multicast blocks in 5 minutes. 7-day observed maximum was 919.
Broadcast storms — a switching loop, DHCP starvation, NetBIOS amplification. Split out from the unicast rules on 2026-09-09 after measuring that 98.3 per cent of all firewall blocks were broadcast noise: a device opening Spotify was tripping the security rules. Deliberately NOT a security signal; the threshold sits above a known-benign spike.
FirewallBroadcastSourceStorm
warning
sum by (src_ip) (count_over_time({job="pfsense", action="block"} | dst_ip =~ ".*[.]255" [5m])) > 1500tripsA single source broadcasting more than 1,500 times in 5 minutes. 7-day maximum was 787, which was benign.
One device flooding the segment. Network health rather than security — the per-source unicast rule below is the one tuned for adversary behaviour.
NetworkDriftUnexpectedService
warning
sum(count_over_time({job="network-twin", detection="DRIFT-004"} [10m])) > 0tripsAny confirmed DRIFT-004 in the last 10 minutes. No additional wait: the finding already survived two consecutive trustworthy observations before a line was written.
A reachable service that is not in the NetBox inventory. The finding this platform exists for. Proven end to end on 2026-09-10 with a listener bound on tcp/8888, confirmed after two observations and delivered to the phone.
NetworkDriftMissingService
warning
sum(count_over_time({job="network-twin", detection="DRIFT-005"} [10m])) > 0tripsAny confirmed DRIFT-005 in the last 10 minutes. Requires three consecutive misses upstream, roughly 45 minutes.
A documented service that has stopped answering. Heavily gated on purpose: a service is 'missing' for one observation during any restart, and no missing-service finding is ever raised from a failed or suspect scan.
NetworkDriftHighSeverity
critical
sum(count_over_time({job="network-twin", event_type="network_drift", severity=~"HIGH|CRITICAL"} [10m])) > 0tripsAny drift event the engine rated HIGH or CRITICAL.
Severity is set deterministically by the drift engine, not by this rule. If it ever emits HIGH or CRITICAL that judgement was already made from evidence, so it pages rather than sitting on a dashboard.
Health & availability
These fire when the lab is hurting itself — but a stopped service or a flooded disk is also how the impact stage of an attack looks, so they double as tripwires.
InstanceDown
critical
up == 0tripsA scrape target unreachable for 10 continuous minutes.
A service, or the whole box, is gone. Primarily availability, but a service stopped or cut off is also how the impact stage of an attack looks.
DiskAlmostFull
warning
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) > 85tripsRoot filesystem over 85% full for 15 minutes.
The classic silent killer: a full disk stops logging, databases and backups. Catching it early keeps the detection stack alive to detect anything else.
Health
How it was built →
HostMemoryPressure
warning
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.15tripsAvailable memory under 15% for 5 minutes.
Memory exhaustion on a box that has frozen before under concurrent load.
Health
How it was built →
HostSwapNearlyFull
warning
see /etc/prometheus/alert_rules.ymltripsSwap nearly exhausted for 10 minutes.
The step before the out-of-memory killer starts choosing victims.
Health
How it was built →
HostSwapThrashing
critical
rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m]) > 128tripsSustained swap in/out above 128 pages per second for 5 minutes.
The machine spending its time moving memory rather than doing work.
Health
How it was built →
HostIOStall
critical
rate(node_pressure_io_stalled_seconds_total[5m]) > 0.05tripsI/O pressure stalling above 5% for 5 minutes.
Processes blocked waiting on disk, measured from pressure stall information rather than inferred from utilisation.
Health
How it was built →
HostDiskSaturation
warning
rate(node_disk_io_time_seconds_total{device=~"sd.*|vd.*|nvme.*"}[5m]) > 0.60tripsDisk busy over 60% for 5 minutes.
Sustained saturation on the underlying storage.
Health
How it was built →
HostMajorPageFaultStorm
warning
rate(node_vmstat_pgmajfault[5m]) > 25tripsMore than 25 major page faults per second for 5 minutes.
Memory pressure showing up as disk reads.
Health
How it was built →
HostClockUnsynchronised
warning
node_timex_sync_status == 0tripsClock unsynchronised for 15 minutes.
Time drift breaks log correlation, certificate validation and every time-windowed detection above.
HostClockDrifting
warning
abs(node_timex_offset_seconds) > 0.5tripsClock offset over 0.5 seconds for 10 minutes.
Drift before it becomes a correlation problem.
Health
How it was built →
LabClockWrongVsWorld
critical
abs(lab_clock_external_skew_seconds) > 30tripsLocal clock more than 30 seconds from external reference for 10 minutes.
The whole lab agreeing on a time that is wrong, which internal comparison alone cannot catch.
LabTimeServerUnsynchronised
warning
lab_ntp_server_stratum == 0 or lab_ntp_server_stratum >= 16tripsLab time server unsynchronised for 20 minutes.
The source everything else trusts having no source of its own.
Health
How it was built →
LabTimeServerOffset
critical
abs(lab_ntp_server_offset_seconds) > 5tripsTime server offset over 5 seconds for 10 minutes.
The time server drifting while still claiming to be synchronised.
Health
How it was built →
LabClockCheckStale
warning
lab_clock_external_check_up == 0 or absent(lab_clock_external_check_up)tripsThe external clock check itself missing for 30 minutes.
A check that has stopped running reports no problems, which looks identical to no problems.
Assurance
How it was built →
LabBackupStale
critical
time() - lab_backup_last_success_seconds > 108000tripsNo successful backup in 30 hours.
Backups silently stopping, which is only ever discovered when one is needed.
LabBackupUndersized
critical
lab_backup_last_size_bytes < 2000000tripsLast backup under 2 MB.
A backup that ran, reported success, and contains almost nothing.
LabBackupIncomplete
critical
lab_backup_last_artefacts < 4tripsFewer than 4 artefacts in the last backup.
A partial set that would restore some services and not others.
LabBackupMetricAbsent
warning
absent(lab_backup_last_success_seconds)tripsThe backup metric missing entirely for 45 minutes.
Not a failed backup but a missing answer, which is the more dangerous of the two.
Assurance
How it was built →
NetworkTwinSilent
warning
absent_over_time({job="network-twin"} [40m])tripsNo records at all in 40 minutes, allowing for two missed 15-minute runs plus the randomised delay.
Absence-as-signal. The twin writes an audit record every run, so silence means network drift is unmonitored. A component that only writes on findings cannot be told apart from a dead one.
LabCertExpiringSoon
warning
(lab_cert_expiry_timestamp_seconds - time()) / 86400 < 21tripsAny monitored certificate within 21 days of expiry.
21 days is deliberate: Caddy renews at roughly 30 days remaining, so this stays silent in normal operation and speaks only when a renewal has failed or a hand-copied certificate has gone stale. The pfSense GUI serves a manual copy of Caddy's wildcard, which does not renew with it.
LabCertExpiringCritical
critical
(lab_cert_expiry_timestamp_seconds - time()) / 86400 < 5tripsAny monitored certificate within 5 days of expiry.
The last useful warning before a browser starts refusing the firewall admin page — which is exactly the moment you least want to be training yourself to click through certificate warnings.
LabCertExpired
critical
lab_cert_expiry_timestamp_seconds < time()tripsA monitored certificate is already past its expiry date.
Reported rather than inferred: the probe reads the certificate without verifying it, so an already-expired certificate still yields a date instead of a blank.
LabCertChainInvalid
warning
lab_cert_chain_valid == 0 and lab_cert_check_up == 1tripsA served certificate fails verification against the system trust store while the endpoint is answering.
A certificate can be well within date while the server omits its intermediate, which browsers reject. A different fault from expiry, so it gets its own rule rather than being folded in.
LabCertCheckFailed
warning
lab_cert_check_up == 0tripsAn endpoint did not answer a TLS probe for 45 minutes.
Distinguishes 'the certificate is bad' from 'nothing answered', which are different problems with different fixes.
LabCertExporterStale
warning
time() - lab_cert_exporter_last_run_timestamp_seconds > 3600tripsThe certificate exporter has not completed a run in over an hour.
node-exporter keeps serving a textfile after its writer dies, so every certificate metric would freeze at its last good value and the expiry alerts would never fire. This is the beacon that makes that visible.
LabCertMetricAbsent
warning
absent(lab_cert_exporter_last_run_timestamp_seconds)tripsThe certificate metrics are absent entirely for 45 minutes.
Before this exporter existed the lab had no certificate monitoring of any kind: no blackbox exporter, no ssl_ or cert_ series anywhere.
DetectionNearMiss
warning
lab_detection_headroom_ratio > 0.7tripsA rule has sat above 70 per cent of its own threshold for 30 minutes without firing.
Threshold rules are binary — fired or did not — and everything below the line is discarded. A rule sitting at 90 per cent all week leaves no trace anywhere. Routed to a receiver with no webhook: visible in Grafana, never on the phone.
DetectionPersistentLowLevel
warning
min_over_time(lab_detection_headroom_ratio[6h]) > 0.3 and count_over_time(lab_detection_headroom_ratio[6h]) >= 24tripsA rule has stayed above 30 per cent of its threshold for six hours, and only once at least 24 real samples exist.
Sustained low-level activity that no burst-tuned threshold will ever catch — the shape of a slow scan or a patient brute force. The sample guard was added after the first version went pending ten minutes post-install: min_over_time over six hours does not require six hours of data.
DetectionHeadroomAbsent
warning
absent(lab_detection_headroom_rules_measured)tripsThe headroom metrics are absent for 45 minutes.
If the exporter dies the ratios freeze at their last values and near-miss visibility silently stops.
DetectionHeadroomIncomplete
warning
lab_detection_headroom_rules_measured < 5tripsFewer than 5 rules were successfully measured for 30 minutes.
The parser found fewer rules than expected, so it is measuring a subset while reporting confidently on the whole. Worse than measuring nothing, which is why it is its own alert.
Threat-hunting queries
Not every signal is an alert. These are interactive LogQL queries I run in Grafana Explore — the raw streams the alerts above are built on, and where a hunt starts before it ever becomes a rule.
{job="auth"} |= "Failed password"{job="pfsense"} |= "filterlog"sum by (vhost) (count_over_time({job="caddy"} | status >= 400 [1h])){job="authentik-events", action="login_failed"}sum by (job) (count_over_time({job=~".+"}[1h])){job="syslog"}From match to phone
Metric rules and log rules converge on one path. A number crossing a threshold or a log line matching a pattern becomes the same buzz in my pocket — with an all-clear when it clears.
📈 Prometheus · alert_rules
🪵 Loki ruler · security
rule matches
🔔 Alertmanager
ntfy
📱 phone
🔎 investigate
Every for: duration is deliberate. SSHBruteForce fires at 0m because a brute-force is real the instant it's counted; InstanceDown waits 10m so the nightly shutdown and morning boot never page me. send_resolved means the phone also buzzes with the all-clear.