fix(sync-alerts): mail a nightly failure every night, not every other

The 24h cooldown starts when the mail goes out, a moment after the run
failed. The same job failing at the same minute the next night was a few
hundred milliseconds short of 24h and its mail was suppressed. An expired
Intune secret therefore alerted on 04.11. but not on 05.11. The cooldown is
23h now, which still keeps an hourly schedule to one mail a day.
This commit is contained in:
2026-11-05 14:32:06 +01:00
parent a5c2226248
commit ade9675c8c
3 changed files with 19 additions and 4 deletions
+3 -1
View File
@@ -3315,7 +3315,9 @@ scheduler) is one `sync_runs` row, written by `record_sync_run()` in
`app/services/sync_run_service.py` from its own session. The Scan Jobs page
shows them as **Sync Health** (per source) and **Sync Runs** (the ledger); the
dashboard shows a banner while a source is failed or stale; a failed run mails
the sync-alert recipients once per source per 24h.
the sync-alert recipients at most once per source per 23h (not 24h: the
cooldown starts after the send, so a nightly job's next failure would land
just inside it and every other night's mail was dropped).
## Failed vs. completed with errors
+7 -3
View File
@@ -9,7 +9,7 @@ The ledger uses its OWN session. The job's session may be mid-transaction or
already rolled back when the job dies, and the record of the failure must not
depend on it.
A failed run sends one mail per source per ALERT_COOLDOWN (24h) — an hourly
A failed run sends one mail per source per ALERT_COOLDOWN (~a day) — an hourly
schedule against a dead API must not send 24 mails — and states the last
successful sync, i.e. how long the coverage gap already is.
"""
@@ -57,7 +57,11 @@ CONFIG_KEYS = {
# and never stale without an enabled schedule (manual-only sources).
STALE_AFTER = timedelta(hours=36)
CRON_STALE_AFTER = timedelta(days=8) # a cron expression we do not parse
ALERT_COOLDOWN = timedelta(hours=24)
# 23h, not 24h: `alerted_at` is stamped after the mail went out, so a nightly
# job failing at the same minute the next day is a few hundred ms short of
# 24h and its mail was suppressed — one alert every OTHER night (field report
# 05.11.2026, expired Intune secret).
ALERT_COOLDOWN = timedelta(hours=23)
SETTING_ENABLED = "sync_alert_enabled"
SETTING_RECIPIENTS = "sync_alert_recipients"
MAX_ERRORS_KEPT = 50
@@ -525,7 +529,7 @@ from this source are not detected.</b> Check the connection and credentials in
Settings, then run the sync again from Scan Jobs.</p>
<p><a href="{html.escape(base)}/scans">Open Scan Jobs</a></p>
<p style="color:#6b7280;font-size:12px">You receive at most one mail per source
per 24 hours while it keeps failing.</p>
per day while it keeps failing.</p>
</body></html>"""
for user_id, email, _name in recipients:
+9
View File
@@ -183,6 +183,15 @@ def test_cooldown_suppresses_repeat_mail_for_same_source():
assert svc.in_cooldown(None, now) is False
def test_cooldown_lets_a_nightly_job_alert_every_night():
# alerted_at is stamped after the send; the next night's failure at the
# same minute is just under 24h later and must still mail.
now = datetime.now()
yesterday = SyncRun(source="intune", status="failed", started_at=now,
alerted_at=now - timedelta(hours=24) + timedelta(seconds=1))
assert svc.in_cooldown(yesterday, now) is False
def test_health_flags_failed_and_stale_sources():
now = datetime.now()
ok = SyncRun(source="wazuh", status="completed", started_at=now,