fix(sync-alerts): mail a nightly failure every night, not every other
The 24h cooldown starts when the mail goes out, a moment after the run failed. The same job failing at the same minute the next night was a few hundred milliseconds short of 24h and its mail was suppressed. An expired Intune secret therefore alerted on 04.11. but not on 05.11. The cooldown is 23h now, which still keeps an hourly schedule to one mail a day.
This commit is contained in:
+3
-1
@@ -3315,7 +3315,9 @@ scheduler) is one `sync_runs` row, written by `record_sync_run()` in
|
||||
`app/services/sync_run_service.py` from its own session. The Scan Jobs page
|
||||
shows them as **Sync Health** (per source) and **Sync Runs** (the ledger); the
|
||||
dashboard shows a banner while a source is failed or stale; a failed run mails
|
||||
the sync-alert recipients once per source per 24h.
|
||||
the sync-alert recipients at most once per source per 23h (not 24h: the
|
||||
cooldown starts after the send, so a nightly job's next failure would land
|
||||
just inside it and every other night's mail was dropped).
|
||||
|
||||
## Failed vs. completed with errors
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ The ledger uses its OWN session. The job's session may be mid-transaction or
|
||||
already rolled back when the job dies, and the record of the failure must not
|
||||
depend on it.
|
||||
|
||||
A failed run sends one mail per source per ALERT_COOLDOWN (24h) — an hourly
|
||||
A failed run sends one mail per source per ALERT_COOLDOWN (~a day) — an hourly
|
||||
schedule against a dead API must not send 24 mails — and states the last
|
||||
successful sync, i.e. how long the coverage gap already is.
|
||||
"""
|
||||
@@ -57,7 +57,11 @@ CONFIG_KEYS = {
|
||||
# and never stale without an enabled schedule (manual-only sources).
|
||||
STALE_AFTER = timedelta(hours=36)
|
||||
CRON_STALE_AFTER = timedelta(days=8) # a cron expression we do not parse
|
||||
ALERT_COOLDOWN = timedelta(hours=24)
|
||||
# 23h, not 24h: `alerted_at` is stamped after the mail went out, so a nightly
|
||||
# job failing at the same minute the next day is a few hundred ms short of
|
||||
# 24h and its mail was suppressed — one alert every OTHER night (field report
|
||||
# 05.11.2026, expired Intune secret).
|
||||
ALERT_COOLDOWN = timedelta(hours=23)
|
||||
SETTING_ENABLED = "sync_alert_enabled"
|
||||
SETTING_RECIPIENTS = "sync_alert_recipients"
|
||||
MAX_ERRORS_KEPT = 50
|
||||
@@ -525,7 +529,7 @@ from this source are not detected.</b> Check the connection and credentials in
|
||||
Settings, then run the sync again from Scan Jobs.</p>
|
||||
<p><a href="{html.escape(base)}/scans">Open Scan Jobs</a></p>
|
||||
<p style="color:#6b7280;font-size:12px">You receive at most one mail per source
|
||||
per 24 hours while it keeps failing.</p>
|
||||
per day while it keeps failing.</p>
|
||||
</body></html>"""
|
||||
|
||||
for user_id, email, _name in recipients:
|
||||
|
||||
@@ -183,6 +183,15 @@ def test_cooldown_suppresses_repeat_mail_for_same_source():
|
||||
assert svc.in_cooldown(None, now) is False
|
||||
|
||||
|
||||
def test_cooldown_lets_a_nightly_job_alert_every_night():
|
||||
# alerted_at is stamped after the send; the next night's failure at the
|
||||
# same minute is just under 24h later and must still mail.
|
||||
now = datetime.now()
|
||||
yesterday = SyncRun(source="intune", status="failed", started_at=now,
|
||||
alerted_at=now - timedelta(hours=24) + timedelta(seconds=1))
|
||||
assert svc.in_cooldown(yesterday, now) is False
|
||||
|
||||
|
||||
def test_health_flags_failed_and_stale_sources():
|
||||
now = datetime.now()
|
||||
ok = SyncRun(source="wazuh", status="completed", started_at=now,
|
||||
|
||||
Reference in New Issue
Block a user