feat(app-scan): the Data-jobs scan rebuilds the CVE indexes first

To get a CVE published today, the operator had to rebuild the cvelistV5
index on the host (docker compose exec ... build_product_index(force_fresh=
True)) before pressing the scan button, or wait for the 01:30 refresh. The
"App CVE Scan" button under Data jobs now does that itself: it calls
/app-cve-scan/start?rebuild_index=true, and the job runs the full nightly
refresh (fresh cvelistV5 ZIP + MFSA, GitHub advisories, TeamViewer, Citrix,
IGEL ISNs, MSRC) before it scans. The 01:30 job body moved into
app_cve_scanner_service.refresh_indexes so both run the same code.

Opt-in: the per-asset rescan and API clients keep reading the stored index,
no 557 MB download per click. The button's dialog says what it costs.

A failed rebuild keeps the cached index as before, but the button now says
so ("index rebuilds failed — scanned on the cached index"). That includes a
ZIP download that fell back to the old file on disk; otherwise "completed"
reads as "today's CVEs are in".

The button can now overlap the 01:30 job and the threat-intel cascade, and
all three streamed the ZIP into the same .part file. Each writer now uses
its own temp name; os.replace keeps the swap atomic.

README.DEV.md

@@ -1318,8 +1318,12 @@ normal EPSS/KEV/CVSS enrichment and multi-source cross-confirm apply.
 
 The cvelistV5 path needs a **reverse index** (`{vendor,product} → CVE
 ranges`) built by walking the ~557 MB cvelistV5 ZIP once; cached in a
-Setting, rebuilt by the nightly job. A manual "App CVE Scan" run builds it
-on demand if missing (slower on first run, then cached).
+Setting, rebuilt at 01:30 by `vuln_index_refresh_nightly`
+(`app_cve_scanner_service.refresh_indexes`). The "App CVE Scan" button under
+Data jobs runs that same refresh first (`/app-cve-scan/start?rebuild_index=true`:
+fresh ZIP, vendor indexes, MSRC), so today's CVEs are found without waiting
+for the night or a shell on the host. Every other scan (single asset, API)
+reads the stored index and builds it only if missing.
 
 **False-positive suppression** (same cvelistV5 data, inverse direction):
 Wazuh's own CPE matching sometimes over-reports across product editions

app/routers/vulnerabilities.py

@@ -3253,22 +3253,31 @@ def run_m365_check_endpoint(
 
 
 # Background app-scan job — same reason as the EOL check: a full run walks
-# every asset's inventory against NVD/OSV and rebuilds the cvelistV5 and MSRC
-# indexes on first use, which outlives the browser's HTTP patience. Operators
+# every asset's inventory against NVD/OSV (and, from the Data-jobs button,
+# rebuilds every CVE index first), which outlives the browser's HTTP patience. Operators
 # saw "Backend connection failed" while the scan ran happily to completion.
 _APPSCAN_JOB: dict = {"running": False, "stage": None, "done": 0, "total": 0,
                       "result": None, "error": None, "finished_at": None}
 
 
-def _appscan_job_runner(asset_id: Optional[int]):
+def _appscan_job_runner(asset_id: Optional[int], rebuild_index: bool):
     """Run the app CVE scan on its own DB session, recording progress."""
     import time as _time
     from app.database import SessionLocal
     from app.services import app_cve_scanner_service
     db = SessionLocal()
     try:
+        if rebuild_index:
+            _APPSCAN_JOB["stage"] = ("rebuilding the CVE indexes (fresh cvelistV5 "
+                                     "ZIP, vendor advisories, MSRC)…")
+            failed = app_cve_scanner_service.refresh_indexes(db)
+        _APPSCAN_JOB["stage"] = "scanning…"
         stats = app_cve_scanner_service.run_app_cve_scan(db, asset_id=asset_id)
         _APPSCAN_JOB["result"] = {k: v for k, v in (stats or {}).items() if k != "errors"}
+        if rebuild_index and failed:
+            # The scan still ran, on the cached index — say so, or "completed"
+            # reads as "today's CVEs are in".
+            _APPSCAN_JOB["result"]["index_rebuilds_failed"] = len(failed)
         _APPSCAN_JOB["error"] = None
         _APPSCAN_JOB["stage"] = "completed"
     except Exception as e:
@@ -3285,16 +3294,22 @@ def _appscan_job_runner(asset_id: Optional[int]):
 @router.post("/app-cve-scan/start", status_code=202)
 def start_app_cve_scan_job(
     asset_id: Optional[int] = Query(None, description="Scope to one asset; omit for all"),
+    rebuild_index: bool = Query(False, description="Rebuild the CVE indexes first, as the 01:30 job does (downloads the ~557 MB cvelistV5 ZIP)"),
     current_user: User = Depends(RequireEditor),
 ):
-    """Start the app CVE scan in the background; poll /app-cve-scan/status."""
+    """Start the app CVE scan in the background; poll /app-cve-scan/status.
+
+    `rebuild_index` is what the Data-jobs button sends: the operator wants
+    today's CVEs now, not after 01:30. Opt-in, so the per-asset rescan and
+    API clients keep deciding from the stored index without a download.
+    """
     import threading
     if _APPSCAN_JOB["running"]:
         return {"started": False, "message": "An app CVE scan is already running",
                 "poll_url": "/api/v1/vulnerabilities/app-cve-scan/status"}
     _APPSCAN_JOB.update({"running": True, "stage": "starting", "done": 0, "total": 0,
                          "result": None, "error": None, "finished_at": None})
-    threading.Thread(target=_appscan_job_runner, args=(asset_id,),
+    threading.Thread(target=_appscan_job_runner, args=(asset_id, rebuild_index),
                      daemon=True).start()
     return {"started": True, "poll_url": "/api/v1/vulnerabilities/app-cve-scan/status"}
 

app/scheduler.py

@@ -858,48 +858,11 @@ def vuln_index_refresh_nightly():
     widening the gap.
     """
     import time as _time
-    from app.services import (cvelistv5_scan_service, github_repo_advisory_service,
-                              igel_isn_service, teamviewer_bulletin_service)
+    from app.services import app_cve_scanner_service
     db = SessionLocal()
     t0 = _time.monotonic()
     try:
-        # force_fresh: the /tmp ZIP is shared with the threat-intel refresh,
-        # which runs on an interval anchored to app startup — under the plain
-        # 12h TTL this build kept reusing an afternoon snapshot and missed the
-        # evening's CVEs for a full extra night (Chrome 151.0.7922.169).
-        try:
-            cvelistv5_scan_service.build_product_index(db, force_fresh=True)
-        except Exception as e:
-            logger.warning("cvelistV5 index build failed (non-fatal, cache kept): %s", e)
-        # Vendor indexes: all cache 24h and used to be refreshed by whoever
-        # asked first — the scan itself, one build-duration short of the TTL
-        # every night, so they only ever cleared it every OTHER night (the
-        # MFSA bug, 978d4f3). Notepad++, Wazuh and the IGEL ISNs without a
-        # CVE reach no other source at all; TeamViewer publishes days before
-        # NVD — a night late is a night blind.
-        from app.services import citrix_workspace_service
-        for _mod, _label in ((github_repo_advisory_service, "repo-advisory"),
-                             (teamviewer_bulletin_service, "teamviewer-bulletin"),
-                             (citrix_workspace_service, "citrix-build-catalog"),
-                             (igel_isn_service, "igel-isn")):
-            try:
-                _mod.build_index(db)
-            except Exception as e:
-                logger.warning("%s index build failed (non-fatal, cache kept): %s",
-                               _label, e)
-        # MSRC fixed builds — the one index b802f3a left where it was. The
-        # 03:20 app scan's package pass creates the Edge findings from it, and
-        # it was rebuilt by the MSRC job at 03:50, i.e. AFTER that scan, so a
-        # Chromium CVE MSRC filed under Edge on day D reached the host on D+2
-        # (CVE-2026-84324: published 03.09. 17:00, "app-scan only" on 04.09.).
-        # Edge CVEs exist in no other source, so a night late is a night blind.
-        # The 03:50 job reuses this build (ensure_index) instead of pulling
-        # the 18 documents again; a failed build keeps the cached index.
-        try:
-            from app.services import msrc_scan_service
-            msrc_scan_service.build_product_index(db)
-        except Exception as e:
-            logger.warning("MSRC index build failed (non-fatal, cache kept): %s", e)
+        app_cve_scanner_service.refresh_indexes(db)
         logger.info("Vulnerability index refresh done in %.0fs — the 02:10 "
                     "Intune/Defender sync reads these", _time.monotonic() - t0)
     except Exception as e:

app/services/app_cve_scanner_service.py

@@ -1901,6 +1901,68 @@ def _resolve_stale_app_findings(db: Session, asset, touched_cves: set) -> int:
     return resolved
 
 
+def refresh_indexes(db: Session) -> list:
+    """Rebuild every CVE index the scans decide from: cvelistV5 from a fresh
+    ZIP (+MFSA), GitHub advisories, TeamViewer bulletins, Citrix builds, IGEL
+    ISNs, MSRC fixed builds.
+
+    Run by the 01:30 nightly job (scheduler.vuln_index_refresh_nightly) and by
+    the GUI's Data-jobs app scan, so a CVE published today is found today
+    without a shell on the host. Each rebuild is independent and a failed one
+    keeps the cached index, so an offline vendor site costs that source a
+    run, never the others. Returns the labels of the failed rebuilds, so the
+    GUI can say the scan ran on a cached index."""
+    failed = []
+    from app.services import (cvelistv5_scan_service, github_repo_advisory_service,
+                              igel_isn_service, teamviewer_bulletin_service)
+    # force_fresh: the /tmp ZIP is shared with the threat-intel refresh,
+    # which runs on an interval anchored to app startup — under the plain
+    # 12h TTL this build kept reusing an afternoon snapshot and missed the
+    # evening's CVEs for a full extra night (Chrome 151.0.7922.169).
+    try:
+        t0 = time.time()
+        cvelistv5_scan_service.build_product_index(db, force_fresh=True)
+        # A failed download falls back to the on-disk ZIP on purpose (see
+        # _ensure_zip) — which is exactly what the operator must hear about.
+        zip_path = cvelistv5_scan_service._ZIP_PATH
+        if not os.path.exists(zip_path) or os.path.getmtime(zip_path) < t0:
+            failed.append("cvelistV5")
+    except Exception as e:
+        failed.append("cvelistV5")
+        logger.warning("cvelistV5 index build failed (non-fatal, cache kept): %s", e)
+    # Vendor indexes: all cache 24h and used to be refreshed by whoever
+    # asked first — the scan itself, one build-duration short of the TTL
+    # every night, so they only ever cleared it every OTHER night (the
+    # MFSA bug, 978d4f3). Notepad++, Wazuh and the IGEL ISNs without a
+    # CVE reach no other source at all; TeamViewer publishes days before
+    # NVD — a night late is a night blind.
+    from app.services import citrix_workspace_service
+    for _mod, _label in ((github_repo_advisory_service, "repo-advisory"),
+                         (teamviewer_bulletin_service, "teamviewer-bulletin"),
+                         (citrix_workspace_service, "citrix-build-catalog"),
+                         (igel_isn_service, "igel-isn")):
+        try:
+            _mod.build_index(db)
+        except Exception as e:
+            failed.append(_label)
+            logger.warning("%s index build failed (non-fatal, cache kept): %s",
+                           _label, e)
+    # MSRC fixed builds. The app scan's package pass creates the Edge
+    # findings from it, and it was rebuilt by the MSRC job at 03:50, i.e.
+    # AFTER the 03:20 scan, so a Chromium CVE MSRC filed under Edge on day D
+    # reached the host on D+2 (CVE-2026-84324: published 03.09. 17:00,
+    # "app-scan only" on 04.09.). Edge CVEs exist in no other source, so a
+    # night late is a night blind. The 03:50 job reuses this build
+    # (ensure_index) instead of pulling the 18 documents again.
+    try:
+        from app.services import msrc_scan_service
+        msrc_scan_service.build_product_index(db)
+    except Exception as e:
+        failed.append("msrc")
+        logger.warning("MSRC index build failed (non-fatal, cache kept): %s", e)
+    return failed
+
+
 def run_app_cve_scan(db: Session, asset_id: Optional[int] = None,
                      trigger: str = "manual") -> dict:
     """Scan all assets with software inventory (Wazuh packages + Intune

app/services/cvelistv5_scan_service.py

@@ -27,6 +27,7 @@ import json
 import logging
 import os
 import re
+import threading
 import time
 import zipfile
 from datetime import datetime, timedelta
@@ -1183,7 +1184,8 @@ def _ensure_zip(force: bool = False) -> bool:
     started in the late afternoon the ZIP is only ~8h old at 01:30 and the
     TTL kept the build on a snapshot taken hours before the day's CVEs were
     published (Chrome 151.0.7922.169 surfaced >24h late that way). On-demand
-    callers keep the cache so the GUI never triggers a 557 MB download.
+    callers keep the cache; the one GUI path that forces a download is the
+    Data-jobs app scan, whose dialog says so (refresh_indexes).
     """
     on_disk = (os.path.exists(_ZIP_PATH)
                and os.path.getsize(_ZIP_PATH) > 100_000_000)
@@ -1192,7 +1194,10 @@ def _ensure_zip(force: bool = False) -> bool:
     if fresh and not force:
         return True
     import httpx
-    tmp = _ZIP_PATH + ".part"
+    # Per-writer temp name: the nightly refresh, the GUI app scan and the
+    # threat-intel cascade (vuln_override_service) all download this file and
+    # can overlap; a shared .part interleaved their bytes into a "fresh" ZIP.
+    tmp = f"{_ZIP_PATH}.{os.getpid()}.{threading.get_ident()}.part"
     try:
         logger.info("cvelistv5-scan: downloading ZIP snapshot (~557 MB)…")
         with httpx.Client(timeout=httpx.Timeout(600.0, connect=15.0), follow_redirects=True) as c:

app/services/vuln_override_service.py

@@ -942,6 +942,7 @@ class VulnOverrideService:
         import httpx
         import os
         import re
+        import threading
         import time
         import zipfile
 
@@ -969,7 +970,8 @@ class VulnOverrideService:
                 "cvelistV5: downloading fresh ZIP (cache miss / stale) to %s",
                 cache_path,
             )
-            tmp_path = cache_path + ".part"
+            # Per-writer name — see cvelistv5_scan_service._ensure_zip.
+            tmp_path = f"{cache_path}.{os.getpid()}.{threading.get_ident()}.part"
             try:
                 with httpx.Client(timeout=httpx.Timeout(120.0, connect=15.0),
                                   follow_redirects=True) as client:

frontend/app/vulnerabilities/page.tsx

@@ -271,8 +271,12 @@ function AppCveScanButton({ onDone }: { onDone: () => void }) {
                     'curated product list, low false-positives.\n\n' +
                     'Findings carry source "app-scan" (MSRC ones "msrc"), ' +
                     'cross-confirm with other scanners, and auto-resolve once ' +
-                    'the software is updated. Runs in the background; the first ' +
-                    'run takes a while (index downloads).'
+                    'the software is updated.\n\n' +
+                    'First rebuilds the CVE indexes the same way the 01:30 ' +
+                    'nightly does (fresh cvelistV5 download, ~557 MB, plus the ' +
+                    'vendor advisories and MSRC), so CVEs published today are ' +
+                    'found today. Runs in the background and takes a few ' +
+                    'minutes longer for that.'
                 )) return;
                 setBusy(true);
                 setResult(null);
@@ -281,7 +285,7 @@ function AppCveScanButton({ onDone }: { onDone: () => void }) {
                     // scan (plus first-run index builds) outlives the browser's
                     // HTTP patience and surfaced as "Backend connection failed"
                     // while the backend ran happily to the end.
-                    const start = await api.post('/api/v1/vulnerabilities/app-cve-scan/start');
+                    const start = await api.post('/api/v1/vulnerabilities/app-cve-scan/start?rebuild_index=true');
                     if (start.data?.started === false) setResult('Already running — attaching…');
                     // Counts consecutive polls that gave no usable answer, so a
                     // backend restart mid-scan cannot leave the button spinning

frontend/lib/scanStats.ts

@@ -27,6 +27,7 @@ const LABELS: Record<string, string> = {
     repo_advisory_findings: 'via repository advisories',
     notifications: 'notifications sent',
     resolve_skipped_inactive: 'assets skipped (not active)',
+    index_rebuilds_failed: 'index rebuilds failed — scanned on the cached index, see backend log',
     checked: 'checked',
     suppressed: 'suppressed',
     // EOL check

tests/test_appscan_gui_index_refresh.py

@@ -0,0 +1,109 @@
+"""The Data-jobs App CVE scan refreshes the CVE indexes before it scans.
+
+Run: python tests/test_appscan_gui_index_refresh.py
+
+A CVE published during the day reached the GUI scan only after the 01:30
+refresh: the scan reads the stored cvelistV5 index at any age, so getting it
+the same day meant a `docker compose exec … build_product_index(force_fresh=
+True)` on the host first. The Data-jobs button now runs the 01:30 refresh
+itself (cvelistV5 ZIP fresh, MFSA, vendor indexes, MSRC), then scans.
+
+Opt-in: the per-asset rescan and API clients keep the cached index — a 557 MB
+download per click is the cost the button's dialog names.
+"""
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+os.environ.setdefault("JWT_SECRET_KEY", "test-only-not-a-secret-0000000000000000")
+
+import app.database as database                              # noqa: E402
+from app.routers import vulnerabilities as v                 # noqa: E402
+from app.services import app_cve_scanner_service as acs      # noqa: E402
+
+
+class _DB:
+    def close(self):
+        pass
+
+
+def _patch(obj, **attrs):
+    """Set attributes, return the restore callable."""
+    old = {k: getattr(obj, k) for k in attrs}
+    for k, val in attrs.items():
+        setattr(obj, k, val)
+    return lambda: [setattr(obj, k, val) for k, val in old.items()]
+
+
+def test_data_jobs_scan_refreshes_indexes():
+    events = []
+    restore = [
+        _patch(database, SessionLocal=_DB),
+        _patch(acs,
+               refresh_indexes=lambda d: events.append(
+                   f"refresh[{v._APPSCAN_JOB['stage']}]"),
+               run_app_cve_scan=lambda d, asset_id=None, **k:
+                   events.append(f"scan({asset_id})") or {"findings": 0}),
+    ]
+    try:
+        # Data-jobs button: indexes first, the operator sees why it takes long
+        v._appscan_job_runner(None, True)
+        assert len(events) == 2 and events[0].startswith("refresh[") \
+            and "index" in events[0] and events[1] == "scan(None)", events
+        assert v._APPSCAN_JOB["stage"] == "completed", v._APPSCAN_JOB
+
+        # Without the flag: cached index, no download
+        events.clear()
+        v._appscan_job_runner(7, False)
+        assert events == ["scan(7)"], events
+    finally:
+        [r() for r in restore]
+
+    # A failed rebuild must reach the button: the ZIP download falls back to
+    # the on-disk copy on purpose, and "completed" would read as "today's
+    # CVEs are in".
+    import tempfile
+    from app.services import (citrix_workspace_service, cvelistv5_scan_service as c5,
+                              github_repo_advisory_service, igel_isn_service,
+                              msrc_scan_service, teamviewer_bulletin_service)
+
+    def _msrc_down(d):
+        raise RuntimeError("msrc unreachable")
+
+    with tempfile.NamedTemporaryFile() as old_zip:
+        os.utime(old_zip.name, (0, 0))                  # download "failed"
+        restore = [
+            _patch(c5, _ZIP_PATH=old_zip.name,
+                   build_product_index=lambda d, force_fresh=False: {}),
+            _patch(msrc_scan_service, build_product_index=_msrc_down),
+        ] + [_patch(m, build_index=lambda d: {})
+             for m in (github_repo_advisory_service, teamviewer_bulletin_service,
+                       citrix_workspace_service, igel_isn_service)]
+        try:
+            assert acs.refresh_indexes(None) == ["cvelistV5", "msrc"]
+        finally:
+            [r() for r in restore]
+
+    restore = [
+        _patch(database, SessionLocal=_DB),
+        _patch(acs, refresh_indexes=lambda d: ["msrc"],
+               run_app_cve_scan=lambda d, **k: {"findings": 3}),
+    ]
+    try:
+        v._appscan_job_runner(None, True)
+        assert v._APPSCAN_JOB["result"] == {"findings": 3,
+                                            "index_rebuilds_failed": 1}, v._APPSCAN_JOB
+    finally:
+        [r() for r in restore]
+
+    # The start endpoint is opt-in — API clients keep the old behaviour
+    import inspect
+    default = inspect.signature(v.start_app_cve_scan_job).parameters["rebuild_index"].default
+    assert getattr(default, "default", default) is False, default
+
+    print("OK — the Data-jobs app scan rebuilds the indexes first; "
+          "everything else keeps the cache")
+
+
+if __name__ == "__main__":
+    test_data_jobs_scan_refreshes_indexes()

tests/test_nightly_index_order.py

@@ -27,8 +27,8 @@ Sources that decide CVE findings from such an index:
     or a day before it by itself.
 
 Fix, as with MFSA: the nightly path rebuilds before it decides; the TTL keeps
-guarding the on-demand callers (GUI scan, single asset), which must not pull
-vendor pages. A failed rebuild falls back to the cached index — an offline
+guarding the on-demand callers (single asset, API), which must not pull vendor
+pages. The GUI's Data-jobs app scan opts in to the same rebuild. A failed rebuild falls back to the cached index — an offline
 night stays no worse than before.
 
 EOL-shaped caches are deliberately NOT in here: endoflife.date (24h), the MS