How it works

checking…
U
user #0 · admin

How the prognosis service works

A six-stage pipeline that turns raw IoT telemetry into anomaly events, forecasts and drift alerts. All stages run on a scheduler; every one can also be triggered manually here.

1 · Data flow

Only assets explicitly enrolled from Management → Assets flow through this pipeline. New assets default to inactive after each registry sync.

What happens per minute

  1. derive reads new rows from master history for enrolled assets and writes 1-minute aggregates into derived_series. The kind of aggregate depends on the series classification (smooth → median; event-driven → baseline + spike-count/peak/area; sparse → median).
  2. score compares each new bucket against the latest forecast — score is (actual − q50) / max(q90 − q10, 1e-9). ≥ SCORE_ANOMALOUS_THRESHOLD = anomalous, ≥ SCORE_SUSPICIOUS_THRESHOLD = suspicious, else normal.
  3. If a series stays anomalous for ANOMALY_DEBOUNCE_BUCKETS consecutive minutes, a new event opens. Maintenance mode on the asset suppresses opening but not scoring.

What happens every 15 minutes

  • forecast regenerates a 24-hour quantile (q10, q50, q90) forecast for every active series. The previous forecast is superseded so charts always show the freshest one.

What happens every 5 minutes

  • drift computes rolling MAPE / coverage per series, writes forecast_health, logs a warning when >50 % of series drift simultaneously (usually upstream — clock skew or sensor outage).

2 · Scheduler

Single-process APScheduler. Jobstore lives in the prognosis DB so the worker can be split into its own container later without code changes.

sync_master_caches
Snapshot device + parameter metadata from master into local caches (labels in the UI come from here).
cron · daily @ 02:00 IST
profile_series
Classify every (asset, param) into smooth / event-driven / digital / sparse based on the last 30 d.
cron · Sun @ 03:00 IST
build_derived
Read new rows from master history and write 1-minute buckets into derived_series. Advances the watermark in pipeline_state.
interval · every 1 min
run_forecasts
Quantile forecast for every active series, horizon = 24 h. Uses the backend selected in .env.
interval · every 15 min
score_and_debounce
Score new buckets, debounce ≥ N consecutive anomalous buckets into events, close stale events.
interval · every 1 min
watch_drift
Roll up MAPE / coverage / drift score into forecast_health; log warnings on widespread drift.
interval · every 5 min

Scheduler control

loading…

Live state of APScheduler. Pause/resume affects only the running process — restarting the app brings every job back from register_jobs.

Job Trigger Next run Status Actions
loading…

4 · Live job state (manual triggers)

Auto-refreshes every 3 s while jobs are running. This is GET /api/admin/jobs.

JobStatusDurationStartedFinishedResult / error
no manual jobs run yet — click Run now above.

5 · On-demand forecast

Forecasts run automatically every 15 minutes for every active series. To force one immediately — for the entire asset fleet — click the button below. To regenerate a forecast for a single series, open the Dashboard, pick the parameter, and click Run forecast now.

Open dashboard →

6 · Where to tune things

  • FORECAST_BACKEND — defaults to timesfm (real model, ~2 GB checkpoint). Under-warmed series (fewer than FORECAST_MIN_CONTEXT samples) fall back to naive automatically. Set the env var to naive to disable TimesFM entirely.
  • SCORE_SUSPICIOUS_THRESHOLD / SCORE_ANOMALOUS_THRESHOLD — z-score gates. Lower = more sensitive.
  • ANOMALY_DEBOUNCE_BUCKETS — how many consecutive anomalous minutes open an event. 2–3 is sensible.
  • MAPE_DRIFT_THRESHOLD — when a series's rolling MAPE exceeds this, the drift score climbs.
  • Threshold per series — manual overrides via POST /api/threshold/... are preserved across re-profiling (auto-thresholds use source = 'auto' in ON CONFLICT).

How it works — how to read this page

What this page is

The "How it works" page is a guided tour of the whole prognosis pipeline — what each stage does, in what order, and how often. Use it as a one-page primer when joining the project, or to remember the cadence of a particular background job.

How the system fits together (TL;DR)

  1. Sync — every night, copy the asset registry from the central database so we know which assets exist.
  2. Profile — every Sunday, study each signal and classify it (smooth / event-driven / sparse / digital).
  3. Derive — every minute, condense the raw readings into one-minute summaries.
  4. Forecast — every 15 minutes, predict the next 24 hours for each signal.
  5. Score & debounce — every minute, compare actuals to predictions; open events for sustained anomalies.
  6. Drift — every 5 minutes, score how well the predictions are tracking reality.

How to use this page

You don't have to read every line — the diagram up top is usually enough. The "Trigger now" buttons in the schedule table let admins kick off any job on-demand, useful when investigating an incident or seeding a fresh deployment.