# Predictive Model TODO

Priority key: `P0` = critical/blocking, `P1` = important, `P2` = later optimization.

## 1) Scope and Success Criteria
- [x] [P0] Lock v1 target: predict `rain_next_1h >= 0.2mm`.
- [x] [P0] Define the decision use-case (alerts vs dashboard signal).
- [x] [P0] Set acceptance metrics and thresholds (precision, recall, ROC-AUC).
- [x] [P0] Freeze training window with explicit UTC start/end timestamps.

## 2) Data Quality and Label Validation
- [x] [P0] Audit `observations_ws90` and `observations_baro` for missingness, gaps, duplicates, and out-of-order rows. (completed on runtime machine)
- [x] [P0] Validate rain label construction from `rain_mm` (counter resets, negative deltas, spikes). (completed on runtime machine)
- [x] [P0] Measure class balance by week (rain-positive vs rain-negative). (completed on runtime machine)
- [x] [P1] Document known data issues and mitigation rules. (see `docs/rain_data_issues.md`)

## 3) Dataset and Feature Engineering
- [x] [P1] Extract reusable dataset-builder logic from training script into a maintainable module/workflow.
- [x] [P1] Add lag/rolling features (means, stddev, deltas) for core sensor inputs.
- [x] [P1] Encode wind direction properly (cyclical encoding).
- [x] [P2] Add calendar features (hour-of-day, day-of-week, seasonality proxies). (`feature-set=extended_calendar`)
- [x] [P1] Join aligned forecast features from `forecast_openmeteo_hourly` (precip prob, cloud cover, wind, pressure).
- [x] [P1] Persist versioned dataset snapshots for reproducibility.

## 4) Modeling and Validation
- [x] [P0] Keep logistic regression as baseline.
- [x] [P1] Add at least one tree-based baseline (e.g. gradient boosting). (implemented via `hist_gb`; runtime evaluation pending local Python deps)
- [x] [P0] Use strict time-based train/validation/test splits (no random shuffling).
- [x] [P1] Add walk-forward backtesting across multiple temporal folds. (`train_rain_model.py --walk-forward-folds`)
- [x] [P1] Tune hyperparameters on validation data only. (`train_rain_model.py --tune-hyperparameters`)
- [x] [P1] Calibrate probabilities (Platt or isotonic) and compare calibration quality. (`--calibration-methods`)
- [x] [P0] Choose and lock the operating threshold based on use-case costs.

## 5) Evaluation and Reporting
- [x] [P0] Report ROC-AUC, PR-AUC, confusion matrix, precision, recall, and Brier score.
- [x] [P1] Compare against naive baselines (persistence and simple forecast-threshold rules).
- [x] [P2] Slice performance by periods/weather regimes (day/night, rainy weeks, etc.). (`sliced_performance_test`)
- [x] [P1] Produce a short model card (data window, features, metrics, known limitations). (`--model-card-out`)

## 6) Packaging and Deployment
- [x] [P1] Version model artifacts and feature schema together.
- [x] [P0] Implement inference path with feature parity between training and serving.
- [x] [P0] Add prediction storage table for predicted probabilities and realized outcomes.
- [x] [P1] Expose predictions via API and optionally surface in web dashboard.
- [x] [P2] Add scheduled retraining with rollback to last-known-good model. (`run_rain_ml_worker.py` candidate promote + `RAIN_MODEL_BACKUP_PATH`)

## 7) Monitoring and Operations
- [x] [P1] Track feature drift and prediction drift over time. (view: `rain_feature_drift_daily`, `rain_prediction_drift_daily`)
- [x] [P1] Track calibration drift and realized performance after deployment. (view: `rain_calibration_drift_daily`)
- [x] [P1] Add alerts for training/inference/data pipeline failures. (`scripts/check_rain_pipeline_health.py`)
- [x] [P1] Document runbook for train/evaluate/deploy/rollback. (see `docs/rain_model_runbook.md`)

## 8) Immediate Next Steps (This Week)
- [x] [P0] Run first full data audit and label-quality checks. (completed on runtime machine)
- [x] [P0] Train baseline model on full available history and capture metrics. (completed on runtime machine)
- [x] [P1] Add one expanded feature set and rerun evaluation. (completed on runtime machine 2026-03-12 with `feature_set=extended`, `model_version=rain-auto-v1-extended-202603120932`)
- [x] [P0] Decide v1 threshold and define deployment interface.