update for 4 hour rain forecast

2026-04-06 18:32:33 +10:00
parent fb50c8ed71
commit 3a7309b2cf
20 changed files with 716 additions and 132 deletions
@@ -4,6 +4,14 @@ Operational guide for training, evaluating, deploying, monitoring, and rolling b

 ## 1) One-time Setup

+Apply 4-hour prediction table migration:
+
+```sh
+docker compose exec -T timescaledb \
+  psql -U postgres -d micrometeo \
+  -f /docker-entrypoint-initdb.d/003_rain_predictions_4h.sql
+```
+
 Apply monitoring views:

 ```sh
@@ -21,6 +29,7 @@ python scripts/train_rain_model.py \
  --site "home" \
  --start "2026-02-01T00:00:00Z" \
  --end "2026-03-03T23:55:00Z" \
+  --horizon-hours 4 \
  --feature-set "extended" \
  --model-family "auto" \
  --forecast-model "ecmwf" \
@@ -29,7 +38,7 @@ python scripts/train_rain_model.py \
  --calibration-methods "none,sigmoid,isotonic" \
  --threshold-policy "walk_forward" \
  --walk-forward-folds 4 \
-  --model-version "rain-auto-v1-extended" \
+  --model-version "rain-auto-v2-extended-4h" \
  --out "models/rain_model.pkl" \
  --report-out "models/rain_model_report.json" \
  --model-card-out "models/model_card_{model_version}.md" \
@@ -53,7 +62,8 @@ Review in report:
 python scripts/predict_rain_model.py \
  --site home \
  --model-path "models/rain_model.pkl" \
-  --model-name "rain_next_1h" \
+  --model-name "rain_next_4h" \
+  --horizon-hours 4 \
  --dry-run
 ```

@@ -63,7 +73,8 @@ python scripts/predict_rain_model.py \
 python scripts/predict_rain_model.py \
  --site home \
  --model-path "models/rain_model.pkl" \
-  --model-name "rain_next_1h"
+  --model-name "rain_next_4h" \
+  --horizon-hours 4
 ```

 ## 4) Rollback
@@ -72,6 +83,7 @@ python scripts/predict_rain_model.py \
 2. If promotion fails or no candidate model is produced, the worker keeps the active model unchanged.
 3. If inference starts without `RAIN_MODEL_PATH` but backup exists, the worker restores from backup automatically.
 4. Keep failed candidate artifacts for postmortem.
+5. During 4-hour rollout stabilization, keep `predictions_rain_1h` and `rain_next_1h` model artifacts available for immediate fallback.

 ## 5) Monitoring

@@ -118,12 +130,13 @@ Use the health-check script in cron, systemd timer, or your alerting scheduler:
 ```sh
 python scripts/check_rain_pipeline_health.py \
  --site home \
-  --model-name rain_next_1h \
+  --model-name rain_next_4h \
+  --horizon-hours 4 \
  --max-ws90-age 20m \
  --max-baro-age 30m \
  --max-forecast-age 3h \
  --max-prediction-age 30m \
-  --max-pending-eval-age 3h \
+  --max-pending-eval-age 6h \
  --max-pending-eval-rows 200
 ```

@@ -138,6 +151,7 @@ The script exits non-zero on failure, so it can directly drive alerting.
 - `RAIN_THRESHOLD_POLICY`
 - `RAIN_WALK_FORWARD_FOLDS`
 - `RAIN_ALLOW_EMPTY_DATA`
+- `RAIN_HORIZON_HOURS`
 - `RAIN_MODEL_BACKUP_PATH`
 - `RAIN_MODEL_CARD_PATH`

@@ -156,3 +170,88 @@ python scripts/recommend_rain_model.py \
  --top-k 5 \
  --json-out "models/rain_model_recommendation.json"
 ```
+
+## 9) Staged 4h Rollout Checklist
+
+Run this sequence in production/staging to satisfy the 4h cutover gate:
+
+1. Apply schema migration for 4h predictions:
+
+```sh
+docker compose exec -T timescaledb \
+  psql -U postgres -d micrometeo \
+  -f /docker-entrypoint-initdb.d/003_rain_predictions_4h.sql
+```
+
+2. Re-apply monitoring views (now include 1h + 4h unions):
+
+```sh
+docker compose exec -T timescaledb \
+  psql -U postgres -d micrometeo \
+  -f /docker-entrypoint-initdb.d/002_rain_monitoring_views.sql
+```
+
+3. Run a full 4h training/evaluation cycle and save report:
+
+```sh
+python scripts/train_rain_model.py \
+  --site "home" \
+  --start "2026-02-01T00:00:00Z" \
+  --end "2026-03-03T23:55:00Z" \
+  --horizon-hours 4 \
+  --feature-set "extended" \
+  --model-family "auto" \
+  --forecast-model "ecmwf" \
+  --tune-hyperparameters \
+  --threshold-policy "walk_forward" \
+  --walk-forward-folds 4 \
+  --model-version "rain-auto-v2-extended-4h" \
+  --out "models/rain_model_4h.pkl" \
+  --report-out "models/rain_model_report_4h.json"
+```
+
+4. Compare 4h metrics against the latest 1h benchmark report before switching dashboard defaults:
+
+```sh
+python scripts/compare_rain_reports.py \
+  --baseline "models/rain_model_report_1h.json" \
+  --candidate "models/rain_model_report_4h.json"
+```
+5. Run dry-run inference, then live inference with 4h model name/horizon:
+
+```sh
+python scripts/predict_rain_model.py \
+  --site home \
+  --model-path "models/rain_model_4h.pkl" \
+  --model-name "rain_next_4h" \
+  --horizon-hours 4 \
+  --dry-run
+
+python scripts/predict_rain_model.py \
+  --site home \
+  --model-path "models/rain_model_4h.pkl" \
+  --model-name "rain_next_4h" \
+  --horizon-hours 4
+```
+
+6. Validate health checks and dashboard data path for 4h:
+
+```sh
+python scripts/check_rain_pipeline_health.py \
+  --site home \
+  --model-name rain_next_4h \
+  --horizon-hours 4 \
+  --max-pending-eval-age 6h
+```
+
+7. Keep 1h path live in parallel until 4h drift/calibration remains stable for at least 7 days.
+
+### Fast rollback to 1h
+
+If 4h performance or pipeline health regresses:
+
+1. Set worker env back to:
+   `RAIN_HORIZON_HOURS=1`, `RAIN_MODEL_NAME=rain_next_1h`, and a known-good 1h model path/version.
+2. Restart `rainml` service.
+3. Confirm `check_rain_pipeline_health.py --horizon-hours 1 --model-name rain_next_1h` returns `ok`.
+4. Keep `predictions_rain_4h` data for postmortem; do not drop tables during rollback.
@@ -1,14 +1,14 @@
-# Rain Prediction (Next 1 Hour)
+# Rain Prediction (Next 4 Hours)

 This project includes a baseline workflow for **binary rain prediction**:

-> **Will we see >= 0.2 mm of rain in the next hour?**
+> **Will we see >= 0.2 mm of rain in the next 4 hours?**

 It uses local observations (WS90 + barometer), trains a logistic regression
 baseline, and writes model-driven predictions back to TimescaleDB.

 ## P0 Decisions (Locked)
- Target: `rain_next_1h_mm >= 0.2`.
+- Target: `rain_next_4h_mm >= 0.2`.
 - Primary use-case: low-noise rain heads-up signal for dashboard + alert candidate.
 - Frozen v1 training window (UTC): `2026-02-01T00:00:00Z` to `2026-03-03T23:55:00Z`.
 - Threshold policy: choose threshold on validation set by maximizing recall under
@@ -40,7 +40,7 @@ pip install -r scripts/requirements.txt
 - `scripts/train_rain_model.py`: strict time-based split training and metrics report, with optional
  validation-only hyperparameter tuning, calibration comparison, naive baseline comparison, and walk-forward folds.
 - `scripts/predict_rain_model.py`: inference using saved model artifact; upserts into
-  `predictions_rain_1h`.
+  `predictions_rain_4h`.
 - `scripts/run_rain_ml_worker.py`: long-running worker for periodic training + prediction.
 - `scripts/check_rain_pipeline_health.py`: freshness/failure check for alerting.
 - `scripts/recommend_rain_model.py`: rank saved training reports and recommend a deployment candidate.
@@ -60,7 +60,15 @@ Model-family options (`train_rain_model.py`):

 ## Usage
 ### 1) Apply schema update (existing DBs)
-`001_schema.sql` includes `predictions_rain_1h`.
+`003_rain_predictions_4h.sql` adds `predictions_rain_4h`.
+
+```sh
+docker compose exec -T timescaledb \
+  psql -U postgres -d micrometeo \
+  -f /docker-entrypoint-initdb.d/003_rain_predictions_4h.sql
+```
+
+`001_schema.sql` still remains safe to re-run for full schema parity.

 ```sh
 docker compose exec -T timescaledb \
@@ -76,6 +84,8 @@ docker compose exec -T timescaledb \
  -f /docker-entrypoint-initdb.d/002_rain_monitoring_views.sql
 ```

+All examples below assume a 4-hour horizon (`--horizon-hours 4`) and `model-name=rain_next_4h`.
+
 ### 2) Run data audit
 ```sh
 export DATABASE_URL="postgres://postgres:postgres@localhost:5432/micrometeo?sslmode=disable"
@@ -203,7 +213,8 @@ python scripts/train_rain_model.py \
 python scripts/predict_rain_model.py \
  --site home \
  --model-path "models/rain_model.pkl" \
-  --model-name "rain_next_1h"
+  --model-name "rain_next_4h" \
+  --horizon-hours 4
 ```

 ### 5) One-command P0 workflow
@@ -236,7 +247,7 @@ docker compose logs -f rainml
 - Model card: `models/model_card_<model_version>.md`
 - Model artifact: `models/rain_model.pkl`
 - Dataset snapshot: `models/datasets/rain_dataset_<model_version>_<feature_set>.csv`
- Prediction rows: `predictions_rain_1h` (probability + threshold decision + realized
+- Prediction rows: `predictions_rain_4h` (probability + threshold decision + realized
  outcome fields once available)

 ### 7) Recommend deploy candidate from saved reports