# Rain Model Runbook Operational guide for training, evaluating, deploying, monitoring, and rolling back the rain model. ## 1) One-time Setup Apply 4-hour prediction table migration: ```sh docker compose exec -T timescaledb \ psql -U postgres -d micrometeo \ -f /docker-entrypoint-initdb.d/003_rain_predictions_4h.sql ``` Apply monitoring views: ```sh docker compose exec -T timescaledb \ psql -U postgres -d micrometeo \ -f /docker-entrypoint-initdb.d/002_rain_monitoring_views.sql ``` ## 2) Train + Evaluate Recommended evaluation run (includes validation-only tuning, calibration comparison, naive baselines, and walk-forward folds): ```sh python scripts/train_rain_model.py \ --site "home" \ --start "2026-02-01T00:00:00Z" \ --end "2026-03-03T23:55:00Z" \ --horizon-hours 4 \ --feature-set "extended" \ --model-family "auto" \ --forecast-model "ecmwf" \ --tune-hyperparameters \ --max-hyperparam-trials 12 \ --calibration-methods "none,sigmoid,isotonic" \ --threshold-policy "walk_forward" \ --walk-forward-folds 4 \ --model-version "rain-auto-v2-extended-4h" \ --out "models/rain_model.pkl" \ --report-out "models/rain_model_report.json" \ --model-card-out "models/model_card_{model_version}.md" \ --dataset-out "models/datasets/rain_dataset_{model_version}_{feature_set}.csv" ``` Review in report: - `candidate_models[*].hyperparameter_tuning` - `candidate_models[*].calibration_comparison` - `naive_baselines_test` - `sliced_performance_test` - `threshold_tuning_walk_forward` - `walk_forward_backtest` ## 3) Deploy 1. Promote the selected artifact path to the inference worker (`RAIN_MODEL_PATH` or CLI `--model-path`). 2. Run one dry-run inference: ```sh python scripts/predict_rain_model.py \ --site home \ --model-path "models/rain_model.pkl" \ --model-name "rain_next_4h" \ --horizon-hours 4 \ --dry-run ``` 3. Run live inference: ```sh python scripts/predict_rain_model.py \ --site home \ --model-path "models/rain_model.pkl" \ --model-name "rain_next_4h" \ --horizon-hours 4 ``` ## 4) Rollback 1. The worker now keeps a backup model at `RAIN_MODEL_BACKUP_PATH` and promotes new models only after candidate training succeeds. 2. If promotion fails or no candidate model is produced, the worker keeps the active model unchanged. 3. If inference starts without `RAIN_MODEL_PATH` but backup exists, the worker restores from backup automatically. 4. Keep failed candidate artifacts for postmortem. 5. During 4-hour rollout stabilization, keep `predictions_rain_1h` and `rain_next_1h` model artifacts available for immediate fallback. ## 5) Monitoring ### Feature drift ```sql SELECT * FROM rain_feature_drift_daily WHERE site = 'home' ORDER BY day DESC LIMIT 30; ``` Alert heuristic: any absolute z-score > 3 for 2+ consecutive days. ### Prediction drift ```sql SELECT * FROM rain_prediction_drift_daily WHERE site = 'home' ORDER BY day DESC LIMIT 30; ``` Alert heuristic: `predicted_positive_rate` shifts by > 2x relative to trailing 14-day median. ### Calibration/performance drift ```sql SELECT * FROM rain_calibration_drift_daily WHERE site = 'home' ORDER BY day DESC LIMIT 30; ``` Alert heuristic: sustained Brier-score increase > 25% from trailing 30-day average. ## 6) Pipeline Failure Alerts Use the health-check script in cron, systemd timer, or your alerting scheduler: ```sh python scripts/check_rain_pipeline_health.py \ --site home \ --model-name rain_next_4h \ --horizon-hours 4 \ --max-ws90-age 20m \ --max-baro-age 30m \ --max-forecast-age 3h \ --max-prediction-age 30m \ --max-pending-eval-age 6h \ --max-pending-eval-rows 200 ``` The script exits non-zero on failure, so it can directly drive alerting. ## 7) Continuous Worker Defaults `docker-compose.yml` provides these controls for `rainml`: - `RAIN_TUNE_HYPERPARAMETERS` - `RAIN_MAX_HYPERPARAM_TRIALS` - `RAIN_CALIBRATION_METHODS` - `RAIN_THRESHOLD_POLICY` - `RAIN_WALK_FORWARD_FOLDS` - `RAIN_ALLOW_EMPTY_DATA` - `RAIN_HORIZON_HOURS` - `RAIN_MODEL_BACKUP_PATH` - `RAIN_MODEL_CARD_PATH` Recommended production defaults: - Enable tuning daily or weekly (`RAIN_TUNE_HYPERPARAMETERS=true`) - Set `RAIN_THRESHOLD_POLICY=walk_forward` with `RAIN_WALK_FORWARD_FOLDS=4` for temporally robust threshold selection ## 8) Auto-Recommend Candidate To compare saved training reports and pick a deployment candidate automatically: ```sh python scripts/recommend_rain_model.py \ --reports-glob "models/rain_model_report*.json" \ --require-walk-forward \ --top-k 5 \ --json-out "models/rain_model_recommendation.json" ``` ## 9) Staged 4h Rollout Checklist Run this sequence in production/staging to satisfy the 4h cutover gate: 1. Apply schema migration for 4h predictions: ```sh docker compose exec -T timescaledb \ psql -U postgres -d micrometeo \ -f /docker-entrypoint-initdb.d/003_rain_predictions_4h.sql ``` 2. Re-apply monitoring views (now include 1h + 4h unions): ```sh docker compose exec -T timescaledb \ psql -U postgres -d micrometeo \ -f /docker-entrypoint-initdb.d/002_rain_monitoring_views.sql ``` 3. Run a full 4h training/evaluation cycle and save report: ```sh python scripts/train_rain_model.py \ --site "home" \ --start "2026-02-01T00:00:00Z" \ --end "2026-03-03T23:55:00Z" \ --horizon-hours 4 \ --feature-set "extended" \ --model-family "auto" \ --forecast-model "ecmwf" \ --tune-hyperparameters \ --threshold-policy "walk_forward" \ --walk-forward-folds 4 \ --model-version "rain-auto-v2-extended-4h" \ --out "models/rain_model_4h.pkl" \ --report-out "models/rain_model_report_4h.json" ``` 4. Compare 4h metrics against the latest 1h benchmark report before switching dashboard defaults: ```sh python scripts/compare_rain_reports.py \ --baseline "models/rain_model_report_1h.json" \ --candidate "models/rain_model_report_4h.json" ``` 5. Run dry-run inference, then live inference with 4h model name/horizon: ```sh python scripts/predict_rain_model.py \ --site home \ --model-path "models/rain_model_4h.pkl" \ --model-name "rain_next_4h" \ --horizon-hours 4 \ --dry-run python scripts/predict_rain_model.py \ --site home \ --model-path "models/rain_model_4h.pkl" \ --model-name "rain_next_4h" \ --horizon-hours 4 ``` 6. Validate health checks and dashboard data path for 4h: ```sh python scripts/check_rain_pipeline_health.py \ --site home \ --model-name rain_next_4h \ --horizon-hours 4 \ --max-pending-eval-age 6h ``` 7. Keep 1h path live in parallel until 4h drift/calibration remains stable for at least 7 days. ### Fast rollback to 1h If 4h performance or pipeline health regresses: 1. Set worker env back to: `RAIN_HORIZON_HOURS=1`, `RAIN_MODEL_NAME=rain_next_1h`, and a known-good 1h model path/version. 2. Restart `rainml` service. 3. Confirm `check_rain_pipeline_health.py --horizon-hours 1 --model-name rain_next_1h` returns `ok`. 4. Keep `predictions_rain_4h` data for postmortem; do not drop tables during rollback.