enhance utilisation of postgres features
continuous-integration/drone/push Build is passing

This commit is contained in:
2026-04-20 10:19:27 +10:00
parent 98e92a8264
commit 8ccf5a7009
28 changed files with 2836 additions and 422 deletions
+61
View File
@@ -269,6 +269,67 @@ The target architecture is:
- Retain explicit backfill and rebuild commands for compatibility tables and reports.
- Clean up obsolete styling rules and duplicated visual patterns once the new UI system is fully adopted.
## Implementation Checklist
### 0. Baseline and Guardrails
- [x] Capture baseline metrics for hourly capture, daily aggregation, monthly aggregation, and report generation.
- [x] Confirm current API/endpoint contract and report filename behavior with a regression snapshot.
- [x] Add new settings with defaults and config wiring:
- [x] `settings.capture_write_batch_size=1000`
- [x] `settings.snapshot_table_compat_mode=true`
- [x] `settings.async_report_generation=true`
- [x] Add/confirm stage-level logging and timing around capture, reconcile, totals refresh, and report generation.
- [x] Document migration guardrails: no auth-model changes, SQLite support retained, compatibility mode enabled by default.
- Evidence snapshot: see `phase0-baseline.md` for metrics, API/report contract snapshot, and guardrail verification.
### 1. Phase 1: Hot-Path Runtime Wins
- [x] Implement batched hourly writes for canonical tables in capture flow.
- [x] Add PostgreSQL multi-row insert/upsert path (or `COPY`) for `vm_hourly_stats`.
- [x] Keep SQLite transactional batched upsert path without PostgreSQL-only ingestion features.
- [x] Decouple XLSX/report generation from capture hot path via async/deferred stage.
- [x] Ensure scheduled daily aggregation reads canonical data from `vm_hourly_stats` only.
- [x] Ensure scheduled monthly aggregation reads canonical data from `vm_daily_rollup` only.
- [x] Keep legacy compatibility tables enabled during this phase.
- [x] Introduce UI token layer (`--theme_*`) and map shared component primitives before page-specific redesign.
### 2. Phase 2: Canonical Dataflow
- [x] Refactor capture/reconcile ordering so canonical caches are updated first.
- [x] Move deletion/event reconciliation to one post-capture phase per vCenter.
- [x] Remove prior-snapshot table mutations from capture hot path (except explicit compatibility needs).
- [x] Keep SQL union/legacy scan paths available only for fallback, repair, and backfill.
- [x] Verify `snapshot_registry` logical hourly registration remains correct without normal hourly table scans.
- [x] Implement shared Templ page shell improvements across header/footer/cards/buttons/tables/forms.
- [x] Refresh dashboard, snapshots, vCenter totals, and VM trace views to the tokenized design system.
### 3. Phase 3: Postgres-Ready Scale-Up
- [x] Validate/add canonical `vm_hourly_stats` indexes for snapshot time, vCenter+time, VM identity+time, and trace lookup.
- [x] Add PostgreSQL monthly partitioning for `vm_hourly_stats` behind migration controls.
- [ ] Benchmark Go vs SQL on canonical Postgres tables using representative production-scale data.
- Benchmark harness implemented via `-benchmark-aggregations` and `-benchmark-runs`; production-scale Postgres run pending.
- [x] Keep Go as scheduled default unless SQL shows clear and repeatable runtime wins.
- [x] If SQL wins, roll out behind a controlled flag before any default switch.
### 4. Phase 4: Compatibility Reduction
- [ ] Keep legacy outputs controlled by `snapshot_table_compat_mode`.
- [ ] Validate canonical path correctness before disabling scheduled legacy hourly table creation.
- [ ] Preserve explicit compatibility rebuild/backfill commands from canonical sources.
- [ ] Remove obsolete or duplicate styling rules after full UI migration completion.
### 5. Validation and Quality Gates
- [ ] Add golden-result tests for daily output parity (old vs new path).
- [ ] Add golden-result tests for monthly output parity (old vs new path).
- [ ] Add lifecycle edge-case coverage (partial presence, missing create times, deletion refinement, pool and resource changes).
- [ ] Add integration tests for canonical write/read paths and totals cache correctness.
- [ ] Add compatibility tests for legacy table generation, reports, and rebuild flows.
- [ ] Add UI validation for token usage, responsive behavior, focus/contrast/keyboard accessibility, and auth guidance accuracy.
- [ ] Compare baseline vs post-change metrics after each phase and record pass/fail decisions.
### 6. Rollout and Documentation
- [ ] Update operator docs for new settings and default behavior.
- [ ] Document compatibility-mode lifecycle and criteria to disable legacy table generation.
- [ ] Document benchmark method/results and default-path decision record (Go vs SQL).
- [ ] Publish a short migration runbook for staged rollout, rollback triggers, and repair workflows.
## Test Plan
### Correctness Tests