vctp2/README.md

# Overview
vCTP is a vSphere Chargeback Tracking Platform, designed for a specific customer, so some decisions may not be applicable for your use case.

## Snapshots and Reports
- Hourly snapshots capture inventory per vCenter (concurrency via `hourly_snapshot_concurrency`).
- Daily summaries aggregate the hourly snapshots for the day; monthly summaries aggregate daily summaries for the month (or hourly snapshots if configured).
- Snapshots are registered in `snapshot_registry` so regeneration via `/api/snapshots/aggregate` can locate the correct tables (fallback scanning is also supported).
- Reports (XLSX with totals/charts) are generated automatically after hourly, daily, and monthly jobs and written to a reports directory.
- Hourly totals in reports are interval-based: each row represents `[HH:00, HH+1:00)` and uses the first snapshot at or after the hour end (including cross-day snapshots) to prorate VM presence by creation/deletion overlap.
- Monthly aggregation reports include a Daily Totals sheet with full-day interval labels (`YYYY-MM-DD to YYYY-MM-DD`) and prorated totals derived from daily summaries.
- Prometheus metrics are exposed at `/metrics`:
  - Snapshots/aggregations: `vctp_hourly_snapshots_total`, `vctp_hourly_snapshots_failed_total`, `vctp_hourly_snapshot_last_unix`, `vctp_hourly_snapshot_last_rows`, `vctp_daily_aggregations_total`, `vctp_daily_aggregations_failed_total`, `vctp_daily_aggregation_duration_seconds`, `vctp_monthly_aggregations_total`, `vctp_monthly_aggregations_failed_total`, `vctp_monthly_aggregation_duration_seconds`, `vctp_reports_available`
  - vCenter health/perf: `vctp_vcenter_connect_failures_total{vcenter}`, `vctp_vcenter_snapshot_duration_seconds{vcenter}`, `vctp_vcenter_inventory_size{vcenter}`

## Prorating and Aggregation Logic
Daily aggregation runs per VM using sample counts for the day:
- `SamplesPresent`: count of snapshot samples in which the VM appears.
- `TotalSamples`: count of unique snapshot timestamps for the vCenter in the day.
- `AvgIsPresent`: `SamplesPresent / TotalSamples` (0 when `TotalSamples` is 0).
- `AvgVcpuCount`, `AvgRamGB`, `AvgProvisionedDisk` (daily): `sum(values_per_sample) / TotalSamples` to time‑weight config changes and prorate partial‑day VMs.
- `PoolTinPct`, `PoolBronzePct`, `PoolSilverPct`, `PoolGoldPct` (daily): `(pool_hits / SamplesPresent) * 100`, so pool percentages reflect only the time the VM existed.
- `CreationTime`: only set when vCenter provides it; otherwise it remains `0`.

Monthly aggregation builds on daily summaries (or the daily rollup cache):
- For each VM, daily averages are converted to weighted sums: `daily_avg * daily_total_samples`.
- Monthly averages are `sum(weighted_sums) / monthly_total_samples` (per vCenter).
- Pool percentages are weighted the same way: `(daily_pool_pct / 100) * daily_total_samples`, summed, then divided by `monthly_total_samples` and multiplied by 100.

## RPM Layout (summary)
The RPM installs the service and defaults under `/usr/bin`, config under `/etc/dtms`, and data under `/var/lib/vctp`:
- Binary: `/usr/bin/vctp-linux-amd64`
- Systemd unit: `/etc/systemd/system/vctp.service`
- Defaults/env: `/etc/dtms/vctp.yml` (override with `-settings`), `/etc/default/vctp` (environment)
- TLS cert/key: `/etc/dtms/vctp.crt` and `/etc/dtms/vctp.key` (generated if absent)
- Data: SQLite DB and reports default to `/var/lib/vctp` (reports under `/var/lib/vctp/reports`)
- Scripts: preinstall/postinstall handle directory creation and permissions.

# Settings File
Configuration now lives in the YAML settings file. By default the service reads
`/etc/dtms/vctp.yml`, or you can override it with the `-settings` flag.

```shell
vctp -settings /path/to/vctp.yml
```

If you just want to run a single inventory snapshot across all configured vCenters and
exit (no scheduler/server), use:

```shell
vctp -settings /path/to/vctp.yml -run-inventory
```

## Database Configuration
By default the app uses SQLite and creates/opens `db.sqlite3`. You can opt into PostgreSQL
by updating the settings file:

- `settings.database_driver`: `sqlite` (default) or `postgres`
- `settings.database_url`: SQLite file path/DSN or PostgreSQL DSN

Examples:
```yaml
settings:
  database_driver: sqlite
  database_url: ./db.sqlite3

settings:
  database_driver: postgres
  database_url: postgres://user:pass@localhost:5432/vctp?sslmode=disable
```

PostgreSQL migrations live in `db/migrations_postgres`, while SQLite migrations remain in
`db/migrations`.

## Snapshot Retention
Hourly and daily snapshot table retention can be configured in the settings file:

- `settings.hourly_snapshot_max_age_days` (default: 60)
- `settings.daily_snapshot_max_age_months` (default: 12)

## Settings Reference
All configuration lives under the top-level `settings:` key in `vctp.yml`.

General:
- `settings.log_level`: logging verbosity (e.g., `debug`, `info`, `warn`, `error`)
- `settings.log_output`: log format, `text` or `json`

Database:
- `settings.database_driver`: `sqlite` or `postgres`
- `settings.database_url`: SQLite file path/DSN or PostgreSQL DSN

HTTP/TLS:
- `settings.bind_ip`: IP address to bind the HTTP server
- `settings.bind_port`: TCP port to bind the HTTP server
- `settings.bind_disable_tls`: `true` to serve plain HTTP (no TLS)
- `settings.tls_cert_filename`: PEM certificate path (TLS mode)
- `settings.tls_key_filename`: PEM private key path (TLS mode)

vCenter:
- `settings.vcenter_username`: vCenter username
- `settings.vcenter_password`: vCenter password (encrypted at startup)
- `settings.vcenter_insecure`: `true` to skip TLS verification
- `settings.vcenter_event_polling_seconds`: event polling interval (0 disables)
- `settings.vcenter_inventory_polling_seconds`: inventory polling interval (0 disables)
- `settings.vcenter_inventory_snapshot_seconds`: hourly snapshot cadence (seconds)
- `settings.vcenter_inventory_aggregate_seconds`: daily aggregation cadence (seconds)
- `settings.vcenter_addresses`: list of vCenter SDK URLs to monitor

Snapshots:
- `settings.hourly_snapshot_concurrency`: max concurrent vCenter snapshots (0 = unlimited)
- `settings.hourly_snapshot_max_age_days`: retention for hourly tables
- `settings.daily_snapshot_max_age_months`: retention for daily tables
- `settings.snapshot_cleanup_cron`: cron expression for cleanup job
- `settings.reports_dir`: directory to store generated XLSX reports (default: `/var/lib/vctp/reports`)
- `settings.hourly_snapshot_retry_seconds`: interval for retrying failed hourly snapshots (default: 300 seconds)
- `settings.hourly_snapshot_max_retries`: maximum retry attempts per vCenter snapshot (default: 3)

Filters/chargeback:
- `settings.tenants_to_filter`: list of tenant name patterns to exclude
- `settings.node_charge_clusters`: list of cluster name patterns for node chargeback
- `settings.srm_activeactive_vms`: list of SRM Active/Active VM name patterns

# Developer setup

## Pre-requisite tools

```shell
go install github.com/a-h/templ/cmd/templ@latest
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
go install github.com/swaggo/swag/cmd/swag@latest
```

## Database
This project now uses [goose](https://github.com/pressly/goose) for DB migrations.

Install via `brew install goose` on a mac, or install via golang with command `go install github.com/pressly/goose/v3/cmd/goose@latest`

Create a new up/down migration file with this command
```shell
goose -dir db/migrations sqlite3 ./db.sqlite3 create init sql
```

```shell
sqlc generate
```

## HTML templates
Run `templ generate -path ./components` to generate code based on template files

## Documentation
Run `swag init --exclude "pkg.mod,pkg.build,pkg.tools" -o server/router/docs`

## CI/CD (Drone)
- `.drone.yml` defines a Docker pipeline:
  - Restore/build caches for Go modules/tools.
  - Build step installs generators (`templ`, `sqlc`, `swag`), regenerates code/docs, runs project scripts, and produces the `vctp-linux-amd64` binary.
  - RPM step packages via `nfpm` using `vctp.yml`, emits RPMs into `./build/`.
  - Optional SFTP deploy step uploads build artifacts (e.g., `vctp*`) to a remote host.
  - Cache rebuild step preserves Go caches across runs.