Operator architecture

How the pgroles operator watches Kubernetes resources, talks to PostgreSQL, and enforces safe reconciliation.

Overview

The operator is a Kubernetes controller around the same core model as the CLI:

1. read desired state from a PostgresPolicy
2. inspect live PostgreSQL state
3. compute a convergent diff
4. either apply changes in a single transaction or publish a non-mutating plan
5. write status back to Kubernetes

The important difference is that the operator has to do this continuously, safely, and in the presence of concurrent policy updates, secret changes, and transient infrastructure failures.

Control-plane diagram

Main components

CRD and policy model

PostgresPolicy is the operator-facing API. Its spec mirrors the CLI manifest format, with Kubernetes-specific fields for:

• Secret-based connection lookup
• reconciliation interval
• execution mode (apply or observe) and approval mode (auto or manual)
• reconciliation mode (authoritative, additive, or adopt)
• suspend/pause behavior

The controller converts the CRD into the same manifest types used by the CLI, so both paths share expansion, diffing, and SQL rendering semantics.

Watch sources

The operator currently reconciles from four event-driven trigger sources, plus the periodic interval:

• PostgresPolicy generation changes
• PostgresPolicy reconcile.pgroles.io/requestedAt annotation changes
• Secret resourceVersion changes for referenced database credentials
• PostgresPolicyPlan changes, mapped back to the owning policy by controller-owner UID — this is what makes an approval or rejection recorded on a plan's status take effect immediately rather than at the next interval

Generation and annotation filtering matter. The controller intentionally ignores status-only PostgresPolicy updates and unrelated annotation changes as reconcile triggers, otherwise successful status patches and GitOps tracking metadata can create hot loops that starve other policies targeting the same database.

By default these watches cover all namespaces. Setting WATCH_NAMESPACE (or the Helm value operator.watchNamespace) scopes every policy, plan, Secret, access-policy, and access-request watch to one namespace. The chart also reduces the operator's RBAC to that namespace.

Ephemeral requests share one reflector with a watch-fed index keyed by access policy name, resolved access-policy UID, and resolved target-policy UID. The index is refreshed atomically after watcher relists. Controller-owned UID labels are routing hints for server-side inspection; reconciliation always verifies the immutable UIDs stored in status.resolvedAccess.

Database connection handling

The operator resolves credentials from a Secret in the same namespace as the policy — either a whole connection URL via connection.secretRef, or individual connection.params fields, each of which may be a literal or its own Secret reference. It caches sqlx::PgPool instances. In URL mode the cache identity is:

namespace / secret name / secret key

In connection.params mode the key covers every connection field, and the operator additionally fingerprints the resourceVersion of all referenced Secrets.

When any referenced Secret changes, the controller refetches it and refreshes the cached pool on the next reconcile. This is what enables credential rotation and recovery without restarting the operator.

Reconcile engine

Each reconcile follows this path:

PostgresPolicy
  -> Secret fetch
  -> PolicyManifest conversion
  -> manifest expansion
  -> live database inspection
  -> diff engine
  -> SQL rendering
  -> apply transaction or plan-only status update
  -> status patch

The diff/apply behavior is shared with the CLI. The operator is not a separate implementation of role management logic; it is a controller wrapped around the same core engine.

Safety model

Ownership and conflict detection

Multiple policies may target the same database only if their ownership claims are disjoint.

The controller derives ownership claims from declared roles and schema/profile expansions, then rejects overlapping policies by setting conflict status instead of letting policies revoke each other's grants.

Per-database serialization

Reconciliation is serialized per database target in two layers:

1. in-process locking to prevent concurrent reconciles in one operator replica
2. PostgreSQL advisory locking to prevent concurrent reconciles across replicas

This is the core safety boundary for production use. It ensures one database target has one active inspect/diff/apply cycle at a time.

Retry and backoff

The controller uses different retry behavior depending on failure class:

• invalid specs and ownership conflicts: normal reconcile interval
• secret-missing and other non-transient configuration errors: normal reconcile interval
• transient infrastructure/database failures: exponential backoff with jitter
• lock contention: short dedicated retry path

That keeps the operator from hammering the Kubernetes API or the database during persistent misconfiguration.

Status model

The operator writes status conditions and summaries back to the PostgresPolicy, including:

• Ready
• Drifted
• Reconciling
• Degraded
• Conflict
• Paused
• ApprovalUnset and ApprovalIgnored, which are advisory: they report a configuration that will not behave as it looks, not a failed reconcile

It also records:

• last attempted generation
• last successful reconcile time
• last handled force-reconcile annotation timestamp
• managed database identity
• owned roles and schemas
• last error
• transient failure count
• change summary
• last reconcile mode
• current plan reference

This is the main operator-facing debugging surface for SREs. The rendered SQL lives on the referenced PostgresPolicyPlan.

Observability

The operator exposes:

• /livez
• /readyz

Metrics are exported through OpenTelemetry OTLP. The intended deployment model is:

pgroles-operator -> OpenTelemetry Collector -> metrics backend

The operator deliberately does not default to a built-in Prometheus scrape endpoint.

Ephemeral access uses the same database and advisory locks as durable reconciliation. Request-owned scoped plans establish execution provenance, and their active membership edges are composed into the single effective desired graph before diffing. See ephemeral access for the request, approval, expiry, and session contracts, and securing ephemeral access for the deployment trust and audit boundaries.

For object-local debugging, the controller also emits transition-based Kubernetes Events for notable status changes such as conflicts, suspend/resume, observe-mode drift detection, recovery, secret failures, database connectivity failures, and insufficient privileges. The intended split is:

• status: current state of the policy
• Events: notable transitions visible in kubectl describe
• OTLP metrics: fleet-level trends and alerting

Relationship to the CLI

The CLI remains the simplest path for explicit, reviewed role changes. The operator is the continuous control plane version of the same model.

That split is intentional:

• the CLI is the easiest way to validate and review manifests
• the operator is the right place to enforce drift correction continuously inside Kubernetes

For the external user model, see the operator guide. For the shared workspace structure, see the general architecture page.