Status, conditions and telemetry

What the operator reports about itself, and what to alert on.

Status

The operator reports status on the custom resource:

status:
  conditions:
    - type: Ready
      status: "True"
      reason: Reconciled
      message: "Applied 5 changes"
      last_transition_time: "2026-03-06T10:30:00Z"
  observed_generation: 3
  last_successful_reconcile_time: "2026-03-06T10:30:00Z"
  lastHandledReconcileAt: "2026-03-06T10:31:00Z"
  transient_failure_count: 0
  change_summary:
    roles_created: 2
    roles_altered: 0
    roles_dropped: 0
    grants_added: 3
    grants_revoked: 0
    default_privileges_set: 2
    default_privileges_revoked: 0
    members_added: 1
    members_removed: 0
    total: 8
  plan_warnings:
    - "adopt mode transfers ownership of schema \"etl\" to \"pgloader_pg\""

plan_warnings lists advisory warnings from the last reconciliation's computed plan — an undeclared default_owner, or adopt-mode schema ownership transfers in observe-mode policies. The policy still reconciles; these are shapes worth reviewing, surfaced so they outlive the operator log window. Apply-mode adopt policies go further: schema-ownership transfers block reconciliation with an OwnerTransferBlocked condition unless spec.allow_schema_owner_transfers: true.

An insufficient-privilege failure looks more like:

status:
  conditions:
    - type: Ready
      status: "False"
      reason: InsufficientPrivileges
      message: "error returned from database: permission denied to create role"
    - type: Degraded
      status: "True"
      reason: InsufficientPrivileges
  last_error: "error returned from database: permission denied to create role"
  transient_failure_count: 0

Conditions

ApprovalUnset and ApprovalIgnored are advisory: they report a configuration that will not do what it looks like, and neither indicates a failed reconciliation. Both clear on the next reconcile once the configuration is corrected.

On failure, the operator chooses a retry path based on the failure mode:

• lock contention: short jittered retry
• transient operational failures: exponential backoff with jitter
• invalid specs, conflicts, and unsafe role-drop blockers: normal reconcile interval

Health and telemetry

The operator exposes health probes on its internal HTTP port:

• /livez
• /readyz

The Helm chart configures these probes automatically. /readyz returns 503 until all eight controller watches finish their initial lists and the reflected caches/request index have processed those objects. Empty collections also synchronize. A watch relist makes readiness false until that list completes; SIGTERM/Ctrl+C clears readiness before draining controller work. Unexpected controller-loop completion drains the remaining loops and exits with an error.

Readiness establishes cache synchronization and controller lifecycle state. It does not prove ongoing API connectivity, database access, telemetry delivery or policy convergence, and cannot detect every stalled controller. Quiet watches and reconnects have no time-since-event timeout, since an unchanged collection can be healthy. /livez reports that the HTTP handler can respond.

Metrics are exported via OpenTelemetry OTLP when standard OTel endpoint environment variables are set, for example:

operator:
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://otel-collector.observability.svc.cluster.local:4317
    - name: OTEL_METRICS_EXPORTER
      value: otlp

The intended deployment model is operator -> OpenTelemetry Collector -> your metrics backend. See Monitoring the operator for optional policy-state collection, tested alerts and dashboard queries.

Useful alerting signals: Degraded=True for reconcile failure, sustained Drifted=True on an auto-applying policy, pgroles.lock_contention.total rising steadily, and pgroles.deprecated.approval_unset as a cumulative count of reconciliations still relying on the deprecated inference.

A policy waiting on plan approval is healthy and reports Ready=True with reason Planned, alongside Drifted=True. Drifted is what distinguishes it from a policy with nothing to do; Ready=False always means something is wrong.

The operator also emits transition-based Kubernetes Events on the policy. Status transitions:

• ConflictDetected, ConflictResolved
• Suspended
• Reconciled, Recovered
• DriftDetected, PlanClean
• ApprovalUnset, ApprovalIgnored
• AbsenceAssertionsIgnored
• InvalidSpec
• SecretFetchFailed
• DatabaseConnectionFailed
• GcpAuthFailed
• InsufficientPrivileges
• UnsafeRoleDropsBlocked

Plan lifecycle:

• PlanCreated, PlanApproved, PlanRejected
• ApplyStarted, ApplySucceeded, ApplyFailed

Ephemeral access requests carry their own Events, recorded on the EphemeralAccessRequest object rather than on the policy, with the action EphemeralAccessLifecycle. Terminal failures — Failed, Denied, and ApprovalExpired — are Warning; every other phase transition is Normal. So kubectl describe -n <namespace> ephemeralaccessrequest <name> is where one request's history lives, not kubectl describe pgr.

Not every failure becomes an Event. InvalidDatabaseTarget, MissingDatabaseObject, InvalidConnectionParams, and UnsatisfiableWildcardGrant are condition reasons only — they appear in status.conditions[].reason and never as an Event, so alert on the condition rather than watching for an Event that will not arrive.