Operator troubleshooting index

On this page

Start with the policy, then follow its reason to the failing boundary.


First five commands

Set the policy name and namespace once:

NAMESPACE=default
POLICY=my-policy

Then collect the controller's view before changing anything:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" -o wide
kubectl describe pgr "$POLICY" --namespace "$NAMESPACE"
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" -- "}{.message}{"\n"}{end}'
kubectl get pgplan --namespace "$NAMESPACE"
kubectl get events --namespace "$NAMESPACE" \
  --field-selector involvedObject.name="$POLICY" \
  --sort-by='.lastTimestamp'

status.conditions[].reason is the primary index below. status.last_error usually preserves the underlying PostgreSQL, Kubernetes, or connection error:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.last_error}{"\n"}'

For controller-side detail, read the operator log:

kubectl logs --namespace pgroles-system \
  -l app.kubernetes.io/instance=pgroles-operator \
  --all-containers --tail=200

Adjust pgroles-system and the instance label if you used different Helm release or namespace names.

Reason index

Reason or symptomMeaning and next step
No status, or the policy kind is unknownCRD, operator Deployment, RBAC, or admission problem. Diagnose no status.
SecretMissing, SecretFetchFailedA connection or role-password Secret cannot be read. Check Secrets.
InvalidConnectionParamsA structured field or URL is empty or invalid. Check the connection.
DatabaseConnectionFailedDNS, network, TLS, database, or PostgreSQL login failure. Check the connection.
GcpAuthFailed, SetRoleFailedWorkload Identity token or post-login role switch failed. Check authentication.
InvalidSpecThe object passed admission but is not a valid pgroles policy. Check policy validation.
AbsenceAssertionsIgnored=TrueThe policy uses ensure: absent with additive reconciliation, which never revokes. Switch to adopt or authoritative to enforce it.
InvalidDatabaseTargetA database grant names a database other than the one the connection reaches. Set object.name to current_database(). Check database objects.
InvalidRoutineTargetA function or procedure grant names a routine PostgreSQL cannot resolve. The message lists the input-type signatures to use. Check database objects.
InsufficientPrivilegesThe executor can connect but cannot inspect or apply an operation. Check executor privileges.
ExecutorAuthorityThe authority preflight found a statement the executor cannot run, before any SQL executed. Resolve the authority gap.
MissingDatabaseObjectAn external object referenced by the policy is absent. Check database objects.
UnsatisfiableWildcardGrantA wildcard matched objects the executor cannot grant on. Check wildcard grants.
ConflictingPolicyAnother policy on the same database or server can undo this one's work. Resolve the conflict.
UnsafeRoleDropsA role drop is blocked by owned objects or other dependencies. Plan a safe retirement.
OwnerTransferBlockedAn adopt-mode apply would transfer schema ownership the policy does not allow. Decide the owner.
Planned, Drifted=True, plan PendingA preview or manual approval gate is working. Inspect the plan.
ApprovalIgnored, ApprovalUnsetApproval configuration is ineffective or implicit. Fix the execution gates.
ApplyFailed, DatabaseInspectionFailedA transient or uncategorized PostgreSQL operation failed. Check transient failures.
KubernetesApiError, PlanSqlStorageFailedKubernetes state or large-plan storage failed. Check infrastructure.
LockContentionAnother reconcile owns the same database lock. Check contention.
EphemeralAccessCleanupPendingDeletion is waiting for attached ephemeral-access resources. Check deletion.

No status or no reconcile

If kubectl does not recognize PostgresPolicy, install or update the CRDs. Helm installs CRDs on the first install but does not upgrade them on helm upgrade:

kubectl get crd postgrespolicies.pgroles.io
kubectl get crd postgrespolicyplans.pgroles.io
helm status pgroles-operator --namespace pgroles-system
kubectl get pods --namespace pgroles-system \
  -l app.kubernetes.io/instance=pgroles-operator

If the object exists but has no status, inspect the operator pod and logs. Check that its ServiceAccount can get, list, watch, and patch status for the CRDs, and can read Secrets in the policy namespace. An admission error from kubectl apply happens before the controller sees the object; read that error directly rather than waiting for a status condition.

Secret errors

SecretMissing means the named Secret or key is absent. Connection URL Secrets default to the key DATABASE_URL; role password keys have their own defaults. Confirm the reference, namespace, and available keys without printing secret values:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.spec.connection}{"\n"}'
kubectl get secret --namespace "$NAMESPACE"
SECRET=quick-start-database
kubectl get secret "$SECRET" --namespace "$NAMESPACE" \
  -o go-template='{{range $key, $_ := .data}}{{$key}}{{"\n"}}{{end}}'

SecretFetchFailed means the Kubernetes read itself failed, or generated-role password storage failed. Check the operator ServiceAccount permissions and the specific API error in last_error. Updating a referenced Secret's resourceVersion triggers reconnection automatically.

Connection and authentication

For InvalidConnectionParams, inspect last_error for the exact field. Common causes are an empty Secret value, invalid port, invalid sslMode, or malformed URL. For DatabaseConnectionFailed, test four boundaries from inside the cluster: DNS resolution, network reachability, TLS requirements, and the PostgreSQL username/password/database combination. A host that works on your laptop may not be reachable from the operator pod.

For GcpAuthFailed, verify the Kubernetes ServiceAccount annotation, Workload Identity binding, Cloud SQL login permission, IAM database username, and token endpoint response. For SetRoleFailed, the login succeeded but connection.params.setRole did not: the authenticated identity must be a member of the target PostgreSQL role.

See database connections for every connection shape and the Cloud SQL IAM flow.

Policy validation

InvalidSpec covers policy rules that are more contextual than the CRD schema: manifest expansion, interval parsing, password rules, and other cross-field constraints. The condition message names the invalid field or relationship. Fix the source manifest and re-apply it; do not edit status.

If kubectl apply rejects the YAML, that is CRD admission rather than InvalidSpec. Useful distinctions:

  • unknown or misspelled fields are rejected by the structural schema
  • unquoted role config values are rejected because those values are strings
  • invalid combinations such as a password on a non-login role may be caught by CRD validation or by the reconcile-time validator

Use the PostgresPolicy resource and the manifest reference to check field spelling and semantics.

Executor privileges

Permission denied

InsufficientPrivileges means the database credential is valid but cannot perform the requested inspection or mutation. The PostgreSQL error in last_error identifies the failed operation, such as permission denied to create role.

Do not respond by granting broad privileges blindly. Map the failed statement to the executor requirements: role management normally needs CREATEROLE and sometimes ADMIN OPTION; object grants need ownership or WITH GRANT OPTION; default privileges need membership in their owner role. The complete matrix and bootstrap SQL are in executor privileges.

Executor authority blocks execution

ExecutorAuthority means the read-only preflight found a planned statement the executor cannot run, and stopped before the first one. The whole plan waits — PostgreSQL applies it in one transaction — including changes the executor could make on their own. The message names each blocker; the commonest is a membership or object grant attributed to a grantor the executor cannot act as, such as a provider administrator on a managed service.

Nothing has changed in the database, and reconciling again will not supply the missing authority. Under approval: auto the plan stays Pending with status.current_plan_ref pointing at it, and is reused until authority exists. Resolve it one of three ways:

  • change the policy so it no longer needs that statement — keep the edge, or match the live membership options — and preview the change as a candidate first; candidates are still planned while an apply guard blocks the policy
  • grant the executor the specific authority named, never broad provider administration to get past the check
  • have someone holding that authority run the named statement once, then let the next reconcile converge

Denying the waiting plan stops that plan only; see plan and approval.

Missing database objects

InvalidDatabaseTarget is narrower than a missing object: a type: database grant explicitly names a database other than current_database(). Change the grant target or the connection so those names agree; the operator will not inspect one database and render ACL changes for another.

InvalidRoutineTarget means a function or procedure grant names a routine that PostgreSQL cannot resolve: an ensure: present rule for a routine that does not exist, a bare name shared by several overloads, or a named-argument signature spelled differently from the catalog. The condition message lists the input-type signatures of routines with that name in the schema. Name the routine by one of them, such as process(integer). See function signatures.

Before issuing DDL, the operator checks external schema references. Schemas declared in spec.schemas are excluded because the operator can create them; schemas referenced only by grants or default privileges must already exist.

MissingDatabaseObject therefore usually means one of three things:

  • the policy references a misspelled or not-yet-migrated schema
  • the operator should own the schema, but it is not declared in spec.schemas
  • the connection points at the wrong PostgreSQL database

The same reason can classify PostgreSQL undefined-object errors that pass the preflight. Read last_error, then create or declare the object, remove the reference, or correct the database connection.

Wildcard grants

UnsatisfiableWildcardGrant is a safety stop. A wildcard matched at least one table, sequence, or function where the executor lacks the ability to grant the requested privilege. The operator does not create a partial plan for that reconcile.

The condition message includes example objects and missing privileges. Give the executor ownership or the required grant option, narrow the wildcard, or manage those objects under a different ownership boundary. See grants and privileges for wildcard semantics.

Policy conflicts

ConflictingPolicy means another PostgresPolicy can undo this policy's work, or this one can undo the other's. Both stop applying until the conflict is resolved. The condition message names the other namespace/policy and each rule that applied. When the other policy connects through a different Secret or namespace, the message adds via <namespace>/<secret>/<key>; when it manages a different database on the same server, it adds on another database of the same server.

Roles and memberships belong to the whole server, so these rules apply to policies on any database of one server:

  • both manage the same role (neither declares it external: true). A policy that sets global default privileges for an owner manages that owner for this purpose;
  • one is adopt or authoritative and removes undeclared members from a role the other adds members to. Those modes remove undeclared members of the roles they create, and of any role they list with exclusive: true, including predefined pg_* roles;
  • both declare the same membership with different inherit or admin options;
  • one is authoritative and retires a role the other references in any way: as a declared or external role, grantee, member, or schema or default-privilege owner. Retirement drops the role and can drop or reassign what it owns; DROP OWNED also revokes its privileges on other databases.

These apply only to policies on the same database:

  • both declare the same schema under schemas;
  • one grants a privilege the other declares ensure: absent. Function names are compared without their argument lists, because f and f(integer) can name the same function;
  • one is adopt or authoritative, declares a role (external or not), and the other grants that role a privilege where the first one inspects it: in a schema the first names in a grant, default privilege or schemas entry; on the database, if the first declares a database grant; or in the global default privileges of an owner the first sets them for. Those modes revoke undeclared privileges of declared roles there;
  • both write PUBLIC privileges on the same database.

Policies that only grant to the same role do not conflict, as long as each grants where the other does not inspect. To let several policies grant to one shared role, create it in one policy and declare it with external: true in the others, so its grants are inspected and converge. Each policy can run in any mode if no two of them name the same schema; where they do, run the ones that do not create the role in additive mode.

An adopt or authoritative policy removes undeclared members of the roles it creates, whoever added them. If another policy's executor is a member of such a role, for example to grant on the objects it owns, declare that membership in the policy that creates the role.

Policies in observe mode and suspended policies cannot execute, so they do not block other policies. They still report a conflict themselves, and run the check again before they next apply.

A policy that can no longer connect keeps the target it last recorded. If its Secret now points elsewhere, or its database is gone, it can keep blocking a peer; suspend or delete it to release the peer. Changing reconcile timing does not solve a conflict.

Unsafe role drops

UnsafeRoleDrops blocks a role removal when PostgreSQL dependencies make a plain DROP ROLE unsafe. Decide explicitly where owned objects should go, then use a retirement with reassign_owned_to and/or drop_owned as appropriate. If active sessions are the issue, a retirement can terminate them when the executor has the required privilege.

Review the generated SQL in observe mode before approving destructive retirement steps. Do not remove the safety blocker merely to make the policy green.

Schema ownership transfer refused

OwnerTransferBlocked means an apply-mode policy in adopt reconciliation would run ALTER SCHEMA ... OWNER TO on a schema whose live owner differs from the declared one, and the policy does not set spec.allow_schema_owner_transfers. Adopt mode filters role drops, not ownership convergence, so the refusal is what stops pgroles from taking a schema away from whatever system owns it now. The guard runs before the plan's single transaction: nothing executed, and every other change in the plan waits with it.

Decide who should own the schema, then do one of these:

  • declare the current owner, so the policy no longer plans a transfer
  • set spec.allow_schema_owner_transfers: true on this policy, when the transfer is intended — the flag is per policy, so allowing it in staging does not allow it in production
  • transfer ownership out of band, with whatever else the transfer needs (CREATE on the schema for the incoming owner's objects, for example), and let the next reconcile find nothing to change

Preview the change as a candidate against each environment's policy first. Candidates run the same guard with that policy's settings, so a transfer the production policy would refuse shows as ExecutionBlocked before it merges, and a repair is still previewed while the policy is blocked.

A plan does not execute

Approval pending

A plan in Pending with manual approval needs a decision before it can execute.

First inspect all three execution gates:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='suspend={.spec.suspend}{" mode="}{.spec.mode}{" approval="}{.spec.approval}{"\n"}'
  • suspend: true stops reconciliation entirely
  • mode: observe computes plans but never executes mutating SQL
  • mode: apply with approval: manual waits for a terminal Approved decision on the current plan's status
  • mode: apply with approval: auto applies without a human gate, once the executor has the authority the plan needs; until then the plan stays Pending and the policy reports ExecutorAuthority

Follow the current plan and inspect its phase and conditions:

PLAN="$(kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.current_plan_ref.name}')"
kubectl get pgplan "$PLAN" --namespace "$NAMESPACE" -o yaml

A decision recorded on an observe-mode policy's plan produces ApprovalIgnored; switch the policy to mode: apply if you intend SQL to execute. ApprovalUnset means the operator inferred an approval mode from spec.mode; set approval explicitly because that inference is deprecated.

If policy or database state changed after review, the approved plan may become Superseded. Review and approve the newly referenced plan. If a plan was rejected, its replacement is created on the next reconcile rather than in the same cycle. Reject a plan by writing a terminal Denied condition and decidedBy to its status subresource; see plan and approval for the exact command and the full lifecycle.

A superseded plan names its cause in the Superseded=True condition message — effects changed, effects cleared, replaced by a newer plan, the policy stopped referencing it, another candidate's content was promoted and applied (reason SupersededByPromotion), or the target moved (reason TargetChanged, TargetIdentityUnavailable, or TargetIdentityAppeared).

Transient and infrastructure failures

ApplyFailed and DatabaseInspectionFailed preserve the underlying SQL error in last_error and use transient backoff. Check database availability, failovers, statement cancellation, and network stability before changing the policy.

KubernetesApiError points to API reachability, authorization, or an update conflict. PlanSqlStorageFailed means the operator could not persist a large SQL preview in its ConfigMap-backed storage. Check API errors, namespace quota, ConfigMap permissions, and object-size limits. The operator never executes SQL stored in the ConfigMap; it re-renders from current state before apply.

Use transient_failure_count to distinguish a repeating infrastructure problem from a single retry:

kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
  -o jsonpath='{.status.transient_failure_count}{"\n"}'

Lock contention

LockContention means another reconcile currently holds the in-process or PostgreSQL advisory lock for the same database target. The operator retries after a short jittered delay; an occasional occurrence needs no intervention.

If it persists, look for multiple policies or replicas targeting the same database and for slow inspection/apply cycles. Do not disable the lock: it is what prevents overlapping inspect/diff/apply transactions.

Deletion is stuck

Deleting a policy means “stop managing,” not “undo database changes.” The finalizer first waits for attached ephemeral-access policies to be removed. EphemeralAccessCleanupPending reports that guarded wait.

Inspect attached EphemeralAccessPolicy and EphemeralAccessRequest resources, allow their revocation/finalizers to complete, and then retry deletion. Do not strip finalizers until you have verified that temporary PostgreSQL memberships were revoked; forced Kubernetes deletion can otherwise leave access behind.

Force a fresh reconcile

After fixing the cause, you can wait for spec.interval, edit the policy, or request an immediate reconcile with a unique timestamp:

kubectl annotate pgr "$POLICY" --namespace "$NAMESPACE" \
  reconcile.pgroles.io/requestedAt="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --overwrite

Then re-read conditions rather than relying only on Events. Several reasons, including InvalidDatabaseTarget, MissingDatabaseObject, InvalidConnectionParams, and UnsatisfiableWildcardGrant, do not emit dedicated Events.