Operator troubleshooting index
On this page
Start with the policy, then follow its reason to the failing boundary.
First five commands
Set the policy name and namespace once:
NAMESPACE=default
POLICY=my-policy
Then collect the controller's view before changing anything:
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" -o wide
kubectl describe pgr "$POLICY" --namespace "$NAMESPACE"
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" -- "}{.message}{"\n"}{end}'
kubectl get pgplan --namespace "$NAMESPACE"
kubectl get events --namespace "$NAMESPACE" \
--field-selector involvedObject.name="$POLICY" \
--sort-by='.lastTimestamp'
status.conditions[].reason is the primary index below. status.last_error usually preserves the underlying PostgreSQL, Kubernetes, or connection error:
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='{.status.last_error}{"\n"}'
For controller-side detail, read the operator log:
kubectl logs --namespace pgroles-system \
-l app.kubernetes.io/instance=pgroles-operator \
--all-containers --tail=200
Adjust pgroles-system and the instance label if you used different Helm release or namespace names.
Reason index
| Reason or symptom | Meaning and next step |
|---|---|
| No status, or the policy kind is unknown | CRD, operator Deployment, RBAC, or admission problem. Diagnose no status. |
SecretMissing, SecretFetchFailed | A connection or role-password Secret cannot be read. Check Secrets. |
InvalidConnectionParams | A structured field or URL is empty or invalid. Check the connection. |
DatabaseConnectionFailed | DNS, network, TLS, database, or PostgreSQL login failure. Check the connection. |
GcpAuthFailed, SetRoleFailed | Workload Identity token or post-login role switch failed. Check authentication. |
InvalidSpec | The object passed admission but is not a valid pgroles policy. Check policy validation. |
AbsenceAssertionsIgnored=True | The policy uses ensure: absent with additive reconciliation, which never revokes. Switch to adopt or authoritative to enforce it. |
InvalidDatabaseTarget | A database grant names a database other than the one the connection reaches. Set object.name to current_database(). Check database objects. |
InvalidRoutineTarget | A function or procedure grant names a routine PostgreSQL cannot resolve. The message lists the input-type signatures to use. Check database objects. |
InsufficientPrivileges | The executor can connect but cannot inspect or apply an operation. Check executor privileges. |
ExecutorAuthority | The authority preflight found a statement the executor cannot run, before any SQL executed. Resolve the authority gap. |
MissingDatabaseObject | An external object referenced by the policy is absent. Check database objects. |
UnsatisfiableWildcardGrant | A wildcard matched objects the executor cannot grant on. Check wildcard grants. |
ConflictingPolicy | Another policy on the same database or server can undo this one's work. Resolve the conflict. |
UnsafeRoleDrops | A role drop is blocked by owned objects or other dependencies. Plan a safe retirement. |
OwnerTransferBlocked | An adopt-mode apply would transfer schema ownership the policy does not allow. Decide the owner. |
Planned, Drifted=True, plan Pending | A preview or manual approval gate is working. Inspect the plan. |
ApprovalIgnored, ApprovalUnset | Approval configuration is ineffective or implicit. Fix the execution gates. |
ApplyFailed, DatabaseInspectionFailed | A transient or uncategorized PostgreSQL operation failed. Check transient failures. |
KubernetesApiError, PlanSqlStorageFailed | Kubernetes state or large-plan storage failed. Check infrastructure. |
LockContention | Another reconcile owns the same database lock. Check contention. |
EphemeralAccessCleanupPending | Deletion is waiting for attached ephemeral-access resources. Check deletion. |
No status or no reconcile
If kubectl does not recognize PostgresPolicy, install or update the CRDs. Helm installs CRDs on the first install but does not upgrade them on helm upgrade:
kubectl get crd postgrespolicies.pgroles.io
kubectl get crd postgrespolicyplans.pgroles.io
helm status pgroles-operator --namespace pgroles-system
kubectl get pods --namespace pgroles-system \
-l app.kubernetes.io/instance=pgroles-operator
If the object exists but has no status, inspect the operator pod and logs. Check that its ServiceAccount can get, list, watch, and patch status for the CRDs, and can read Secrets in the policy namespace. An admission error from kubectl apply happens before the controller sees the object; read that error directly rather than waiting for a status condition.
Secret errors
SecretMissing means the named Secret or key is absent. Connection URL Secrets default to the key DATABASE_URL; role password keys have their own defaults. Confirm the reference, namespace, and available keys without printing secret values:
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='{.spec.connection}{"\n"}'
kubectl get secret --namespace "$NAMESPACE"
SECRET=quick-start-database
kubectl get secret "$SECRET" --namespace "$NAMESPACE" \
-o go-template='{{range $key, $_ := .data}}{{$key}}{{"\n"}}{{end}}'
SecretFetchFailed means the Kubernetes read itself failed, or generated-role password storage failed. Check the operator ServiceAccount permissions and the specific API error in last_error. Updating a referenced Secret's resourceVersion triggers reconnection automatically.
Connection and authentication
For InvalidConnectionParams, inspect last_error for the exact field. Common causes are an empty Secret value, invalid port, invalid sslMode, or malformed URL. For DatabaseConnectionFailed, test four boundaries from inside the cluster: DNS resolution, network reachability, TLS requirements, and the PostgreSQL username/password/database combination. A host that works on your laptop may not be reachable from the operator pod.
For GcpAuthFailed, verify the Kubernetes ServiceAccount annotation, Workload Identity binding, Cloud SQL login permission, IAM database username, and token endpoint response. For SetRoleFailed, the login succeeded but connection.params.setRole did not: the authenticated identity must be a member of the target PostgreSQL role.
See database connections for every connection shape and the Cloud SQL IAM flow.
Policy validation
InvalidSpec covers policy rules that are more contextual than the CRD schema: manifest expansion, interval parsing, password rules, and other cross-field constraints. The condition message names the invalid field or relationship. Fix the source manifest and re-apply it; do not edit status.
If kubectl apply rejects the YAML, that is CRD admission rather than InvalidSpec. Useful distinctions:
- unknown or misspelled fields are rejected by the structural schema
- unquoted role
configvalues are rejected because those values are strings - invalid combinations such as a password on a non-login role may be caught by CRD validation or by the reconcile-time validator
Use the PostgresPolicy resource and the manifest reference to check field spelling and semantics.
Executor privileges
Permission denied
InsufficientPrivileges means the database credential is valid but cannot perform the requested inspection or mutation. The PostgreSQL error in last_error identifies the failed operation, such as permission denied to create role.
Do not respond by granting broad privileges blindly. Map the failed statement to the executor requirements: role management normally needs CREATEROLE and sometimes ADMIN OPTION; object grants need ownership or WITH GRANT OPTION; default privileges need membership in their owner role. The complete matrix and bootstrap SQL are in executor privileges.
Executor authority blocks execution
ExecutorAuthority means the read-only preflight found a planned statement the executor cannot run, and stopped before the first one. The whole plan waits — PostgreSQL applies it in one transaction — including changes the executor could make on their own. The message names each blocker; the commonest is a membership or object grant attributed to a grantor the executor cannot act as, such as a provider administrator on a managed service.
Nothing has changed in the database, and reconciling again will not supply the missing authority. Under approval: auto the plan stays Pending with status.current_plan_ref pointing at it, and is reused until authority exists. Resolve it one of three ways:
- change the policy so it no longer needs that statement — keep the edge, or match the live membership options — and preview the change as a candidate first; candidates are still planned while an apply guard blocks the policy
- grant the executor the specific authority named, never broad provider administration to get past the check
- have someone holding that authority run the named statement once, then let the next reconcile converge
Denying the waiting plan stops that plan only; see plan and approval.
Missing database objects
InvalidDatabaseTarget is narrower than a missing object: a type: database grant explicitly names a database other than current_database(). Change the grant target or the connection so those names agree; the operator will not inspect one database and render ACL changes for another.
InvalidRoutineTarget means a function or procedure grant names a routine that PostgreSQL cannot resolve: an ensure: present rule for a routine that does not exist, a bare name shared by several overloads, or a named-argument signature spelled differently from the catalog. The condition message lists the input-type signatures of routines with that name in the schema. Name the routine by one of them, such as process(integer). See function signatures.
Before issuing DDL, the operator checks external schema references. Schemas declared in spec.schemas are excluded because the operator can create them; schemas referenced only by grants or default privileges must already exist.
MissingDatabaseObject therefore usually means one of three things:
- the policy references a misspelled or not-yet-migrated schema
- the operator should own the schema, but it is not declared in
spec.schemas - the connection points at the wrong PostgreSQL database
The same reason can classify PostgreSQL undefined-object errors that pass the preflight. Read last_error, then create or declare the object, remove the reference, or correct the database connection.
Wildcard grants
UnsatisfiableWildcardGrant is a safety stop. A wildcard matched at least one table, sequence, or function where the executor lacks the ability to grant the requested privilege. The operator does not create a partial plan for that reconcile.
The condition message includes example objects and missing privileges. Give the executor ownership or the required grant option, narrow the wildcard, or manage those objects under a different ownership boundary. See grants and privileges for wildcard semantics.
Policy conflicts
ConflictingPolicy means another PostgresPolicy can undo this policy's work, or this one can undo the other's. Both stop applying until the conflict is resolved. The condition message names the other namespace/policy and each rule that applied. When the other policy connects through a different Secret or namespace, the message adds via <namespace>/<secret>/<key>; when it manages a different database on the same server, it adds on another database of the same server.
Roles and memberships belong to the whole server, so these rules apply to policies on any database of one server:
- both manage the same role (neither declares it
external: true). A policy that sets global default privileges for an owner manages that owner for this purpose; - one is
adoptorauthoritativeand removes undeclared members from a role the other adds members to. Those modes remove undeclared members of the roles they create, and of any role they list withexclusive: true, including predefinedpg_*roles; - both declare the same membership with different
inheritoradminoptions; - one is
authoritativeand retires a role the other references in any way: as a declared or external role, grantee, member, or schema or default-privilege owner. Retirement drops the role and can drop or reassign what it owns;DROP OWNEDalso revokes its privileges on other databases.
These apply only to policies on the same database:
- both declare the same schema under
schemas; - one grants a privilege the other declares
ensure: absent. Function names are compared without their argument lists, becausefandf(integer)can name the same function; - one is
adoptorauthoritative, declares a role (external or not), and the other grants that role a privilege where the first one inspects it: in a schema the first names in a grant, default privilege orschemasentry; on the database, if the first declares a database grant; or in the global default privileges of an owner the first sets them for. Those modes revoke undeclared privileges of declared roles there; - both write
PUBLICprivileges on the same database.
Policies that only grant to the same role do not conflict, as long as each grants where the other does not inspect. To let several policies grant to one shared role, create it in one policy and declare it with external: true in the others, so its grants are inspected and converge. Each policy can run in any mode if no two of them name the same schema; where they do, run the ones that do not create the role in additive mode.
An adopt or authoritative policy removes undeclared members of the roles it creates, whoever added them. If another policy's executor is a member of such a role, for example to grant on the objects it owns, declare that membership in the policy that creates the role.
Policies in observe mode and suspended policies cannot execute, so they do not block other policies. They still report a conflict themselves, and run the check again before they next apply.
A policy that can no longer connect keeps the target it last recorded. If its Secret now points elsewhere, or its database is gone, it can keep blocking a peer; suspend or delete it to release the peer. Changing reconcile timing does not solve a conflict.
Unsafe role drops
UnsafeRoleDrops blocks a role removal when PostgreSQL dependencies make a plain DROP ROLE unsafe. Decide explicitly where owned objects should go, then use a retirement with reassign_owned_to and/or drop_owned as appropriate. If active sessions are the issue, a retirement can terminate them when the executor has the required privilege.
Review the generated SQL in observe mode before approving destructive retirement steps. Do not remove the safety blocker merely to make the policy green.
Schema ownership transfer refused
OwnerTransferBlocked means an apply-mode policy in adopt reconciliation would run ALTER SCHEMA ... OWNER TO on a schema whose live owner differs from the declared one, and the policy does not set spec.allow_schema_owner_transfers. Adopt mode filters role drops, not ownership convergence, so the refusal is what stops pgroles from taking a schema away from whatever system owns it now. The guard runs before the plan's single transaction: nothing executed, and every other change in the plan waits with it.
Decide who should own the schema, then do one of these:
- declare the current owner, so the policy no longer plans a transfer
- set
spec.allow_schema_owner_transfers: trueon this policy, when the transfer is intended — the flag is per policy, so allowing it in staging does not allow it in production - transfer ownership out of band, with whatever else the transfer needs (
CREATEon the schema for the incoming owner's objects, for example), and let the next reconcile find nothing to change
Preview the change as a candidate against each environment's policy first. Candidates run the same guard with that policy's settings, so a transfer the production policy would refuse shows as ExecutionBlocked before it merges, and a repair is still previewed while the policy is blocked.
A plan does not execute
Approval pending
A plan in Pending with manual approval needs a decision before it can execute.
First inspect all three execution gates:
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='suspend={.spec.suspend}{" mode="}{.spec.mode}{" approval="}{.spec.approval}{"\n"}'
suspend: truestops reconciliation entirelymode: observecomputes plans but never executes mutating SQLmode: applywithapproval: manualwaits for a terminalApproveddecision on the current plan's statusmode: applywithapproval: autoapplies without a human gate, once the executor has the authority the plan needs; until then the plan staysPendingand the policy reportsExecutorAuthority
Follow the current plan and inspect its phase and conditions:
PLAN="$(kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='{.status.current_plan_ref.name}')"
kubectl get pgplan "$PLAN" --namespace "$NAMESPACE" -o yaml
A decision recorded on an observe-mode policy's plan produces ApprovalIgnored; switch the policy to mode: apply if you intend SQL to execute. ApprovalUnset means the operator inferred an approval mode from spec.mode; set approval explicitly because that inference is deprecated.
If policy or database state changed after review, the approved plan may become Superseded. Review and approve the newly referenced plan. If a plan was rejected, its replacement is created on the next reconcile rather than in the same cycle. Reject a plan by writing a terminal Denied condition and decidedBy to its status subresource; see plan and approval for the exact command and the full lifecycle.
A superseded plan names its cause in the Superseded=True condition message — effects changed, effects cleared, replaced by a newer plan, the policy stopped referencing it, another candidate's content was promoted and applied (reason SupersededByPromotion), or the target moved (reason TargetChanged, TargetIdentityUnavailable, or TargetIdentityAppeared).
Transient and infrastructure failures
ApplyFailed and DatabaseInspectionFailed preserve the underlying SQL error in last_error and use transient backoff. Check database availability, failovers, statement cancellation, and network stability before changing the policy.
KubernetesApiError points to API reachability, authorization, or an update conflict. PlanSqlStorageFailed means the operator could not persist a large SQL preview in its ConfigMap-backed storage. Check API errors, namespace quota, ConfigMap permissions, and object-size limits. The operator never executes SQL stored in the ConfigMap; it re-renders from current state before apply.
Use transient_failure_count to distinguish a repeating infrastructure problem from a single retry:
kubectl get pgr "$POLICY" --namespace "$NAMESPACE" \
-o jsonpath='{.status.transient_failure_count}{"\n"}'
Lock contention
LockContention means another reconcile currently holds the in-process or PostgreSQL advisory lock for the same database target. The operator retries after a short jittered delay; an occasional occurrence needs no intervention.
If it persists, look for multiple policies or replicas targeting the same database and for slow inspection/apply cycles. Do not disable the lock: it is what prevents overlapping inspect/diff/apply transactions.
Deletion is stuck
Deleting a policy means “stop managing,” not “undo database changes.” The finalizer first waits for attached ephemeral-access policies to be removed. EphemeralAccessCleanupPending reports that guarded wait.
Inspect attached EphemeralAccessPolicy and EphemeralAccessRequest resources, allow their revocation/finalizers to complete, and then retry deletion. Do not strip finalizers until you have verified that temporary PostgreSQL memberships were revoked; forced Kubernetes deletion can otherwise leave access behind.
Force a fresh reconcile
After fixing the cause, you can wait for spec.interval, edit the policy, or request an immediate reconcile with a unique timestamp:
kubectl annotate pgr "$POLICY" --namespace "$NAMESPACE" \
reconcile.pgroles.io/requestedAt="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--overwrite
Then re-read conditions rather than relying only on Events. Several reasons, including InvalidDatabaseTarget, MissingDatabaseObject, InvalidConnectionParams, and UnsatisfiableWildcardGrant, do not emit dedicated Events.