Troubleshooting Common Problems

This topic provides information that can help troubleshoot problems you may encounter using Postgres for Kubernetes.

Monitor Deployment Progress

Use watch kubectl get all to monitor the progress of the Postgres operator deployment. The deployment is complete when the postgres operator pod is in the Running state. For example:

$ watch kubectl get all
NAME                                     READY   STATUS    RESTARTS   AGE
pod/postgres-operator-567dbc67b9-nrq5t   1/1     Running   0          57s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP    <none>        443/TCP   2d4h
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/postgres-operator   1/1     1            1           57s
NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/postgres-operator-567dbc67b9   1         1         1       57s

Viewing Postgres Operator Logs

Check the logs of the operator to ensure that it is running properly.

$ kubectl logs -l app=postgres-operator
2019-08-05T17:24:16.182Z    INFO    controller-runtime.controller   Starting EventSource{"controller": "postgres", "source": "kind source: /, Kind="}
2019-08-05T17:24:16.182Z    INFO    setup   starting manager
2019-08-05T17:24:16.285Z    INFO    controller-runtime.controller   Starting Controller {"controller": "postgres"}
2019-08-05T17:24:16.386Z    INFO    controller-runtime.controller
Starting workers    {"controller": "postgres", "worker count": 1}

List All Postgres Instances in the Cluster

When you create Postgres instances, each instance is created in its own namespace. To see all Postgres instances in the cluster, add the -all-namespaces option to the kubectl get command.

$ kubectl get postgres --all-namespaces
default     my-postgres    Running   19d
default     my-postgres2   Running   15d
test        my-postgres    Failed    15d
test        my-postgres3   Failed    15d

Find the Versions of the Deployed Postgres Operator and Instances

To find the currently deployed version of the Postgres operator, use the helm command:

$ helm ls
NAME                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                        APP VERSION
postgres-operator   default     1           2020-06-30 13:26:00.769535 -0500 CDT    deployed    postgres-operator-v1.0.0     v1.0.0

The version is in the chart name and the APP VERSION column.

To find the version of a Postgres instance, use the kubectl command to describe the instance’s pod.

$ kubectl get pods
$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
my-postgres-0                        1/1     Running   0          9s
postgres-operator-85f777b9db-wbj9b   1/1     Running   0          4m15s
$ kubectl describe pod/my-postgres-0
Name:           my-postgres-0
Namespace:      default
Priority:       0
Node:           minikube/
Start Time:     Tue, 30 Jun 2020 14:10:38 -0500
Labels:         app=postgres
Annotations:    <none>
Status:         Running
Controlled By:  StatefulSet/my-postgres
    Container ID:  docker://6c651d690a6fdb6d1c0d3644ad8225037d31da1c33fd3f88f1625bdfd45cea3a
    Image:         postgres-instance:v1.0.0
    Image ID:      docker://sha256:00359ca344dd96eb05f2bd430430c97a6d46a40996c395fca44c209cb954a6e7
    Port:          5432/TCP
    Host Port:     0/TCP

The Tanzu Postgres version can be found in the image name of the pg-container entry.

Cannot Reinstall Operator After Deleting

It is important to delete running Postgres instances before you delete the Postgres operator.

If you accidentally delete the Postgres operator with one or more existing Postgres instances, and then try to reinstall the operator, you will get an error message like the following:

$ helm install postgres-operator operator/
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name:

Follow these steps to work around this problem.

  1. Patch the Postgres operator CRD to remove the finalizer.

    $ kubectl patch crd -p '{"metadata":{"finalizers":[]}}' --type=merge patched
  2. Reinstall the Postgres operator.

    $ helm install postgres-operator operator/

    When the operator reaches the Running state, it will terminate all of the running Postgres instances because it is deleting the CRDs.

    $ kubectl get pods
    NAME                                     READY   STATUS        RESTARTS   AGE
    pod/my-postgres-0                        1/1     Terminating   0          2m11s
    pod/my-postgres-2-0                      0/1     Terminating   0          2m10s
    pod/postgres-operator-6c46559444-2zlg5   1/1     Running       0          10s
  3. Redeploy the Postgres instances.

    $ kubectl create -f pg-instance-example.yaml

When the Postgres instances are recreated with the same names, the persistent volumes will be reattached.

Cannot Reduce Instance Data Size After Deployment

When deploying an instance using a specific storage size in the instance yaml deployment file, you cannot reduce the instance data storage size at a later stage. For example, after creating an instance and setting the storage size to 100M:

$ kubectl exec create -f pg-instance-example.yaml

verify the storage size using a command similar to:

$ kubectl get -o jsonpath='{.spec.storageSize}'

If you later patch the instance to decrease the storage size from 100M to 2M:

$ kubectl patch --type merge -p '{"spec":{"storageSize": "2M"}}'

the operation returns an error similar to:

Error from server (storageSize cannot be reduced after the instance is created. No changes have been made to the running instance.): error when applying patch:
Resource: ", Resource=postgres", GroupVersionKind: ", Kind=Postgres"

To reduce the instance data size, create a new instance and migrate the source data over. Ensure that the source data fits in the reduced data size allocation of the newly created instance.