Troubleshooting Common Problems

This topic provides information that can help troubleshoot problems you may encounter using Postgres for Kubernetes.

Monitor Deployment Progress

Use watch kubectl get all to monitor the progress of the Postgres operator deployment. The deployment is complete when the postgres operator pod is in the Running state. For example:

$ watch kubectl get all
NAME                                     READY   STATUS    RESTARTS   AGE
pod/postgres-operator-567dbc67b9-nrq5t   1/1     Running   0          57s
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d4h
NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/postgres-operator   1/1     1            1           57s
NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/postgres-operator-567dbc67b9   1         1         1       57s

Viewing Postgres Operator Logs

Check the logs of the operator to ensure that it is running properly.

$ kubectl logs -l app=postgres-operator
2019-08-05T17:24:16.182Z    INFO    controller-runtime.controller   Starting EventSource{"controller": "postgres", "source": "kind source: /, Kind="}
2019-08-05T17:24:16.182Z    INFO    setup   starting manager
2019-08-05T17:24:16.285Z    INFO    controller-runtime.controller   Starting Controller {"controller": "postgres"}
2019-08-05T17:24:16.386Z    INFO    controller-runtime.controller
Starting workers    {"controller": "postgres", "worker count": 1}

List All Postgres Instances in the Cluster

When you create Postgres instances, each instance is created in its own namespace. To see all Postgres instances in the cluster, add the -all-namespaces option to the kubectl get command.

$ kubectl get postgres --all-namespaces
NAMESPACE   NAME           STATUS    AGE
default     my-postgres    Running   19d
default     my-postgres2   Running   15d
test        my-postgres    Failed    15d
test        my-postgres3   Failed    15d

Find the Versions of the Deployed Postgres Operator and Instances

To find the currently deployed version of the Postgres operator, use the helm command:

$ helm ls
NAME                NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                                   APP VERSION
postgres-operator   default     1           2020-06-30 13:26:00.769535 -0500 CDT    deployed    postgres-operator-v0.4.0.90.g0c7bec9    v0.4.0.90.g0c7bec9

The version is in the chart name and the APP VERSION column.

To find the version of a Postgres instance, use the kubectl command to describe the instance’s pod.

$ kubectl get pods
$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
my-postgres-0                        1/1     Running   0          9s
postgres-operator-85f777b9db-wbj9b   1/1     Running   0          4m15s
$ kubectl describe pod/my-postgres-0
Name:           my-postgres-0
Namespace:      default
Priority:       0
Node:           minikube/192.168.64.32
Start Time:     Tue, 30 Jun 2020 14:10:38 -0500
Labels:         app=postgres
                controller-revision-hash=my-postgres-55b7dd9b4c
                postgres-instance=my-postgres
                statefulset.kubernetes.io/pod-name=my-postgres-0
Annotations:    <none>
Status:         Running
IP:             172.17.0.8
Controlled By:  StatefulSet/my-postgres
Containers:
  pg-container:
    Container ID:  docker://6c651d690a6fdb6d1c0d3644ad8225037d31da1c33fd3f88f1625bdfd45cea3a
    Image:         postgres-instance:v0.4.0.90.g0c7bec9
    Image ID:      docker://sha256:00359ca344dd96eb05f2bd430430c97a6d46a40996c395fca44c209cb954a6e7
    Port:          5432/TCP
    Host Port:     0/TCP

The Tanzu Postgres version can be found in the image name of the pg-container entry.

Cannot Reinstall Operator After Deleting

It is important to delete running Postgres instances before you delete the Postgres operator.

If you accidentally delete the Postgres operator with one or more existing Postgres instances, and then try to reinstall the operator, you will get an error message like the following:

$ helm install postgres-operator operator/
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: postgres.sql.tanzu.vmware.com

Follow these steps to work around this problem.

  1. Patch the Postgres operator CRD to remove the finalizer.

    $ kubectl patch crd postgres.sql.tanzu.vmware.com -p '{"metadata":{"finalizers":[]}}' --type=merge
    
    customresourcedefinition.apiextensions.k8s.io/postgres.sql.tanzu.vmware.com patched
    
  2. Reinstall the Postgres operator.

    $ helm install postgres-operator operator/
    

    When the operator reaches the Running state, it will terminate all of the running Postgres instances because it is deleting the CRDs.

    $ kubectl get pods
    
    NAME                                     READY   STATUS        RESTARTS   AGE
    pod/my-postgres-0                        1/1     Terminating   0          2m11s
    pod/my-postgres-2-0                      0/1     Terminating   0          2m10s
    pod/postgres-operator-6c46559444-2zlg5   1/1     Running       0          10s
    
  3. Redeploy the Postgres instances.

    $ kubectl create -f pg-instance-example.yaml
    

When the Postgres instances are recreated with the same names, the persistent volumes will be reattached.