Ceph
PLEASE NOTE: This document applies to v1.1 version and not to the latest stable release v1.9
Common Issues
To help troubleshoot your Rook clusters, here are some tips on what information will help solve the issues you might be seeing. If after trying the suggestions found on this page and the problem is not resolved, the Rook team is very happy to help you troubleshoot the issues in their Slack channel. Once you have registered for the Rook Slack, proceed to the General channel to ask for assistance.
Ceph
For common issues specific to Ceph, see the Ceph Common Issues page.
Troubleshooting Techniques
Kubernetes status and logs are the the main resources needed to investigate issues in any Rook cluster.
Kubernetes Tools
Kubernetes status is the first line of investigating when something goes wrong with the cluster. Here are a few artifacts that are helpful to gather:
- Rook pod status:
kubectl get pod -n <cluster-namespace> -o wide- e.g.,
kubectl get pod -n rook-ceph -o wide
- e.g.,
- Logs for Rook pods
- Logs for the operator:
kubectl logs -n <cluster-namespace> -l app=<storage-backend-operator>- e.g.,
kubectl logs -n rook-ceph -l app=rook-ceph-operator
- e.g.,
- Logs for a specific pod:
kubectl logs -n <cluster-namespace> <pod-name>, or a pod using a label such as mon1:kubectl logs -n <cluster-namespace> -l <label-matcher>- e.g.,
kubectl logs -n rook-ceph -l mon=a
- e.g.,
- Logs on a specific node to find why a PVC is failing to mount:
- Connect to the node, then get kubelet logs (if your distro is using systemd):
journalctl -u kubelet
- Connect to the node, then get kubelet logs (if your distro is using systemd):
- Pods with multiple containers
- For all containers, in order:
kubectl -n <cluster-namespace> logs <pod-name> --all-containers - For a single container:
kubectl -n <cluster-namespace> logs <pod-name> -c <container-name>
- For all containers, in order:
- Logs for pods which are no longer running:
kubectl -n <cluster-namespace> logs --previous <pod-name>
- Logs for the operator:
Some pods have specialized init containers, so you may need to look at logs for different containers within the pod.
kubectl -n <namespace> logs <pod-name> -c <container-name>- Other Rook artifacts:
kubectl -n <cluster-namespace> get all