Skip to content

CephX Key Rotation

Attention

This feature is experimental.

Rook is able to rotate CephX authentication keys used by Ceph daemons and clients.

For this experimental feature, some caveats should be noted:

  • Only Ceph versions v19.2.3 and higher have the capabilities Rook requires for key rotation.
  • Ceph Monitor (mon) keys cannot be rotated in Ceph v19 due to Ceph architecture limitations.

Overview

CephX keys can be rotated when desired on a one-off basis. To provide this capability, Rook utilizes an approximation of Kubernetes's resource generation. A one-time key rotation is initiated by specifying KeyGeneration as the desired policy (the default policy is Disabled) and also specify a key generation higher than the current generation.

CephX keys can be divided into two categories, below.

Daemon keys

Daemon keys are used internally within a Ceph cluster, and their rotation does not affect CSI volumes or connections to a Ceph cluster from outside.

Daemon key rotation is configured via the CephCluster spec.security.cephx.daemon config. This will also rotate daemon keys for any CephFilesystem MDSes and CephObjectStore RGWs.

Rotation requires most Ceph daemons to restart, so this operation is best done at the same time the CephCluster spec.cephVersion.image is updated -- when daemons will normally need to restart.

"Non-daemon" keys

Non-daemon keys may reasonably require user action beyond Rook API controls.

Because these keys affect non-daemon connections, Rook allows users to initiate rotation independently during their desired maintenance window.

Below is a list of non-daemon keys along with the controlling config.

  • CephCluster CSI keys are controlled via CephCluster spec.security.cephx.csi
    • Rotated CSI keys only take effect for new PVCs. For CSI alone, Rook is able to create new keys while also keeping a number of prior key generations active. This is configured using the keepPriorKeyCountMax option.
  • The CephCluster RBD mirror peer key is controlled via CephCluster spec.security.cephx.rbdMirrorPeer
    • Each CephBlockPool that has mirroring configured will have a peerToken status that references the CephCluster RBD mirror peer key
  • Each CephClient key is controlled via its own spec.security.cephx

Initiating key rotation

To begin experimenting with key rotation, check out CephX config options on Rook CRs.

Example

Most key rotations are initiated from the CephCluster. An example spec that will rotate all CephX keys for most new or upgraded Rook clusters is shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: my-cluster
  namespace: rook-ceph # namespace:cluster
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.3
  security:
    cephx:
      daemon:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
      csi:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
        keepPriorKeyCountMax: 1  # keep one prior key also
      rbdMirrorPeer:
        keyRotationPolicy: KeyGeneration
        keyGeneration: 2
  # ...

Once rotation is complete, CephCluster status should look something like below. Each CephX key type managed for the cluster is listed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
status:
  # ...
  cephx:
    cephExporter:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2
    crashCollector:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2
    csi:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2
      priorKeyCount: 1
    mgr:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2
    mon: {}  # reminder: mon key rotation is unsupported currently
    osd:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2
    rbdMirrorPeer:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2

Additionally, any CephFilesystem or CephObjectStore will show the status of rotation for their daemons:

1
2
3
4
5
6
status:
  # ...
  cephx:
    daemon:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2

If mirroring is enabled on a CephBlockPool, the following status will mirror the CephCluster's rbdMirrorPeer status:

1
2
3
4
5
6
status:
  # ...
  cephx:
    peerToken:
      keyCephVersion: 19.2.3-0
      keyGeneration: 2