Two-Node Cluster (experimental)¶

This guide provides installation instructions for setting up a two-node Kubernetes cluster with Rook using DRBD (Distributed Replicated Block Device) for storage replication for one of the Ceph mons.

For architecture details and design rationale, see the Two-Node Fencing Design Document.

Prerequisites¶

Before starting the installation:

• 2-node Kubernetes cluster with fencing configured (e.g., OpenShift with DualReplica topology)
• Raw block devices on each node (SSD-class, same size, no filesystem)

Installation Steps¶

Step 1: Install DRBD Kernel Modules¶

DRBD kernel modules must be installed and loaded on both nodes before running the setup script.

Follow the DRBD installation documentation for your distribution and install drbd & drbd_transport_tcp:

• DRBD 9.0 Installation Guide - Official installation guide
• DRBD User's Guide - Complete documentation
• DRBD GitHub Repository - Source code and build instructions

Verify Installation¶

On both nodes, verify DRBD modules are loaded:

1
2
3
4
5
6
7
8
9

# Check loaded modules
lsmod | grep drbd

# Expected output:
# drbd_transport_tcp     16384  0
# drbd                  614400  1 drbd_transport_tcp

# Check DRBD version
drbdadm --version

Step 2: Identify Block Devices¶

Use the setup script to list available block devices on both nodes:

1

./deploy/examples/drbd-setup-experimental.sh -l

Example output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

=== Block devices (node0=node-0, node1=node-1) ===
Use the PATH column (e.g. -d /dev/sdb or -d0 / -d1 per-node paths).

--- node-0 ---
NAME   PATH      SIZE    ROTA TYPE FSTYPE
sdb    /dev/sdb  10G    0    disk        <-- Use this

--- node-1 ---
NAME   PATH      SIZE    ROTA TYPE FSTYPE
sdb    /dev/sdb  10G    0    disk        <-- Use this

Same path on both nodes: -d <path>
Different paths (same size): -d0 <path0> -d1 <path1>

Requirements for backing devices:

• Must be SSD/NVMe (ROTA=0, non-rotational)
• Can be a partition of the disk
• Same size on both nodes
• No existing filesystem (FSTYPE empty)
• Writable (not read-only)
• Volume or partition is not larger than 10Gi

Step 3: Run DRBD Setup Script¶

The setup script configures DRBD, performs initial sync, and creates the auto-start DaemonSet.

Same Device Path on Both Nodes¶

1

./deploy/examples/drbd-setup-experimental.sh -d /dev/sdb

Different Device Paths Per Node¶

1

./deploy/examples/drbd-setup-experimental.sh -d0 /dev/sdb -d1 /dev/sdc

Available options:

./deploy/examples/drbd-setup-experimental.sh -h

Step 4: Verify DRBD Setup¶

Check auto-start DaemonSet:

1
2

kubectl get daemonset drbd-autostart -n rook-ceph
kubectl get pods -n rook-ceph -l app=drbd-autostart

Check for the drbd-configure configmap

1

kubectl get cm -n rook-ceph drbd-configure -o yaml

Step 5: Deploy Rook CephCluster¶

Create the CephCluster CR with floating monitor configuration:

Apply the cluster:

1

kubectl apply -f deploy/examples/cluster-tnf.yaml

Step 6: Verify Ceph Cluster¶

Check monitor pods:

1

kubectl get pods -n rook-ceph -l app=rook-ceph-mon

Expected: 3 monitor pods (mon-a, mon-b, mon-c)

Check Ceph status:

1

kubectl exec -n rook-ceph deploy/rook-ceph-tools -- ceph status

Expected output:

1
2
3
4
5
6
7

    cluster:
        id:     <cluster-id>
        health: HEALTH_OK

    services:
        mon: 3 daemons, quorum a,b,c (age ...)
        ...

Configuration Summary¶

The setup creates:

• DRBD resource (r0) with Protocol C synchronous replication
• DRBD device (/dev/drbd0) with XFS filesystem
• Auto-start DaemonSet (drbd-autostart) in rook-ceph namespace
• Configuration ConfigMap (drbd-configure) in rook-ceph namespace
• Two pinned monitors (one per node) with local storage
• One floating monitor (mon-c) backed by DRBD

Two-Node Cluster Recommendations¶

Configure pools with replica-2, pool-tnf.yaml

Troubleshooting¶

DRBD Modules Not Loaded¶

Check:

1

lsmod | grep drbd

Fix: Install DRBD kernel modules (see Step 1)

DRBD Sync Not Progressing¶

Check network connectivity:

1

nc -zv <peer-node-ip> 7794

Check DRBD status:

1

drbdadm status r0

Floating Monitor Not Starting¶

Check DRBD is Secondary on both nodes:

1
2

drbdadm role r0
# Expected: Secondary/Secondary

Check ConfigMap:

1

kubectl get configmap drbd-configure -n rook-ceph -o yaml

Re-run Setup Script¶

The script is idempotent and safe to re-run:

1

./deploy/examples/drbd-setup-experimental.sh -d /dev/sdb