Ceph

EdgeFS Rook integrated CSI driver, provisioner and attacher

Container Storage Interface (CSI) driver, provisioner, and attacher for EdgeFS Scale-Out NFS

Overview

EdgeFS CSI plugins implements interface between CSI enabled Container Orchestrator (CO) and EdgeFS local cluster site. It allows dynamic and static provisioning of EdgeFS NFS exports or iSCSI LUNs, and attaching them to stateful application workloads. With EdgeFS NFS implementation, I/O load can be spread-out across multiple PODs, thus eliminating networking I/O bottlenecks of classic single-node NFS. Current implementation of EdgeFS CSI plugin was tested in Kubernetes environment (requires Kubernetes 1.11+), however the code is Kubernetes version agnostic and should be able to run with any CSI enabled CO.

Deployment

Ensure that Rook EdgeFS cluster up and running.

Configure new NFS service (lets say named as “nfs01”) via efscli and create Rook EdgeFS NFS resouce.

Configure CSI driver options and cluster discovery using kubernetes secret json file as an example.

Secret file configuration options example:

{
	"k8sEdgefsNamespace": "rook-edgefs",
	"cluster": "cltest",
	"tenant": "test",
	"serviceFilter": "nfs01",
	"username": "admin",
	"password": "admin"
}

By using k8sEdgefsNamespace parameter, driver is capable of detecting ClusterIPs and Endpoint IPs to provision and attach volumes.

Check configuration options and create kubernetes secret for NexentaEdge CSI plugin

cd cluster/examples/kubernetes/edgefs/csi/secret
kubectl create secret generic rook-edgefs-cluster --from-file=./cluster-config.json

After secret is created successfully, deploy EdgeFS CSI plugin, provisioner and attacher using the following command

cd cluster/examples/kubernetes/edgefs/csi
kubectl apply -f .

Note that for Kubernetes versions >= v1.12.1 CSI architecutre introduced kind=CSIDriver CRD. To use earlier version change pwd to ‘k8s-prior-12.1’ subdirectory.

There should be number of EdgeFS CSI plugin PODs available running as a DaemonSet

...
NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
default       csi-attacher-nedgeplugin-0              2/2       Running   0          18s
default       csi-provisioner-nedgeplugin-0           2/2       Running   0          18s
default       edgefs-csi-plugin-7s6wc                 2/2       Running   0          19s
...

At this point configuration is all ready and available for consumption by appliations.

Pre-provisioned volumes (NFS)

This method allows to use already created exports in EdgeFS services. This method keeps exports provisioned after application PODs terminated. Read more on how to create PersistentVolume specification for pre-provisioned volumes:

link to Pre-provisioned volumes manifest specification

To test creation and mount pre-provisioned volume to pod execute example

Note:

Make sure that volumeHandle: clus1/ten1/buk1 in nginx.yaml already exist on EdgeFS cluster

Examples:

cd cluster/examples/kubernetes/edgefs/csi/examples
kubectl apply -f ./pre-provisioned-nginx.yaml #one pod with pre-provisioned volume
kubectl apply -f ./deployment.yaml            # 10 pods deployment shares one EdgeFS bucket

Dynamically provisioned volumes (NFS)

To setup the system for dynamic provisioning, administrator needs to setup a StorageClass pointing to the CSI driver’s external-provisioner and specifying any parameters required by the driver

link to dynamically provisioned volumes specification

Note:

For dynamically provisioned volumes kubernetes will generate volume name automatically (for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2) Additional creation options should be passed as parameters in storage class definition i.e :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-sc-nedgeplugin
provisioner: edgefs-csi-plugin
parameters:
  tenant: ten1
  encryption: true

Options:

Note:

Options are case sensitive and should be in lower case

Example:

cd cluster/examples/kubernetes/edgefs/csi/examples
kubectl apply -f ./dynamic-nginx.yaml

Troubleshooting and log collection

For details about other configuration and deployment of EdgeFS CSI plugin, see Wiki pages:

• Quick Start Guide

Please submit an issue at: Issues

Tips

In case any problems using EdgeFS CSI driver

1. Check CSI plugin pods state

   kubectl describe pod edgefs-csi-plugin-xxxxx
   

2. Check provisioned pods state

   kubectl describe pods nginx
   

3. Check CSI plugin logs

   kubectl logs csi-attacher-nedgeplugin-0 -c nfs
   kubectl logs csi-provisioner-nedgeplugin-0 -c nfs
   kubectl logs edgefs-csi-plugin-j8ljf -c nfs