Managing Drives

Prerequisites

Working DirectPV plugin.

To install the plugin, refer to the plugin installation guide.
Working DirectPV CSI driver in Kubernetes.

To install the driver, refer to the driver installation guide.

Drives

Add drives

DirectPV must have access to drives to provision volumes. This involves a two step process:

Run the discover command.

The discover command probes for eligible drives from DirectPV nodes and stores drive information in a YAML file. Examine the generated YAML file and set the select field to yes for each drive you want to add. If you do not want to add a particular drive, set its select value to no.

DirectPV sets the select field to yes for all discovered drives by default. You must manually deselect any discovered drives that DirectPV should not format and use. Review the generated YAML file carefully.

Below is an example of the discover command:

# Probe and save drive information to drives.yaml file.
$ kubectl directpv discover

 Discovered node 'master' ✔
 Discovered node 'node1' ✔

┌─────────────────────┬────────┬───────┬─────────┬────────────┬──────┬───────────┬─────────────┐
│ ID                  │ NODE   │ DRIVE │ SIZE    │ FILESYSTEM │ MAKE │ AVAILABLE │ DESCRIPTION │
├─────────────────────┼────────┼───────┼─────────┼────────────┼──────┼───────────┼─────────────┤
│ 252:16$ud8mwCjPT... │ master │ vdb   │ 512 MiB │ -          │ -    │ YES       │ -           │
│ 252:16$gGz4UIuBj... │ node1  │ vdb   │ 512 MiB │ -          │ -    │ YES       │ -           │
└─────────────────────┴────────┴───────┴─────────┴────────────┴──────┴───────────┴─────────────┘

Generated 'drives.yaml' successfully.

# Show generated drives.yaml file.
$ cat drives.yaml
version: v1
nodes:
    - name: master
      drives:
        - id: 252:16$ud8mwCjPTH8147TysmiQ2GGLpffqUht6bz7NtHqReJo=
          name: vdb
          size: 536870912
          make: ""
          select: "yes"
    - name: node1
      drives:
        - id: 252:16$gGz4UIuBjQlO1KibOv7bZ+kEDk3UCeBneN/UJdqdQl4=
          name: vdb
          size: 536870912
          make: ""
          select: "yes"

Run the init command.

The init command creates a request to add the selected drives in the YAML file generated using the discover command in the previous step.

This process wipes out all data on the selected drives. Incorrect drive selection will result in permanent data loss.

Before running this command, modify the YAML file to mark any undesired drives as no for the select field.

Below is an example of init command:

$ kubectl directpv init drives.yaml --dangerous


 ███████████████████████████████████████████████████████████████████████████ 100%

 Processed initialization request 'c24e22f5-d582-49ba-a883-2ce56909904e' for node 'master' ✔
 Processed initialization request '7e38a453-88ed-412c-b146-03eef37b23bf' for node 'node1' ✔

┌──────────────────────────────────────┬────────┬───────┬─────────┐
│ REQUEST_ID                           │ NODE   │ DRIVE │ MESSAGE │
├──────────────────────────────────────┼────────┼───────┼─────────┤
│ c24e22f5-d582-49ba-a883-2ce56909904e │ master │ vdb   │ Success │
│ 7e38a453-88ed-412c-b146-03eef37b23bf │ node1  │ vdb   │ Success │
└──────────────────────────────────────┴────────┴───────┴─────────┘

Refer to the discover command and the init command for more details and optional parameters.

List drives

To get information about drives from DirectPV, run the list drives command.

$ kubectl directpv list drives
┌────────┬──────┬──────┬─────────┬─────────┬─────────┬────────┐
│ NODE   │ NAME │ MAKE │ SIZE    │ FREE    │ VOLUMES │ STATUS │
├────────┼──────┼──────┼─────────┼─────────┼─────────┼────────┤
│ master │ vdb  │ -    │ 512 MiB │ 506 MiB │ -       │ Ready  │
│ node1  │ vdb  │ -    │ 512 MiB │ 506 MiB │ -       │ Ready  │
└────────┴──────┴──────┴─────────┴─────────┴─────────┴────────┘

Refer to the list drives command for more information.

Label drives

Drives are labeled to set custom tagging which can be used in volume provisioning.

# Set label 'tier' key to 'hot' value on all drives.
$ kubectl directpv label drives tier=hot --all

# Remove label 'tier' from all drives.
$ kubectl directpv label drives tier- --all

Once set, use labels to schedule drives.

Refer to the label drives command for more information.

Replace drive

Replace a faulty drive with a new drive on a same node. In this process, all volumes in the faulty drive are moved to the new drive. Currently, DirectPV does not support moving data on the volume to the new drive. Use the replace.sh script to perform drive replacement.

Below is an example:

# Replace 'sdd' drive by 'sdf' drive on 'node1' node
$ replace.sh sdd sdf node1

The faulty drive stays in DirectPV after the replacement. It ends up with no volumes on it, and it stays unschedulable, so it takes no new volumes. Run the remove drives command on that drive to take it out of DirectPV.

Remove drives

Drives that do not contain any volumes can be removed.

# Remove drive 'vdb' from 'node1' node
$ kubectl directpv remove --drives=vdb --nodes=node1

Refer to the remove command for more information.

Repair Drives

Data Loss

THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS.

DirectPV supports using the xfs_repair utility to attempt to repair managed drives that report XFS filesystem errors, faults, or corruption. This command has no guarantee of success or complete recovery.

The repair command creates one-time Kubernetes Job with the pod name as repair-<DRIVE-ID>. Kubernetes automatically removes this job five minutes after completion.

Progress and status of the drive repair can be viewed using kubectl logs command.

Before beginning a repair, you must first suspend the drive.

Use the repair.sh script to repair a faulty drive.

Repair a drive after its disk returns under a new name

A hardware or enclosure event can take a disk offline and bring it back under a new device name, such as sdf instead of sdc. The drive keeps a stale mount of the old device, I/O on its volumes fails with Input/output error, and DirectPV marks the drive Error. If the returned disk still holds the drive’s filesystem, repair the drive in place with these steps. Do not initialize or replace the disk.

Replay the log first

The repair command does not replay the XFS log; step 7 does, by mounting the drive. Do not run repair before step 8, and do not run repair --force or xfs_repair -L: they destroy the log, which can corrupt the filesystem.

Find the ID and the node of the drive in Error. A plain list drives shows only Ready drives, so add --all:
```
kubectl directpv list drives --all
```
The drive ID is also the UUID of the drive’s filesystem.
On the node, find the new device name and the device number of the stale mount. Run these commands as root on the node. On OpenShift, open a shell with oc debug node/NODE_NAME, then run chroot /host.
```
ls -l /dev/disk/by-uuid/DRIVE_ID
grep DRIVE_ID /proc/1/mountinfo
```
The first command shows the new device name. If the link does not exist, stop: these steps do not apply. The second command prints one line, the stale mount at /var/lib/directpv/mnt/DRIVE_ID. The third field of that line is the device number of the old device, MAJOR:MINOR.

Suspend the drive:

kubectl directpv suspend drives DRIVE_ID --dangerous

Delete the pods that use the drive’s volumes, so they restart on an empty read-only directory. The PODNAME and PODNAMESPACE columns of the first command list them:
```
kubectl directpv list volumes --drive-id=DRIVE_ID
kubectl -n PODNAMESPACE delete pod PODNAME
```
On OpenShift with SELinux, a pod fails to start with relabel failed unless its security context sets seLinuxOptions.type: spc_t.
Check that the stale mount is the only mount of the old device:
```
grep ' MAJOR:MINOR ' /proc/1/mountinfo
```
Continue only when this command prints exactly one line, the stale mount. More lines mean that a pod still uses the old device: wait and check again.
Remove the stale mount in the host mount namespace:
```
nsenter -t 1 -m -- umount /var/lib/directpv/mnt/DRIVE_ID
```
In an oc debug shell, a plain umount returns 0 but leaves the host mount in place. Run the grep from step 5 again. It must print nothing.
Restart the node-server pod on the node. List the DirectPV pods on the node, then delete the pod whose name starts with node-server-:
```
kubectl -n directpv get pods -o wide --field-selector spec.nodeName=NODE_NAME
kubectl -n directpv delete pod NODE_SERVER_POD
```
DirectPV mounts the drive from its new device name, and XFS replays the log. The drive stays in Error.
When the node-server pod on the node is Running, check the filesystem:
```
kubectl directpv repair DRIVE_ID --dry-run
```
When the Job finishes, check the drive status with kubectl directpv list drives --all. If the status is Ready,Suspended, the filesystem is clean: go to step 9. If it stays Repairing,Suspended, the Job failed. Read its log:
```
kubectl -n directpv logs job/repair-DRIVE_ID
```
If the log shows filesystem errors, delete the Job, because a new repair does nothing while the Job exists, and run the repair:
```
kubectl -n directpv delete job repair-DRIVE_ID --ignore-not-found
kubectl directpv repair DRIVE_ID
```
Resume the drive, then delete the pods from step 4 again so they use the drive:
```
kubectl directpv resume drives DRIVE_ID
```

If repair ran before the log was replayed and the drive stays in Repairing, restarting the node-server does not mount it. Do steps 1 to 6. If the grep in step 2 prints no line, repair already removed the stale mount: skip steps 5 and 6. Then replay the log on the node by mounting the new device once:

mkdir -p /var/tmp/replay
nsenter -t 1 -m -- mount /dev/NEW_DEVICE /var/tmp/replay
nsenter -t 1 -m -- umount /var/tmp/replay

Delete the repair Job if it still exists, and continue at step 8.

Suspend drives

Data Loss

THIS IS DANGEROUS OPERATION WHICH LEADS TO DATA LOSS.

By Kubernetes design, a StatefulSet workload is active only if all of its pods are in running state. A faulty drive prevents the StatefulSet from starting up.

DirectPV provides a workaround to suspend failed drives which mounts the respective volumes on empty /var/lib/directpv/tmp directory with read-only access. This can be done by executing the suspend drives command.

kubectl directpv suspend drives af3b8b4c-73b4-4a74-84b7-1ec30492a6f0 --dangerous

Once fixed, return to normal drive functions by resuming the drive. Upon resuming, the corresponding volumes return to using the respective allocated drives. This can be done by using the resume drives command.

kubectl directpv resume drives af3b8b4c-73b4-4a74-84b7-1ec30492a6f0

Managing Drives

On this page