Volumes

Resources#

• Persistent Volumes (PV and PVC)
• Container Storage Interface (CSI)

Storage Classes#

The provided storage classes are used to define the location and the kind of volume Kubernetes will create using CSI.

[location]-[kind]-[disk]-[relication](-[filesystem])

• location: “eu-west-fr-gra” means Gravelines in France
• kind:
  • block: standard logical volume or
  • files: multiaccess shared file system
• disk:
  • hdd: standard performance
  • nvme: high performance
• replication: “ec” means the data Ceph pool is using Erasure Coding replication. The associated metadata pools is using replication.
• filesystem: for block volumes, you can choose between XFS or Ext4 filesystems

French storage classes:#

• eu-west-fr-gra-block-hdd-ec-ext4
• eu-west-fr-gra-block-hdd-ec-xfs
• eu-west-fr-gra-block-nvme-ec-ext4
• eu-west-fr-gra-block-nvme-ec-xfs
• eu-west-fr-gra-files-hdd-ec
• eu-west-fr-gra-files-nvme-ec

Tools#

Migrate PVC#

The following script helps to migrate or copy the source PVC to a new PVC using rsync.

If needed, this script allows to migrate the data to another the storage class

Requirements#

• kubectl
• jq

Migrate script#

The following bash script process is:

1. creates the target PVC manifest based on the source PVC configuration and the passed arguments
2. applies the target PVC manifest
3. scale down to 0 the given deployment or statefulset
4. updates and applies the job template
5. opens to edit the deployment or the statefulset to update the claim name
6. scale up to 1 the deployment or the statefulset

Each above command requires a user confirmation (Enter) to process the next step. This helps to check if the previous step has been completed.

When the resource kind is a statefulset, it’s often required to recreate it with a different name to match the new PVC name.

• migrate.sh content:

#!/bin/bash
# Arguments:
# $1 namespace
# $2 pvc source
# $3 pvc target
# $4 pvc target class
# $5 deployment|sts name
# $6 (default: deployment) deployment|sts
# $7 (default: pvc source) access
# $8 (default: pvc source) size
#
# Useful:
# - kubectl -n namespace get pods -o=json | jq -c '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName: .spec |  select( has ("volumes") ).volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'
# - watch kubectl get deployment,pods,jobs,pvc -o wide -n namespace
# - kubectl -n namespace delete jobs.batch --all
#
# Example:
# ./migrate.sh [namespace] [pvc-source] [pvc-target] [target-storage-class] [deployment-name] deployment ([new target size])
# ./migrate.sh [namespace] [pvc-source] [pvc-target] [target-storage-class] [statefulset-name] sts

useConfirm=true
confirm() {
   [ "$useConfirm" = true ] && read -p "(enter to continue)"
}

if [ -n "$6" ]; then
  kind=$6
else
  kind="deployment"
fi
echo ""
echo "migrate: $1:$kind/$5 $2 > $3:$4"
echo ""

kc='kubectl -v=3 -n '"$1"''

echo "= create pvc $3 ="
pvc=$($kc get pvc $2 -o json \
  | jq --arg name "$3" '.metadata.name = $name' \
  | jq --arg scn "$4" '.spec.storageClassName = $scn' \
  | jq '.metadata |= del(.managedFields)' \
  | jq '.metadata |= del(.annotations)' \
  | jq '.metadata |= del(.creationTimestamp)' \
  | jq '.metadata |= del(.finalizers)' \
  | jq '.metadata |= del(.resourceVersion)' \
  | jq '.metadata |= del(.selfLink)' \
  | jq '.metadata |= del(.uid)' \
  | jq '.spec |= del(.volumeMode)' \
  | jq '.spec |= del(.volumeName)' \
  | jq '. |= del(.status)')
if [ -n "$7" ]; then
  pvc=$(echo "$pvc" \
    | jq --arg access "$7" '.spec.accessModes = [$access]')
fi
if [ -n "$8" ]; then
  pvc=$(echo "$pvc" \
    | jq --arg size "$8" '.spec.resources.requests.storage = $size')
fi
echo "$pvc"

echo ""
echo "= apply pvc $3 ="
confirm
echo "$pvc" | $kc apply --wait -f -

echo ""
echo "= scale down $kind $5 ="
confirm
$kc scale $kind/$5 --replicas=0

echo ""
echo "= rsync pvcs data ="
job=$(cat /root/platform/rook/tools/migrate/migrate-job.yaml \
  | sed 's/DEPLOYMENT/'"$3-$2"'/g' \
  | sed 's/SOURCE_PVC/'"$2"'/g' \
  | sed 's/DEST_PVC/'"$3"'/g')
echo "$job"
confirm
echo "$job" | $kc apply --wait -f -

echo ""
echo "= patch $kind ="
confirm
$kc edit $kind/$5

echo ""
echo "= scale up $kind ="
confirm
$kc scale $kind/$5 --replicas=1

Migrate job#

The following job may be ued by itself if you set the right name and set the claimNames with existing PVC

• migrate-job.yaml content:

apiVersion: batch/v1
kind: Job
metadata:
  name: rsync-DEPLOYMENT
spec:
  parallelism: 1
  completions: 1
  template:
    spec:
      containers:
      - name: rsync
        image: byzaneo/rsync:1.0.0
        imagePullPolicy: IfNotPresent
        command: ["rsync",  "-avhs", "/pvcsrc/", "/pvcdest/"]
        # if yyou want to keep the job's pod running
        # and run the the rsync command manually:
        # rsync -avhs --exclude fake/ /pvcsrc/ /pvcdest/
        # or check the pvcs data
        #command: ["sh", "-c", "tail -f /dev/null"]
        volumeMounts:
        - name: source
          mountPath: /pvcsrc
        - name: destination
          mountPath: /pvcdest
        # adjust the resources:
        resources:
          requests:
            memory: "256Mi"
            cpu: "1"
          limits:
            memory: "16Gi"
            cpu: "6"
      volumes:
      # source pvc
      - name: source
        persistentVolumeClaim:
          claimName: SOURCE_PVC
      # destination pvc
      - name: destination
        persistentVolumeClaim:
          claimName: DEST_PVC
      restartPolicy: Never

While the job’s pod is running, you can follow the logs to check the rsync command output.

One the job’s pod is done its status should be “Completed”

Resizing#

Requirements#

• kubectl

PVC resizing#

Increased the PVC requested size by editing the spec.resources.requests.storage propety.

Checks the namespace’s events to follow the CSI resize process.

StatefulSet resizing#

Once you resized the PVC, you may want to update the associated StatefulSet (STS) which manage the volume. But you will get the error below:

* spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

the following process is done with zero downtime

1. Get the STS manifest:

kubectl -n [namespace] get sts [name] -o yaml > sts.yaml

2. Edit the sts.yaml file to set the property spec.storage.VolumeClaimTemplate.spec.resources.requests.storage with the new PVC size

3. Delete the StatefulSet without deleting the pods:

kubectl -n [namespace] delete sts [name] --cascade=orphan

4. Removes clutter from the sts.yaml manifest (you can use kubectl-neat to do it) then apply it

kubectl -n [namespace] apply -f sts.yaml

If the STS is managed by Helm, remember to update your chart values with the new size