cStor volume snapshot and clone: A technical overview

7 min readJan 5, 2019

Prerequisite

Read this blog if you want to know technicalities of a cstor volume. If you want to try out snapshot and clones along with the blog then follow installation setups and provision cstor volume. Also checkout video demo of snapshot, link at the bottom.

The Key Terms: Jargons

Refer to the blog link in prerequisite to learn about other jargons.

Snapshot: A storage snapshot is a set of reference markers for data at a particular point in time. A snapshot acts like a detailed table of contents, providing the user with accessible copies of data that they can roll back to.
VolumeSnapshot: A snapshot request object in kubernetes. When this object is created then the snapshot controller would act on the information and try to create a VolumeSnapshotData.
VolumeSnapshotData: It is a kubernetes object that represents an actual volume snapshot. It contains information about the snapshot e.g. the snapshot name mapped by the actual storage vendor, creation time etc.

You can relate these objects with the following relation:

PersistentVolumeClaim:PersistentVolume::VolumeSnapshot:VolumeSnapshotData

The Whats: Introduction

The upcoming release 0.8 would be bundled with a stable snapshot and clone support for cstor volumes.

Assuming that a cstor volume is already created it is possible to take snapshot of it by creating a VolumeSnapshot object. Snapshotting would be successful only when the cstor volume is in healthy state. i.e. if .status.phase ∉ {Healthy} then snapshotting of the volume would fail.

You can always check if it is possible to take snapshot by inspecting the cstor volume object. Run the below command:

kubectl describe cstorvolume <cstor volume name> -n openebs

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl describe cstorvolume -n openebs pvc-9a3d6a39-f486-11e8-8689-42010a80017d
...
...
...
Status:
  Phase:  Healthy
  Replica Statuses:
    Checkpointed IO Seq:  0
...
...
...

Once a snapshot has been created we can clone the snapshot into actual volume which could be mounted to a workload.

The User Stuff: Snapshot Creation

Relation between CRs as viewed in MayaOnline topology view.

Let us try to take snapshot of the our volume. It must be noted that openebs DOES NOT support application level snapshots for now.

The pvc which I used to create cstor volume is openebs-pvc. This would be referenced in the volumesnapshot manifest.

apiVersion: volumesnapshot.external-storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: snapshot-demo
  namespace: default
spec:
  persistentVolumeClaimName: openebs-pvc

Things are quite clear from the manifest itself. Let us see what happens when we apply this yaml.

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl apply -f snapshot.yaml 
volumesnapshot.volumesnapshot.external-storage.k8s.io/snapshot-demo created

If the snapshot was successful we would expect an object of VolumeSnapshotData to be created which would have a reference to this object.

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl get volumesnapshotdata
NAME                                                       CREATED AT
k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910   1m

So our snapshot seems to be created. Let us see what exact information these two objects have in them.

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl describe volumesnapshot snapshot-demo
Name:         snapshot-demo
Namespace:    default
Labels:       SnapshotMetadata-PVName=pvc-9a3d6a39-f486-11e8-8689-42010a80017d
              SnapshotMetadata-Timestamp=1543572214871829268
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"volumesnapshot.external-storage.k8s.io/v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"snapshot-demo","nam...
API Version:  volumesnapshot.external-storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-11-30T10:03:34Z
  Generation:          1
  Resource Version:    14151955
  Self Link:           /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/snapshot-demo
  UID:                 32fb7599-f487-11e8-8689-42010a80017d
Spec:
  Persistent Volume Claim Name:  openebs-pvc
  Snapshot Data Name:            k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
Status:
  Conditions:
    Last Transition Time:  2018-11-30T10:03:35Z
    Message:               Snapshot created successfully
    Reason:                
    Status:                True
    Type:                  Ready
  Creation Timestamp:      <nil>
Events:                    <none>

The .status.conditions.status phase conveys the information that the snapshot was created. More information about actual snapshot is visible by:

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl describe volumesnapshotdata k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910 
Name:         k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  volumesnapshot.external-storage.k8s.io/v1
Kind:         VolumeSnapshotData
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-11-30T10:03:35Z
  Generation:          1
  Resource Version:    14151954
  Self Link:           /apis/volumesnapshot.external-storage.k8s.io/v1/volumesnapshotdatas/k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
  UID:                 33a81571-f487-11e8-8689-42010a80017d
Spec:
  Openebs Volume:
    Snapshot Id:  pvc-9a3d6a39-f486-11e8-8689-42010a80017d_snapshot-demo_1543572214878429520
  Persistent Volume Ref:
    Kind:  PersistentVolume
    Name:  pvc-9a3d6a39-f486-11e8-8689-42010a80017d
  Volume Snapshot Ref:
    Kind:  VolumeSnapshot
    Name:  default/snapshot-demo-32fb7599-f487-11e8-8689-42010a80017d
Status:
  Conditions:
    Last Transition Time:  2018-11-30T10:03:35Z
    Message:               Snapshot created successfully
    Reason:                
    Status:                True
    Type:                  Ready
  Creation Timestamp:      <nil>
Events:                    <none>

You can see that there are some important info present here like the Persistent volume name, the ID of the snapshot internally being used by openebs, reference to volumesnapshot and the status of the snapshot.

Wasn’t that quick? Yes! that is the advantage of openebs. cStor snapshots are fast because they use Copy-On-Write(Read more here). Inshort when we take snapshots we are not actually copying the volume anywhere, instead cStor internally leverages the zfs to save/record indexes. Thus, both actual volume and snapshot would refer to the same block of data unless one of them updates it.

The Interesting Stuff: Snapshot Technicality

Refer to diagram above the explanation is following:

User registers a request to create volumesnapshot to kube apiserver, this request is then saved to the etcd db.
The snapshot controller watches for volumesnapshot object, as soon as the snapshot controller realizes that one of the volumesnapshot object does not have corresponding volumesnapshotdata object, it tries to reconcile.
It sends a create snapshot request to m-apiserver with CASSnapshot payload. This payload contains snapshot name, pv name, namespace of the pvc, and the cas type which is cStor here.
The m-apiserver receives this post request at the /latest/snapshots/ endpoint.
It then triggers the cas engine which in turn runs the required runtasks to create a cstor volume snapshot.
The run task sends a create request to the target deployment’s cstorvolume mgmt container.
This request is forwarded to all the cstor-pool containers which in turn take snapshot of the volumes.

The pods topology explain things in a better way. (A screenshot from Mayaonline.io topology view)

The User Stuff: Clone Creation

To create a clone out of a snapshot is very simple. All you need to do is create a pvc manifest that refers to the volumesnapshot object.

e.g.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-snap-vol-claim
  namespace: default
  annotations:
    snapshot.alpha.kubernetes.io/snapshot: snapshot-demo
spec:
  storageClassName: openebs-snapshot-promoter
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 5G

You can see that the annotation snapshot.alpha.kubernetes.io/snapshot refers to the volumesnapshot object snapshot-demo. This tells the snapshot provisioner which snapshot has to be used to clone the volume.

By default all the clone pvcs would use openebs-snapshot-promoter storage class.

Let us see what happens when we apply this yaml.

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl apply -f snapshot_claim.yaml 
persistentvolumeclaim/demo-snap-vol-claim created

Let us inspect the pv for this pvc.

Name:            pvc-4041fd14-f487-11e8-8689-42010a80017d
Labels:          openebs.io/cas-type=cstor
                 openebs.io/storageclass=openebs-cstor-default-0.7.0
Annotations:     pv.kubernetes.io/provisioned-by: volumesnapshot.external-storage.k8s.io/snapshot-promoter
                 snapshotProvisionerIdentity: volumesnapshot.external-storage.k8s.io/snapshot-promoter
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    openebs-snapshot-promoter
Status:          Bound
Claim:           default/demo-snap-vol-claim
Reclaim Policy:  Delete
Access Modes:    RWO
Capacity:        5G
Node Affinity:   <none>
Message:         
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       10.63.240.55:3260
    IQN:                iqn.2016-09.com.openebs.cstor:pvc-4041fd14-f487-11e8-8689-42010a80017d
    Lun:                0
    ISCSIInterface      default
    FSType:             ext4
    ReadOnly:           false
    Portals:            []
    DiscoveryCHAPAuth:  false
    SessionCHAPAuth:    false
    SecretRef:          <nil>
    InitiatorName:      <none>
Events:                 <none>

You can see that in the annotation pv.kubernetes.io/provisioned-by: volumesnapshot.external-storage.k8s.io/snapshot-promoter rather than regular openebs.io/provisioner-iscsi. Thus, this is a cloned volume.

From the topology view we can see that both the actual volume as well as the cloned volume is on the same set of pools.

The Interesting Stuff: Clone Technicality

The volume clone is similar to dynamic volume provisioning. The only difference is the fact that the clone volume is provisioned through snapshot provisioner instead of volume provisioner. The flow of control is same. The actual creation of the volume at openebs layer differs as the snapshots are used as a source of volume instead of empty volume creation.

Usecase of snapshot/clone

CI/CD is part of the eco-system that enables applications to be tested through various pipelines (CI) and deployed into production automatically (CD) at the end of successful CI. When one of the stages of pipeline fails, developers would like to reproduce the state of the stateful application for further debugging. DevOps admins need a solution where preserving the state of the application at each stage of the pipeline is very easy and seamlessly integrated into the CI/CD platforms such as Jenkins using the API.

A devops admin can use the snapshot feature of openebs to preserve the state of a stage. Later this can be promoted to a clone volume and inspected. (Read more here)

Future Enhancements

As kubernetes is moving towards CSI compliant storage spec openebs v1.0+ is expected to adapt CSI compliant snapshotting.

Link to important resources