cStor volume snapshot and clone: A technical overview


Read this blog if you want to know technicalities of a cstor volume. If you want to try out snapshot and clones along with the blog then follow installation setups and provision cstor volume. Also checkout video demo of snapshot, link at the bottom.

The Key Terms: Jargons

Refer to the blog link in prerequisite to learn about other jargons.

  1. Snapshot: A storage snapshot is a set of reference markers for data at a particular point in time. A snapshot acts like a detailed table of contents, providing the user with accessible copies of data that they can roll back to.
  2. VolumeSnapshot: A snapshot request object in kubernetes. When this object is created then the snapshot controller would act on the information and try to create a VolumeSnapshotData.
  3. VolumeSnapshotData: It is a kubernetes object that represents an actual volume snapshot. It contains information about the snapshot e.g. the snapshot name mapped by the actual storage vendor, creation time etc.

You can relate these objects with the following relation:


The Whats: Introduction

The upcoming release 0.8 would be bundled with a stable snapshot and clone support for cstor volumes.

Assuming that a cstor volume is already created it is possible to take snapshot of it by creating a VolumeSnapshot object. Snapshotting would be successful only when the cstor volume is in healthy state. i.e. if .status.phase {Healthy} then snapshotting of the volume would fail.

You can always check if it is possible to take snapshot by inspecting the cstor volume object. Run the below command:

kubectl describe cstorvolume <cstor volume name> -n openebs

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl describe cstorvolume -n openebs pvc-9a3d6a39-f486-11e8-8689-42010a80017d
Phase: Healthy
Replica Statuses:
Checkpointed IO Seq: 0

Once a snapshot has been created we can clone the snapshot into actual volume which could be mounted to a workload.

The User Stuff: Snapshot Creation

Relation between CRs as viewed in MayaOnline topology view.

Let us try to take snapshot of the our volume. It must be noted that openebs DOES NOT support application level snapshots for now.

The pvc which I used to create cstor volume is openebs-pvc. This would be referenced in the volumesnapshot manifest.

apiVersion: volumesnapshot.external-storage.k8s.io/v1
kind: VolumeSnapshot
name: snapshot-demo
namespace: default
persistentVolumeClaimName: openebs-pvc

Things are quite clear from the manifest itself. Let us see what happens when we apply this yaml.

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl apply -f snapshot.yaml 
volumesnapshot.volumesnapshot.external-storage.k8s.io/snapshot-demo created

If the snapshot was successful we would expect an object of VolumeSnapshotData to be created which would have a reference to this object.

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl get volumesnapshotdata
k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910 1m

So our snapshot seems to be created. Let us see what exact information these two objects have in them.

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl describe volumesnapshot snapshot-demo
Name: snapshot-demo
Namespace: default
Labels: SnapshotMetadata-PVName=pvc-9a3d6a39-f486-11e8-8689-42010a80017d
Annotations: kubectl.kubernetes.io/last-applied-configuration:
API Version: volumesnapshot.external-storage.k8s.io/v1
Kind: VolumeSnapshot
Cluster Name:
Creation Timestamp: 2018-11-30T10:03:34Z
Generation: 1
Resource Version: 14151955
Self Link: /apis/volumesnapshot.external-storage.k8s.io/v1/namespaces/default/volumesnapshots/snapshot-demo
UID: 32fb7599-f487-11e8-8689-42010a80017d
Persistent Volume Claim Name: openebs-pvc
Snapshot Data Name: k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
Last Transition Time: 2018-11-30T10:03:35Z
Message: Snapshot created successfully
Status: True
Type: Ready
Creation Timestamp: <nil>
Events: <none>

The .status.conditions.status phase conveys the information that the snapshot was created. More information about actual snapshot is visible by:

prince@prince:~/go/src/github.com/princerachit/dump/gke_util$ kubectl describe volumesnapshotdata k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910 
Name: k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
Labels: <none>
Annotations: <none>
API Version: volumesnapshot.external-storage.k8s.io/v1
Kind: VolumeSnapshotData
Cluster Name:
Creation Timestamp: 2018-11-30T10:03:35Z
Generation: 1
Resource Version: 14151954
Self Link: /apis/volumesnapshot.external-storage.k8s.io/v1/volumesnapshotdatas/k8s-volume-snapshot-33a7b5c5-f487-11e8-a0ac-0a580a3c0910
UID: 33a81571-f487-11e8-8689-42010a80017d
Openebs Volume:
Snapshot Id: pvc-9a3d6a39-f486-11e8-8689-42010a80017d_snapshot-demo_1543572214878429520
Persistent Volume Ref:
Kind: PersistentVolume
Name: pvc-9a3d6a39-f486-11e8-8689-42010a80017d
Volume Snapshot Ref:
Kind: VolumeSnapshot
Name: default/snapshot-demo-32fb7599-f487-11e8-8689-42010a80017d
Last Transition Time: 2018-11-30T10:03:35Z
Message: Snapshot created successfully
Status: True
Type: Ready
Creation Timestamp: <nil>
Events: <none>

You can see that there are some important info present here like the Persistent volume name, the ID of the snapshot internally being used by openebs, reference to volumesnapshot and the status of the snapshot.

Wasn’t that quick? Yes! that is the advantage of openebs. cStor snapshots are fast because they use Copy-On-Write(Read more here). Inshort when we take snapshots we are not actually copying the volume anywhere, instead cStor internally leverages the zfs to save/record indexes. Thus, both actual volume and snapshot would refer to the same block of data unless one of them updates it.

The Interesting Stuff: Snapshot Technicality

Refer to diagram above the explanation is following:

  1. User registers a request to create volumesnapshot to kube apiserver, this request is then saved to the etcd db.
  2. The snapshot controller watches for volumesnapshot object, as soon as the snapshot controller realizes that one of the volumesnapshot object does not have corresponding volumesnapshotdata object, it tries to reconcile.
  3. It sends a create snapshot request to m-apiserver with CASSnapshot payload. This payload contains snapshot name, pv name, namespace of the pvc, and the cas type which is cStor here.
  4. The m-apiserver receives this post request at the /latest/snapshots/ endpoint.
  5. It then triggers the cas engine which in turn runs the required runtasks to create a cstor volume snapshot.
  6. The run task sends a create request to the target deployment’s cstorvolume mgmt container.
  7. This request is forwarded to all the cstor-pool containers which in turn take snapshot of the volumes.

The pods topology explain things in a better way. (A screenshot from Mayaonline.io topology view)

The User Stuff: Clone Creation

To create a clone out of a snapshot is very simple. All you need to do is create a pvc manifest that refers to the volumesnapshot object.


apiVersion: v1
kind: PersistentVolumeClaim
name: demo-snap-vol-claim
namespace: default
snapshot.alpha.kubernetes.io/snapshot: snapshot-demo
storageClassName: openebs-snapshot-promoter
accessModes: [ "ReadWriteOnce" ]
storage: 5G

You can see that the annotation snapshot.alpha.kubernetes.io/snapshot refers to the volumesnapshot object snapshot-demo. This tells the snapshot provisioner which snapshot has to be used to clone the volume.

By default all the clone pvcs would use openebs-snapshot-promoter storage class.

Let us see what happens when we apply this yaml.

prince@prince:~/go/src/github.com/princerachit/dump/demo-09-11-18$ kubectl apply -f snapshot_claim.yaml 
persistentvolumeclaim/demo-snap-vol-claim created

Let us inspect the pv for this pvc.

Name:            pvc-4041fd14-f487-11e8-8689-42010a80017d
Labels: openebs.io/cas-type=cstor
Annotations: pv.kubernetes.io/provisioned-by: volumesnapshot.external-storage.k8s.io/snapshot-promoter
snapshotProvisionerIdentity: volumesnapshot.external-storage.k8s.io/snapshot-promoter
Finalizers: [kubernetes.io/pv-protection]
StorageClass: openebs-snapshot-promoter
Status: Bound
Claim: default/demo-snap-vol-claim
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 5G
Node Affinity: <none>
Type: ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
IQN: iqn.2016-09.com.openebs.cstor:pvc-4041fd14-f487-11e8-8689-42010a80017d
Lun: 0
ISCSIInterface default
FSType: ext4
ReadOnly: false
Portals: []
DiscoveryCHAPAuth: false
SessionCHAPAuth: false
SecretRef: <nil>
InitiatorName: <none>
Events: <none>

You can see that in the annotation pv.kubernetes.io/provisioned-by: volumesnapshot.external-storage.k8s.io/snapshot-promoter rather than regular openebs.io/provisioner-iscsi. Thus, this is a cloned volume.

From the topology view we can see that both the actual volume as well as the cloned volume is on the same set of pools.

The Interesting Stuff: Clone Technicality

The volume clone is similar to dynamic volume provisioning. The only difference is the fact that the clone volume is provisioned through snapshot provisioner instead of volume provisioner. The flow of control is same. The actual creation of the volume at openebs layer differs as the snapshots are used as a source of volume instead of empty volume creation.

Usecase of snapshot/clone

CI/CD is part of the eco-system that enables applications to be tested through various pipelines (CI) and deployed into production automatically (CD) at the end of successful CI. When one of the stages of pipeline fails, developers would like to reproduce the state of the stateful application for further debugging. DevOps admins need a solution where preserving the state of the application at each stage of the pipeline is very easy and seamlessly integrated into the CI/CD platforms such as Jenkins using the API.

A devops admin can use the snapshot feature of openebs to preserve the state of a stage. Later this can be promoted to a clone volume and inspected. (Read more here)

Future Enhancements

As kubernetes is moving towards CSI compliant storage spec openebs v1.0+ is expected to adapt CSI compliant snapshotting.

Link to important resources

  1. Snapshot proposal
  2. Snapshot documentation
  3. Snapshot video demo weekly contributors meet
  4. MayaOnline for visualization and monitoring of resources




Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Congratulations to our Careers in Code graduates!

Skip Lambda, Save Data to DynamoDB Directly Using API Gateway; Process Later With Streams


How to Install PHP 8 on CentOS 8


How To Fix Asus Zenfone 5 ZE620KL Not Charging [Troubleshooting Guide]

How to Implement E-mail sending service with Spring-Boot

HackThisSite Realistic Mission 4

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sinha Software Solutions

Sinha Software Solutions

More from Medium

Contract and Inheritance

Systemtap — find who access the file

Messaging and Event Streaming with NATS Server

GitLab Runner Operator on Power