An ODF cluster for 2 main DC – install it !

[14 min read]

An ODF cluster for 2 main DC – install it !

OpenShift Data Foundation in the standard installation would be installed using three different failure domains or just on a single failure domain. Many companies, however, have only two main data centers. This article continues from the preparation in the previous article “An ODF cluster for 2 main DC -preparation” to installation of the ODF stretch cluster.

Installing the ODF and LSO operators

The LSO and the ODF operator need to be installed. Although it was explained in “Installing the first ODF cluster“, I just want to make a little excursion to the selections possible in the operator installation screen. No changes should be done to the channel selection and thus left as “stable”. Since the only supported namespaces are “openshift-local-storage” for LSO and “openshift-storage” for ODF, no change is required here.

For the update approval setting of LSO, I would refer to the discussion in the next section. With a small difference that things don’t change for existing configurations with the LSO and thus the consumption for ODF. It might be easier to simply stick with preselected Automatic if not strictly required to be manual only for change management compliance.

For the ODF operator, since changes to the effective software of the cluster might sometimes change the behavior of things, one might perhaps not want to get changes introduced without testing those first. Especially for production grade clusters, setting this to manual would avoid blindly updating the ODF cluster. Another scenario for this could be testing the behavior of things when nothing should be changed in the cause of the tests that perhaps might take longer. However, one should then observe the availability of new versions and must manually approve the upgrade of the operator which will in turn automatically update the ODF cluster.

The integration into the OpenShift WebConsole should be enabled. Without enabling it, the graphical user interface for observation around ODF will not be available. However, if someone missed it, it’s easy to enable it afterwards in the Operator page – the actual status of the WebConsole integration is shown and can be switched as well there.

Don’t forget to refresh the WebConsole if you asked to do so – only this gives the view for ODF integrated into the left side of the OpenShift console panel, under Storage.

Installation of the storage cluster

As a hint what to see in my cluster: I already have a storage class deployed that was created during the installation of the cluster in IPI mode. The ovirt-csi-sc is the storage class providing storage persistent volumes directly from the RHV environment using the default storage selected during the installation. In my cluster, this is used for the internal registry to provide it with a persistent volume. 

At this time, the storage system, so the effective ODF cluster, is not yet installed. The configuration for the stretch cluster deployment is selected during the installation in the next steps. Note that once the cluster is installed as a stretch cluster over only 2 main DCs, this cannot be changed back into another type of deployment in a supported way.

I’ll start the installation of the ODF cluster the usual way and select “Create a new Storage Class using local storage devices” – remember that this is the only way for a 2 main DC stretch cluster.

Different than in “Installing the first ODF cluster“, after picking the devices, device set, and storage class for the LSO based backing storage, is that we now should see at least 4 nodes with storage:

OpenShift cluster for 2 main DC – simulated

Not shown here but one can check in “Installing the first ODF cluster” for this view, when selecting the “Disks on selected nodes” option, all nodes found will be listed. One needs to select the nodes to go with. On the right hand side of the table, the zone labels are displayed. This eases the understanding which nodes are located in which zone. It’s important to select the same number of nodes and effectively the number of devices per zone. There might be additional nodes listed and those might have media that one doesn’t want to use for ODF – select carefully. For me, that task is easy since I don’t have more than the 4 nodes required in the main DC zones.

There are other Advanced settings to explore and those might be used also in a deployment of a stretch cluster. See “Installing the first ODF cluster” as before explaining those more broadly. 

Going on with hitting the Next button, the following callout will be displayed:

One can see the clear explanation on what is expected for the preparation. The “Arbiter stretch cluster” is exactly the thing I want to install: the ODF HA stretch cluster. So, where one comes across the one or the other description – those are synonyms, effectively.

Next step is to determine the deployment type. Once the capacity is displayed as expected – it can take several minutes – I need to check the “Enable arbiter”. Note that this would require an ODF Advanced subscription. This is available usually through the “OpenShift Platform Plus with ODF Advanced” subscription.

By checking the “Enable arbiter” entry, an additional warning is displayed. The important part here is to select an arbiter zone to use.

Missing it will carry on with the installation and will also try to create an ODF cluster. However, the cluster will then stay in progressing mode but the listing of the pods shows something like this (no additional pods created especially for the internal Ceph cluster):

[root@ocp21-jump ~]# oc -n openshift-storage get pods 
NAME                                               READY   STATUS    RESTARTS   AGE
csi-addons-controller-manager-76cbd5c885-5k9d2     2/2     Running   0          75m
noobaa-operator-7df577bbc8-2xzvc                   1/1     Running   0          76m
ocs-metrics-exporter-596b88dd9f-t7w45              1/1     Running   0          75m
ocs-operator-564879c5cf-9z7nq                      1/1     Running   0          76m
odf-console-75f688bb87-jdq9q                       1/1     Running   0          76m
odf-operator-controller-manager-66b6648454-t8vqd   2/2     Running   0          76m
rook-ceph-operator-bd9f568bb-xpn9h                 1/1     Running   0          76m
[root@ocp21-jump ~]# 

Also, one will find in the logs of the ocs-operator pod entries like:

"Reconciler error","reconciler group":"ocs.openshift.io","reconciler kind":"StorageCluster","name":"ocs-storagecluster","namespace":"openshift-storage","error":"arbiter is set to enable but no arbiterLocation has been provided in the Spec.NodeTopologies.ArbiterLocation"

So, I make sure to select the right one by selecting the “arbiter” zone.

Below the selection for the deployment type, the listing of the nodes shows that I’ve got sufficient resources on my nodes. I don’t taint nodes: this would render the worker nodes unusable for workloads that don’t tolerate the taint …. and this would mean that pretty much no pods will run in addition to ODF. Since these are my only schedulable nodes I have, I strictly want not to set any taints to the nodes. If one would have infra nodes for ODF to run on, also only taints should be applied to these ones if nothing else should use those, except ODF.

In case there would be fewer resources available, this would be called out here and perhaps a minimal deployment would be done, or it wouldn’t be possible to install the cluster at all.

For my cluster, all is fine and the next step would be to choose the optional encryption. Because of the limited resources in my environment with the aim to run real workloads on this cluster, I prefer to ignore the encryption here. Later on in one of the other deployments, I will configure a cluster with encryption enabled. 

Currently, nothing else will be of special interest to me and I will carry it on to complete the installation of the ODF cluster. The last step is to check the configuration. My configuration is very simple but reflects the essentials of the HA stretch cluster configuration on the left picture. Instead, the right picture shows the results if missed to select the arbiter zone.

The installation of the cluster will now be performed by the ocs-operator and the related operators of ODF.  What makes this configuration special was that I wanted a specific distribution of the relevant pods. I expect that

• 2 mon pods run on the nodes in DC1,
• 2 mon pods run on the nodes in DC2,
• 1 mon pod runs on the master in the arbiter zone,
• 4 OSDs run on the nodes in DC1,
• 4 OSDs run on the nodes in DC2,
• 2 MGR pods run in any of the nodes in each DC1 and DC2,
• 2 MDS pods run in any of the nodes in each DC1 and DC2,
• 2 RGW pods run in any of the nodes in each DC1 and DC2.

All the other ODF pods, like the “noobaa” pods that form the Multi-Cloud Gateway (MCG) will run on any node in the cluster, because those will be restarted on any other node if need arises and no further dependencies or requirements for HA distribution are needed.
It takes a couple of minutes, so ~10 minutes with my deployment, until the whole ODF cluster installation is completed.

Let’s check: First the MONs – since the labels are not displayed this way, I added the label in blue after the node name to illustrate the distribution better.

[root@ocp21-jump ~]# oc -n openshift-storage get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName|grep mon
rook-ceph-mon-a-855dbd5464-km472    ocp21-kvm2z-worker-w9gz2   DC2
rook-ceph-mon-b-586c4f44dd-7vm2g    ocp21-kvm2z-worker-xlth6   DC2
rook-ceph-mon-c-fcc677d49-z4vd2     ocp21-kvm2z-worker-bcjpx   DC1
rook-ceph-mon-d-797d6ff6f8-vkw8p    ocp21-kvm2z-worker-swzwh   DC1
rook-ceph-mon-e-7579d658d6-7gdc7    ocp21-kvm2z-master-1       arbiter
[root@ocp21-jump ~]# 

One can easily see that the root-ceph-mon-e-* pod is deployed on the node “ocp21-kvm2z-master-1” that I declared to be in that separate “arbiter” zone. 

[root@ocp21-jump ~]# oc -n openshift-storage get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName|grep mgr
rook-ceph-mgr-a-7bc8dbf849-bt76w                                  ocp21-kvm2z-worker-xlth6  DC2
rook-ceph-mgr-b-59c894b76-scpkj                                   ocp21-kvm2z-worker-bcjpx  DC1
[root@ocp21-jump ~]# 

[root@ocp21-jump ~]# oc -n openshift-storage get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName|grep mds
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-6cdc5c45wbcjm   ocp21-kvm2z-worker-xlth6  DC2
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-975d4cf8j2gdn   ocp21-kvm2z-worker-bcjpx  DC1
[root@ocp21-jump ~]#

[root@ocp21-jump ~]# oc -n openshift-storage get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName|grep rgw
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-59df68f8jgh5   ocp21-kvm2z-worker-swzwh  DC1
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-59df68fvctbp   ocp21-kvm2z-worker-w9gz2  DC2
[root@ocp21-jump ~]#

Checking the installation in the UI:

Also, some statistics shown, with low traffic on the cluster for now – nothing except the MCG database is currently running on ODF, for now:

The object storage part, so the Multi-Cloud Gateway, has been configured properly. Note that the backing storage for the MCG is a bucket provided by the internal Ceph cluster using the RGW pods. This bucket is not explicitly listed here, since this is not an Object Bucket Claim. However, the Resource Providers tile shows “1 S3_Compatible” which is the bucket provided by Ceph.

This concludes the installation of an ODF HA stretch cluster for 2 main DCs.

Comments on the installation chosen

What else should be mentioned ? I put the arbiter pod on the 3rd control plane node – which didn’t require anything to change for me, like to set the node to schedulable. The ODF operator configured the node with the ODF label but didn’t change the setting for scheduling. However, two pods are now running on this node that are tolerated automatically.

[root@ocp21-jump ~]# oc -n openshift-storage get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName|grep ocp21-kvm2z-master-1
rook-ceph-crashcollector-ocp21-kvm2z-master-1-5ff574c86-2frdm     ocp21-kvm2z-master-1
rook-ceph-mon-e-7579d658d6-7gdc7                                  ocp21-kvm2z-master-1
[root@ocp21-jump ~]# 

Another hint might be that the arbiter tolerates a maximum for the Round Trip Time (RTT) of 100ms – instead, the ODF nodes require a maximum of 10 ms RTT between each if the application can tolerate it – see this solution article. This opens different scenarios to place the arbiter somewhere else than on a control plane node: it could run on a separate distant Openshift node – so worker or infra node – somewhere in a cloud. ODF doesn’t depend on the low latency for the arbiter and could be handled very differently to what the OCP control plane nodes would require. The only mandatory circumstance to maintain is that the network access to the node running the arbiter pod must be available at any time the cluster sees a reconfiguration, network separation, or a DC down event.

Running applications on a ODF stretch cluster

The easiest type of stateful applications is the scale-out application running multiple pods, on perhaps different machines at any time while sharing a RWX PV. If half of the cluster becomes unavailable, the remaining pods will simply continue.

The more complicated type of persistent applications is the single reader/writer application pod. If it’s for any reason using a RWX type of PV, the pod can continue to work or can immediately continue to work after it has been restarted on another available node.

Instead, if it’s using a PV of type RWO which would be the proper selection for performance, in case of a need to restart this pod because the node it was running on failed as well, the pod could be restarted but will be blocked from accessing the RWO PV for the default of 15 minutes – to avoid data corruption because the other pod was only meant to be down but wasn’t. Since a RWO PV on block devices, as the ocs-storagecluster-ceph-rbd storage class is providing, would have a local filesystem applied, usually ext4, only one node must have this filesystem mounted at any time. However, this behavior is not an architectural shortcoming of ODF – it’s an inherent safety feature of Kubernetes as the underlying technology of OpenShift. So, be aware that RWO PV based pods perhaps will need some time to become available when a DC outage happens. This is normal to those applications and are not a problem introduced by ODF – it will happen always if a pod with RWO PV is restarted on another node than it was running before.

---

Conclusion

OpenShift Data Foundation Advanced can provide a redundant layer for persistent storage for stateful applications even in environments with 2 main DCs only. The deployment has some requirements that should be met but the installation is easy and very similar to any other installation of ODF.