(Photo by Matthias Münch, 2023, at Kulturkosmos Müritz e.V.)
[10 min read]
Compact format – ODF on a 3-node OpenShift cluster
Since the version 4.6 of Red Hat Openshift Container Storage, it provides the use of compact OpenShift clusters as a Tech Preview. This has emerged over time and the correct term for the compact cluster is now a 3-node cluster. While it was previously not planned to be expandable with additional worker nodes in an initial compact cluster, the 3-node OpenShift cluster is now intended to be used in edge deployments or as a starting point with the option to grow into a full fledged cluster.
An ordinary OpenShift cluster always requires a dedicated set of nodes for the control plane, regardless of the number of worker nodes. A compact cluster converges the cluster into only 3 nodes, providing the control plane, which are also acting as the “worker nodes”. The full redundancy is given this way as well as the ability to provide enough room for perhaps lots of applications, depending on the size of the nodes in terms of CPU, memory, and network. To complete the configuration, the storage management layer should provide redundancy, sufficient performance, and should also provide all kinds of storage needed for any application.
Using Red Hat OpenShift Data Foundation (RHODF, or in short ODF here) for the storage management layer provides first of all a reliable way of backing the scaled internal registry. It provides the reliable block devices for logging and metrics. It provides also object storage for workloads, as a scaling S3 compatible service, or in a different flavor as a more sophisticated S3 compatible service with the ability to compress, deduplicate, and encrypt any objects stored, or any combination of harnessing existing resources from the cloud or on-premises. OpenShift Data Foundation is, not only for 3-node OpenShift clusters, a one-stop shop for all kinds of applications.
The foundation: a 3-node OpenShift cluster
Since the OpenShift cluster is special with the converged nodes, in an installer provisioned installation (IPI), the size of the nodes should be different, at least for providing the resources for the applications to run. This needs to be respected and the default IPI configuration would not work. The same applies to a user provisioned installation for the OpenShift cluster, but since I have a Red Hat Virtualization as the underlying platform, IPI is my favorite in the lab.
When designing a 3-node cluster, first of all from a perspective of ODF, there should be block storage devices available to provide the storage capacity to the ODF cluster. In addition to this basic requirement, the ODF roles with all the instances need to run on those 3 nodes, the only nodes. Based on the default deployment for ODF, this would require 8 vCPU and 19 GB of main memory for the base plus 2 vCPU and 5 GB of main memory for any media or device provided to the ODF cluster (see Planning Guide). The sizing of the nodes should incorporate this, in addition to any control plane pods and application workloads.
The install configuration of the OCP cluster needs to reflect the different hardware resources. Without any changes applied, the openshift-install tool would configure 3 master nodes and 3 worker nodes. The worker nodes must be removed from the configuration, beside the tweaks needed for my single host RHV cluster. These base changes will then be applied by the installation tool and the masters will be set to schedulable automatically.
The installation in my lab starts as usual with an IPI installation powered by the openshift-install tool:
[root@acm40-jump ~]# ./openshift-install create install-config --dir=acm40
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform ovirt
? Cluster Default
? Storage domain Data2
? Network net_212
? Internal API virtual IP 172.21.52.6
? Ingress virtual IP 172.21.52.7
? Base Domain dslab.local
? Cluster Name acm40
? Pull Secret [? for help] ...***************************************************************************************INFO Install-Config created in: acm40 ....
[root@acm40-jump ~]The adjustments being made for reducing the number of worker nodes to 0 and the change for the single node RHV cluster in the lab are:
[root@acm40-jump ~]# vi acm40/install-config.yaml
[root@acm40-jump ~]# cat acm40/install-config.yaml
apiVersion: v1
baseDomain: dslab.local
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
platform:
ovirt:
affinityGroupsNames:
- compute
replicas: 0 # <= change this to have no worker nodes
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
platform:
ovirt:
affinityGroupsNames:
- controlplane
replicas: 3 # <= this remains as 3-node cluster needs
# 3 nodes for control plane
...
...
platform:
ovirt:
affinityGroups:
- description: AffinityGroup for spreading each compute machine to a different
host
enforcing: false # <= in my env only due to a single
# node RHV cluster
name: compute
priority: 3
- description: AffinityGroup for spreading each control plane machine to a different
host
enforcing: false # <= in my env only due to a single
# node RHV cluster
name: controlplane
priority: 5
...
...
Creating the manifests allows me to change the node definitions afterwards. This step will also apply the setting for the worker nodes and will adjust the control nodes to be set to “Schedulable” automatically:
[root@acm40-jump ~]# ./openshift-install create manifests --dir=acm40
INFO Consuming Install Config from target directory
...
**************************************************************WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings ****************************************************************************************************************************************************INFO Manifests created in: acm40/manifests and acm40/openshift **********************************************************************************************************************************
[root@acm40-jump ~]#The sizes of the nodes must be changed after the base installation configuration is prepared. The actual and default configuration is to use 4 vCPU and 16 GB of main memory only. In order to run ODF and additional workloads on the very same nodes alongside the control plane pods, much more resources are needed. Unfortunately, a change for all 3 nodes at once in the previous steps gets not propagated by the tool and so the individual configuration files of all the 3 control plane nodes must be changed manually. The example for the control plane node 0 illustrates what needs to be performed on the other nodes as well:
[root@acm40-jump ~]# vi acm40/openshift/99_openshift-cluster-api_master-machines-0.yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
...
spec:
...
providerSpec:
value:
...
cpu:
cores: 22 # <= number of virtual cores for the VM ...
sockets: 1 # <= ... provided in a single virtual CPU ...
threads: 1 # <= ... having one virtual thread per virtual core
...
memory_mb: 59392 # <= 58 GB of memory
... Finally, I’m done with the changes needed and can install the 3-node OpenShift cluster. For completeness, the start and the end of the installation is shown below.
[root@acm40-jump ~]# nohup ./openshift-install create cluster --dir=acm40 --log-level=debug &
[1] 13021
[root@acm40-jump ~]# nohup: ignoring input and appending output to 'nohup.out'
[root@acm40-jump ~]# tail -f nohup.out
level=debug msg= Reusing previously-fetched Certificate (kube-apiserver-lb-signer)
level=debug msg= Fetching Install Config...
level=debug msg= Reusing previously-fetched Install Config
...
level=info msg=Install complete!
level=info msg=To access the cluster as the system:admin user when using 'oc', run
level=info msg= export KUBECONFIG=/root/acm40/auth/kubeconfig
level=info msg=Access the OpenShift web-console here: https://console-openshift-console.apps.acm40.dslab.local
level=info msg=Login to the console with user: "kubeadmin", and password: "XXXXX-XXXXX-XXXXX-XXXXX"
...
^C
[1]+ Done nohup ./openshift-install create cluster --dir=acm40 --log-level=debug
[root@acm40-jump ~]#For checking the actual cluster, one needs to login to the cluster using the oc tool. I can now check for the number of nodes as well as the size of the nodes applied and the resources consumed already.
[[root@acm40-jump ~]# oc login -u kubeadmin -p XXXXX-XXXXX-XXXXX-XXXXX https://api.acm40.dslab.local:6443
...
Login successful.
...
[root@acm40-jump ~]# oc get nodes -o wide
NAME STATUS ROLES AGE VERSION
acm40-lptrr-master-0 Ready master,worker 154m v1.24.6+5157800
acm40-lptrr-master-1 Ready master,worker 154m v1.24.6+5157800
acm40-lptrr-master-2 Ready master,worker 154m v1.24.6+5157800
[root@acm40-jump ~]#
[root@acm40-jump ~]# oc describe node/acm40-lptrr-master-1 |egrep -B1 "cpu|memory" |egrep -v "Memory|hugepages|--"
Capacity:
cpu: 22 # => resources configured on the node
memory: 59545516Ki # => resources configured on the node
Allocatable:
cpu: 21500m # => available for allocation
memory: 58394540Ki # => available for allocation
cpu 3043m (14%) 0 (0%) # already allocated
memory 10380Mi (18%) 0 (0%) # already allocated
[root@acm40-jump ~]# The first set of entries in bold reflects the resources configured to the nodes. Below the “Allocatable:” line, the next set of cpu and memory reflects the resources that are allocatable for any pods, followed by the last set reflecting the already allocated resources. While nothing was installed yet in addition, already some significant amount of resources is consumed for the OpenShift internal pods. This illustrates that a good planning of the nodes is needed in terms of resources provided.
The ODF cluster complementing the 3-node OpenShift cluster
But why deploy a 3-node cluster at all in my lab ? At the actual state of the story, there are two independent OCP clusters: a 3 nodes/zones standard ODF cluster and a stretched/2-main-DC ODF cluster. In the future with workloads running on the clusters consuming persistent volumes and object storage backed by ODF, the need for supporting disaster recovery procedures might arise. ODF provides those mechanisms but needs a controlling Advanced Cluster Manager (ACM) in order to provide the automated configuration for DR and the automated control of the failover to the secondary cluster. ACM itself should be available and must provide redundancy to the data stored internally. The way of supporting this OCP cluster and ACM itself is by deploying it with the smallest redundant OCP cluster: a 3-node cluster with a redundant storage layer by ODF.
ODF plays a special role by not only offering some sort of redundant persistent volumes but also the S3 object buckets. This would avoid the need for additional external resources that might not be available in the installation environment otherwise.
Independent of the use case being here the ACM cluster: redundancy of the storage provided is key to production environments and thus the same configuration can be used in any standard edge case, small deployments, etc., where 3 nodes are available.
The installation of the ODF cluster itself is the same as with any other installation. As in the previous examples, local storage will be used, provided somehow to the OCP nodes. It’s then a local storage operator (LSO) based storage provisioning to the ODF cluster, like outlined in the other articles, already. In addition, all things described in the previous articles like in “Planning the first OpenShift cluster for ODF” apply also to a 3-node cluster installation, in terms of sizing and other recommendations.
The other differences should be obvious, like that one shouldn’t taint the nodes as it wasn’t done before for the reason having no need for it – however, tainting the only 3 nodes would be catastrophic to other workloads and some base services of the OCP cluster, too.
It might be obvious too that a 3-node cluster cannot span 2 main DCs. So, replica 4 is not needed to be taken into consideration for dimensioning the capacity needed. If available, spreading the nodes across 3 different failure domains, if it makes sense, would be a good idea, too.
When thinking ahead, for the planning of any future changes to the cluster’s use and perhaps topology, although 3 nodes exist in the initial cluster only, one should apply zones for the installation of the ODF cluster. This should be done always, even if it’s not yet distributed across 3 real failure domains: If the cluster eventually gets more ODF nodes, at least, and failure domains like different racks, availability zones, etc., are applied to the cluster, the data placement should reflect those failure domains. Although with the initial installation only 3 nodes are in the picture, those 3 nodes should be reflected in the ODF topology zones. Any new nodes added to the cluster will change this “harmony” and possibly place redundant copies of data into the same failure domain. Note that changing the zone configuration after the installation is not supported. The zone definition itself is similar to a standard ODF cluster installation:
[root@acm40-jump ~]# oc label node acm40-lptrr-master-0 topology.kubernetes.io/zone=DC1
node/acm40-lptrr-master-0 labeled
[root@acm40-jump ~]# oc label node acm40-lptrr-master-1 topology.kubernetes.io/zone=DC2
node/acm40-lptrr-master-1 labeled
[root@acm40-jump ~]# oc label node acm40-lptrr-master-2 topology.kubernetes.io/zone=DC3
node/acm40-lptrr-master-2 labeledAfter going through the usual ODF cluster installation, covered in more detail in “Installing the first ODF cluster“, there should be a working ODF cluster. Without applications running on this initial OCP cluster, this ODF cluster could be used already for the internal services of OpenShift. This will be discussed in the next article.
Summary and further outlook
The 3-node or “compact” cluster is an all-in-one OpenShift cluster without additional workers. It’s a full OpenShift cluster with a smaller footprint. Everything is provided and run in the combined master/worker nodes only – also the ODF storage layer.
This internal deployment was sketched in the article with an internal ODF deployment. In theory, an existing Ceph cluster could be used as an external ODF cluster. This is not yet described in the documentation but is a standard way of integrating with an existing Red Hat Ceph Storage cluster and it might be covered in a future article.