Planning the first OpenShift cluster for ODF

[10 min read]

The first OpenShift cluster

There are many components involved in a complete production environment of Red Hat OpenShift but for the sake of illustration, a basic environment is created to start with. This consists of

• a gateway for general access to the outside world,
• a local network that is used for my private world in the simulated IT infrastructure
• a local router for the local network of a component (here, the first OpenShift cluster)
• a jump host and DHCP server
• the component itself (here, the OpenShift cluster with OpenShift Data Foundation)

In this limited lab environment, I chose the internal deployment of ODF using the worker nodes, since I don’t need to care about subscriptions. All my worker nodes have an additional media attached, as shown in the drawing above. It will be used to provide the capacity for ODF. ODF itself will then provide the capacity for all the PVs as well as the locally provided object storage: Since Ceph as the underlying technology doesn’t need a separate storage area for storing S3 objects, the same media will be used to provide all 3 kinds of storage to the applications, so block for RWO PVs, file for RWX PVs, and object with self-service for creating buckets.

The installation drawing also shows “optional worker” nodes. At any point later on, the cluster can be expanded to include more worker nodes for supporting more applications. Those additional worker nodes will not need any additional special resources to be able to use ODF for PVs, beside the compute resources to work with those in mounting or providing the power for driving the block devices and local filesystems on top.

The OpenShift cluster will live inside a separate network, the pink “local network”, with no access from the outside, for now. The deployment actually doesn’t use any external load balancers, no special registry, and no integration with Single-Sign-On (SSO) for the accounts. All those things are covered for OpenShift directly elsewhere and thus I reduced this illustrative environment down to the bits I have focus on: the ODF relevant components.

Respecting failure domains

Default use of ODF in a public cloud usually involves 3 different failure domains which ODF will bridge by using nodes in all of the failure domains. In my lab environment, I will create those 3 failure domains too for simulating a similar physical layout. The three topology zones illustrate the 3 main data centers. The drawing below contains an OpenShift cluster with redundant storage across 3 main data centers, shown in the drawing as “DC1”, “DC2”, and “DC3”. Those DCs will be simulated and enforced for the installation of ODF by k8s topology zones.

ODF is a loosely coupled system. Managing the data redundancy across any infrastructure boundaries is done via a network available on all DCs. There is always the risk of a “split brain” scenario. The cluster needs to sort out the way to continue to work if one of the cluster DCs, or failure domains, will become unavailable for some reason. The approach to overcome the challenge with it in ODF and Ceph is to use the monitor roles (MON) of Ceph. There must be always the majority of the deployed monitor role instances available to keep the cluster operable, no matter what other roles, like Object Storage Daemons (OSD) or Metadata Servers (MDS), might fail. Based on the Rook operator code and configuration, it’s expected to have 3 monitors, so that one out of three can fail – this is the the default configuration and requires 3 independent zones, or failure domains.

The deployment of ODF uses all nodes that got the proper label cluster.ocs.openshift.io/openshift-storage= applied. In all ODF versions when using the UI based installation, one can select the nodes to use from a list and there is no need to apply the label manually. However, when installing ODF using the CLI only, this label must be applied prior to the installation. Nodes without the label will not be used for running ODF pods, including but not limited to MONs and OSDs. Hence, choosing the right nodes from the desired topology zones is essential. Also important would be to choose nodes with enough cpu and memory resources to allow proper rollout of the ODF components and Ceph roles with the pre-defined resource requests.

Placing the roles of MON, MGR, OSD, MDS and RGW, which is the Object Storage Gateway, might differ for different deployment environments. The role instances of the Ceph cluster will be placed in a way that redundancy is always given if a role instance fails. For example, OSDs will be distributed across all zones and by default 3 OSDs will be deployed for the minimum cluster initially. The cluster should end up having one OSD instance deployed per defined failure domain. Of course, there must be a block storage device of the desired size available to deploy those OSDs and one will need to prepare the worker nodes with the additional block devices.

One of the main roles of ODF is the monitor role. The monitor roles will come in default deployment with 3 different instances that must be applied each one in a different topology zone. Other Ceph roles might have fewer instances, like MDS having only two different instances, while RGW – the object storage gateway of Ceph – as the S3 gateway resource has only one instance or no instance at all like in AWS. A redundancy will be applied for the MON, MDS, and, of course, the OSD roles. Other roles having no counterpart instances deployed will gain the redundancy for accessing the service by restarting the pod if it fails, with no preference to the nodes, except for the ODF label.

In my lab environment, I expect to have the 3 worker nodes having each one a monitor and one OSD deployed. All other pods will be placed on any of the 3 worker nodes.

ODF is a network-defined storage

While the control plane nodes of OCP will need a very low latency of preferably below 2 ms, ODF and Ceph requirements are more relaxed with up to 10 ms of RTT between all the MON and OSD nodes. While it’s supported from a support standpoint, not all applications and workloads might tolerate it. Going with the best available latency, as always, is best for the applications.

In addition to the pure RTT figure, this latency should be also achievable with workloads running and with rebalance operations or self-heal in action to avoid starvation of production workloads. One also should keep in mind that the Multi-Cloud Gateway component of ODF itself uses a PV based on ODF and this might suffer from limited bandwidth availability, too.

Networking is important for client access and ODF internally. The redundancy needed across independent nodes and failure domains to provide a highly available data layer is achieved by creating replicas of the original data to other OSDs. Those are by default located in another topology zone. The standard and fully supported number of replicas is 3. In addition, a single OSD is structured into multiple areas of data placement, or PGs, for distributing the workloads payload data. All those OSDs will receive a portion of the workloads traffic. All those OSDs then will need to replicate the written data for redundancy. This in turn involves a multiplied traffic on the ODF layer for the write traffic by the clients. Also, when a relocation of portions of the data is needed during self-heal or when expanding the cluster, there is no client traffic involved but probably all the OSDs will heavily exchange data to reestablish the data redundancy or distribute the data for better performance in a fully automated way. This kind of traffic might be multiples or orders of a magnitude higher than usual client traffic and occurs in concurrence with the client data traffic.

All in all: the bandwidth ideally provided should easily accommodate this traffic on the network too and especially in addition to client traffic – and all at the desired latency to match the application needs. This is for within and across the DCs, or failure domains, and between all the nodes configured.

---

This is now the end of the initial planning discussion. The next article will cover the installation of the OpenShift cluster. So, stay tuned and thanks for reading so far.