[13 min read]
Deployment scheme and h/w resources
The infrastructure for the example going forward will involve a stretched OpenShift cluster with an internal deployment of Red Hat OpenShift Data Foundation (simply abbreviated further with “ODF”). Stretching the ODF cluster across failure domains is the default type of deployment for running it in public cloud environments. Because it’s able to work across failure domains and is a distributed system naturally, also in any other environment the aim should be to leverage this ability to provide a fully redundant data management layer.
In this internal mode of deployment, all the elements that form an ODF cluster will be running within pods inside the same OpenShift cluster. For the operators and all the roles needed for ODF functionality with an internal deployment, all resources of CPU and memory as well as the networking come from the nodes the pods are running on.
ODF with an internal deployment can be seen as a standard workload running on OpenShift. It can run on worker, or “application”, nodes as shown in the drawing below but also it can run on infra nodes as shown in the next picture further down.
The provided capacity by ODF is managed by so called Object Storage Daemons, or in short OSDs. OSDs manage a single block devices capacity and is responsible for the redundancy of the data stored there. OSDs require CPU and memory resources from the nodes per deployed media or device. Deploying more devices would mean to use more resources from those nodes the devices are attached to or installed into. If there are fixed h/w resources on the nodes, scaling the number OSDs will reduce the h/w resources available to other applications running on the same nodes.
ODF is supported to run on infra nodes. Using dedicated nodes like infra nodes can also help to separate the bandwidth needs from applications network traffic. ODF and Ceph itself is network based. It will need a good bandwidth to provide a performant data access and separating it from application traffic might be a good choice. By moving the ODF components into separate infra nodes, the actual additional storage related network traffic of the worker nodes would be down to the storage access traffic generated by the application pods running on the nodes. Only the application access to data will then require some bandwidth but no worker nodes will be involved with the additional load by storage layer communication, e.g for replication traffic or self-heal. In the picture below, the media used for ODF are connected only to infra nodes used for ODF showing such a separation.
Choosing a deployment scheme based on networking considerations
In cloud environments, a limited network bandwidth is common, although this might be in the range of 10Gb/sec. If enough bandwidth is provided to the worker nodes, running alongside with applications becomes more feasible, especially with smaller footprints. However, depending on the traffic generated by the applications, network bandwidth might be a limiting factor. Also the size of an instance might impose further limitations since the network and storage offerings might differ a lot. Separating the ODF workload into dedicated infra nodes might be a good idea for larger OCP clusters anyway when struggling with the number and size of worker nodes already.
If the bandwidth on the worker nodes is rather limited or the scale of ODF is projected to grow beyond the threshold suitable for consuming the resources from the worker nodes, a separation into infra nodes could be a better approach. Those might also use different networking hardware with higher bandwidth than a worker node is configured with, e.g. 100 Gb/sec for ODF infra vs 10 or 25 Gb/sec for worker nodes.
Which model of deployment one would choose might also be a decision based on the environment and infrastructure available: In a cloud or in virtualised world in general, providing separate nodes for ODF is easy whereas in a bare metal installation it might depend on the scale of the workload capacity and performance.
Influence of the subscription structure
Subscriptions will be required for the worker nodes if the ODF components are running there. Using infra nodes instead, the compute power and memory for ODF will not require any special resources on the worker nodes and this would save on the subscriptions side for the OpenShift cluster. However, more nodes would mean to manage and pay for more nodes.
Especially, for bare metal installations with a smaller capacity footprint but good network bandwidth, using the worker nodes to run ODF alongside with the applications might save on spendings for hardware, footprint in the data center, and network infrastructure, although a share of the OCP subscriptions are then consumed for running ODF. Finding the right balance might become somewhat tricky. When planning a higher number of OSD instance roles because of capacity needs or performance requirements, using dedicated infra nodes usually would be the way to go.
Although infra nodes in OpenShift are used for other services, like routers, as well, the scheme of using infra nodes might depend on which environment the cluster is deployed in. Routers are usually separated to run on infra nodes but ODF might also use those. The decision might be based on the sizing of the infra nodes and the additional effort to scale the overall number of nodes.
Planning for the future
If the OCP cluster is at a fairly static size with no further growth anticipated, the planning could end at this point. Instead, with growth projected for the environment, this could have an influence on the model to go with.
The current scheme of supported scale of an ODF cluster inside the existing nodes is two-fold: First, the size of the media to add in the future should be at the same size as done initially. Secondly, the number of media to add for a specific growth would require additional h/w resources to be available.
When deploying ODF on worker nodes, adding more media to the nodes would require available hardware resources. As mentioned before, this would also reduce the resources available to applications on these nodes. In bare metal environments when running on blade nodes, the possible growth would be limited naturally by the number of available slots for media. In rack based servers, there might be more slots available and perhaps the resources for CPU and memory might also provide enough headroom. However, with smaller worker nodes, the resources used for ODF might limit the number of slots that can be used based on the available free resources to host additional OSD instances. Also, when planning to grow ODF using existing worker nodes, adding media also means freeing up resources for the OSDs and the applications perhaps might need to be moved over to other nodes. Additional planning might be involved, e.g. for dependencies being resolved by using labels for component separation that perhaps cannot be carried forward without adding more nodes.
Adding new nodes is always an option and ODF can run using infra and worker nodes simultaneously for the same cluster. However, there might be other dimensions to take into account which could be a limited rack space, costs for additional chassis, power sockets, network ports, limits in power consumption and cooling. While default deployment of ODF is to be spread across 3 different failure domains, scaling e.g. the number of racks used forming different failure domains is not supported. In such a case, either planning with the flexible scale model where a failure domain is the host used and distributing all the nodes across different racks could be applied. The other option would be to limit the ODF deployment using 3 racks only but to distribute the worker nodes to other racks as well. While racks are mentioned here, in real DCs, other dimensions like rows or structures imposed by power, cooling, and networking might give the guardrails for the effective failure domains to plan with. Upgrading a cluster with default deployment of 3 topology zones will also require an upgrade with media and nodes each time in the same manner for each one of the topology zones.
In on-premises virtualised deployments, adding more media to the cluster might be rather limited by the number of nodes but directly by the size of the VMs used. To understand the costs involved with different decision options, one might also keep in mind that not only the ODF components will require resources. A node in OpenShift consumes 0.5 vCPU for the kubelet and also ~2 GiB of memory. The size of the OS device is at minimum 120 GB but sometimes it’s recommended to even use more space. Scaling the infra nodes by additional ones for ODF will also scale those base requirements for the nodes and thus also consume more capacity. While being safe on the bare metal installation with it in general, in cloud environments this might be considered for the sizing it properly to avoid scaling the costs unnecessarily. Other dimensions might also become important, like growing the number of hypervisor nodes, networking infrastructure, and licenses required. Separating ODF nodes into dedicated hardware nodes instead might be an option to limit the costs in such cases.
In general, in almost all environments, usually separating ODF deployment into infra nodes is the best approach for future growth.
The next article will continue on this, discussing the general configuration of the lab environment and reliability considerations before in the following article the effective installation of the first OCP cluster in the lab environment will be started. Thanks for reading and stay tuned.