[21 min read]
Installing the first ODF cluster
From the documentation, the first step with local storage is to install the Local Storage Operator from the operator hub, followed by the installation of ODF operator itself. There is nothing magic with this but one should respect the namespaces to use.
Installing the Local Storage Operator
The steps are fairly simple and the following cadence of pictures illustrate it. Alternatively, also one can use the CLI based installation, of course.
Installing the OpenShift Data Foundation operator
With a running OCP cluster deployed and zones defined, like shown in the previous article already, I can proceed and install ODF itself. The installation can be performed using CLI or yaml files for automating it if many clusters need to be rolled out over time. In my example, I go with the UI based installation.
From the operator hub, one can search for the OpenShift Data Foundation card, select it and install it. One should make sure to keep all the settings as default, also the namespace used, but also make sure to have selected to enable the console plugin.
At this stage, only the operator gets installed. The installation of the operator will take a couple of minutes. In my test environment, it took roughly 1.5 minutes. While the installation of the operator is running, there will be a hint displayed to refresh the Web Console of OpenShift as shown in the picture above. This is the case if properly selected the integration before. Refreshing the Web Console is necessary to see the ODF integration in the console located in the Storage submenu.
The integrated ODF console
If one have forgotten to select it during the operator installation step, now it’s the time to correct it. Without the proper Web Console integration, installation using the UI will be impossible. You might either not see the right screen for the installation or miss the link in the console at all. There are two different options, one with UI and one using the CLI covered in the troubleshooting guide. After applying this, one should refresh the Web Console, too. If a 404 error is shown on the Data Foundation tab, clearing the web browsers cache will help to get the page content displayed.
Installing the ODF storage cluster
Now I’m ready to install the ODF cluster itself. Following the documentation, Create Storage Cluster will be the next step. This is found either in the operator or in the side menu going to the storage system view, as the pictures below show.
Starting the cluster creation will take one into the installation screen which offers different methods. Some might be unavailable within certain environments, like everything that is not available in the cloud will be invisible as an option. In my view, all those options are visible:
First is to decide on the deployment type. I can decide between performing a full deployment or one with the MultiCloud Object Gateway only. The latter one is supported only for Red Hat Quay installation providing the S3 storage. If nothing else would be our goal, I could choose it and avoid installing a full fledged cluster. However, I would not be able to create any PVs and since I am aiming on the full set of services, I go with the full deployment:
The backing storage
In the previous picture, another selection is on the Backing storage type. This is the type of storage resource the ODF cluster will use for providing the block devices for the OSDs. The last one, the external storage platform, means to use an already existing separately deployed Red Hat Ceph Storage cluster. The first one will use an existing storage class but because I have installed the OCP cluster using IPI on RHV, the only existing storage class would be the one used to create the nodes and where the OS disks are based on. In my case, I don’t want to use the same storage resource which is a SSD RAID-5 but want to use local NVMe devices. Those are configured already to the nodes for ODF but aren’t usable for OCP, yet. Since those are local storage devices to my nodes, I go with the middle option that is creating a new storage class for using my local devices:
After selecting the Next button, a spinning icon is shown that indicates that there is something running in the background. It says, it will take a few minutes, but with a smaller number of nodes and devices that shouldn’t take minutes. However, for larger cluster deployments with lots of devices, this may take a while. If one would argue about the current status, switching to the CLI would be a way to check the logs for what the operator is doing currently.
The next screen shows a number of options and input is required. The name for the LocalVolumeSet is a freeform string adhering to the OpenShift object naming conventions. The StorageClass naming is also freeform in the same way and is preselected to be the same as the LocalVolumeSet. It might be a good idea to rename it with relation to the LocalVolumeSet name to be able to distinguish both quickly. I chose set1 and sc-set1, clearly saying that both are different in their meaning.
The selection of the nodes, if all nodes having devices to be used, could be left at “Disks on all nodes”. If there are more nodes available than 3 and all have storage devices available for ODF, one would perhaps switch to “Disks on selected nodes”. There are multiple reasons for doing so and some of those might be:
- the base services, as those are the operators and perhaps the monitors should by default reside on the first 3 nodes chosen,
- on the other nodes, perhaps not enough resources might be available and by misplacing other ODF pods there, attached devices might be unusable,
- the starting cluster should only use devices from the first 3 nodes but leave all devices on other nodes untouched.
In my example, I’ve got only 3 nodes, the worker nodes, that are available and have devices, and so I select “Disks on all nodes”.
If one will wait long enough, so perhaps a few minutes, the capacity available for creating the ODF cluster will be displayed in the right upper area with the diagram. The time this will take depends on the speed of the nodes and the number of devices to find. By default, all available block devices from the nodes selected will be taken into account. This includes flash devices and HDD. Because of the expected high density of IOPS/GB with container workloads, only flash devices are currently supported for ODF.
I select for my deployment the “SSD” entry form the drop-down menu and now only flash type devices should be counted towards the available capacity. Check that the nodes are selected properly if not going to use devices from all available nodes in the OpenShift cluster.
At this time, it’s important to check the desired capacity. Also check not only the sum of the capacity but also check for the proper listing of the desired devices: The overall capacity should match all the devices matching the actual filter, so flash in my case. Make sure to check the number of nodes and the number of devices. Check the capacity to be really the capacity you expect: missing proper devices but recognising some others instead might slightly change the capacity listed. Especially with larger devices, calculating the expected capacity will help to understand a possibly inappropriate result. If devices are missing on some nodes, the deployment of the cluster might eventually fail with an incomplete configuration because redundancy scheme for media distribution across failure domains, a host or a whole DC in my case, cannot be created.
One can also use the command line to inspect which devices were found. The listing of the devices capacity is in bytes. The last column shows the status of devices found. NotAvailable means that it’s used by something else, already. The devices you aim at for ODF should all have the size you expect and should be available.
[root@ocp31-jump ~]# oc get localvolumediscoveryresults -o wide -o custom-columns='NAME:metadata.name,PATH:status.discoveredDevices[*].path,SIZE:status.discoveredDevices[*].size,AVAILABLE?:status.discoveredDevices[*].status.state'
NAME PATH SIZE AVAILABLE?
discovery-result-acm40-7smkm-master-0 /dev/sda1,/dev/sda2,/dev/sda3,/dev/sda4,/dev/sdb,/dev/sdc,/dev/sdd 1048576,133169152,402653184,128311082496,107374182400,107374182400,107374182400 NotAvailable,NotAvailable,NotAvailable,NotAvailable,NotAvailable,Available,Available
discovery-result-acm40-7smkm-master-1 /dev/sda1,/dev/sda2,/dev/sda3,/dev/sda4,/dev/sdb,/dev/sdc 1048576,133169152,402653184,128311082496,107374182400,107374182400 NotAvailable,NotAvailable,NotAvailable,NotAvailable,Available,Available
discovery-result-acm40-7smkm-master-2 /dev/sda1,/dev/sda2,/dev/sda3,/dev/sda4,/dev/sdb,/dev/sdc 1048576,133169152,402653184,128311082496,107374182400,107374182400 NotAvailable,NotAvailable,NotAvailable,NotAvailable,Available,Available
[root@ocp31-jump ~]# Note that currently no removable devices will be listed here. In Bare Metal deployments, it’s known that some models of servers list internal SSD as removable which will then not be recognised.
Let me now explore the “Advanced” section (see picture below). In the standard deployment of an ODF cluster in a net new OpenShift cluster, there will be only a single storage class for the OS drives. The way of providing local storage resources, so raw block devices available to the nodes to use for ODF, is by creating PVs for consumption as the backing devices for the Ceph cluster. As Ceph doesn’t use a file system on the devices in ODF, block devices are needed for the PVs. Unlike standard PVs created from an existing storage class where usually a file system will be applied during the mount of the PV, those PVs for Ceph will not use the “Filesystem” but need to use “Block” as the Volume type. The Local Storage Operator will pick this setting up and the PVs will then be created using the selected mode for the new storage class.
As there might be more devices available on the nodes, one can wish to use only a subset of those. On the bottom of the Advanced section, the number of devices can be specified to be used during the installation of the cluster. In my example, I want to go for one device per node and provide a number “1” into the field. This course of action will also be available when adding more devices later on if the capacity should be expanded but not all devices should be consumed. It’s important to understand that the selection will allow the same number of devices for each node selected. For example, if there are 8 nodes, one cannot expect to be able to specify to use only one device on 4 nodes and two devices on the other 4 nodes, so using only 12 devices instead of 8 by selecting “1” or 16 by selecting “2”.
The Next button carries this into the first step of the deployment of the resources for the ODF cluster: the preparation of the PVs for the backing storage devices for Ceph. This is explained in the following pop-up window that is asking for confirmation:
Once one has reached this point, it’s really the time when the Local Storage Operator starts to create the device set and the PVs using the devices identified. If I would continue but after this step I would figure out that it cannot complete the installation of the cluster, e.g. because I missed the step to create the labels on the nodes for the zones, there will be no way to stop the action. However, one can abort the installation of the cluster at any time. In this case, the volume set will be there already and the PVs will have been prepared. Restarting the setup process for the storage cluster will then find that those devices are not available anymore for selection but it will also show a new storage class with the name selected in the previous steps. However, remember that using an existing storage class was the first option in the initial step for the storage cluster definition and this would be the option to use for a second attempt if the configuration of the new storage class completed but the installation of the cluster was aborted before finishing the configuration of it.
Continuing with the creation of the LocalVolumeSet would then show a spinning pop-up calling out that the PVs are being provisioned on the selected nodes. This takes some minutes but the next step is already displayed: the check of the nodes and some additional configuration of it. Initially, the Available raw capacity will show up as 0B. This is because of the preparation of the PVs by the Local Storage Operator is still in progress. Once those are ready and available, the capacity displayed should be exactly the same as it was desired and displayed in the previous section with the diagram, except for the restrictions imposed by the Advanced section . Also, the set name should be the same as defined before.
Completing the storage system installation
The nodes selected in the previous screen should show up on the listing here. Check the resources of each and the zone – it’s essential to have the right resources available for the ocs-operator to be able to place all the pods on the proper nodes. Failing to do so could result in an inoperable cluster and would need changing the installation configuration and the nodes. The calculation for the initial nodes should match at least the following math:
- CPU per node: 0.5 vCPU for kubelet + 8 vCPU for ODF non-OSD pods + 2 vCPU per OSD
- memory per node: 2 GB for kubelet + 19 GB for ODF non-OSD pods + 5 GB per OSD
In my environment, this would give me 10 vCPU and 24 GB of memory. However, there are additional devices already configured to the nodes and once I would decide to expand the ODF cluster using those devices I would need more resources: CPU = 0.5 + 8 + (2 * 2) = 13, memory = 2 + 19 + (2* 5) = 31. Checking this on my screen reveals that there are more resources available than needed and that I’m good to go with the actual configuration of the nodes.
The step to taint the nodes would not be required nor would it be appropriate in my case: this would exclude all other workloads from these nodes. In my actual configuration, I use the worker nodes for deploying ODF and so I cannot taint the nodes without rendering the cluster unusable. In general, only when ODF is deployed on infra nodes but those should be used exclusively for ODF only but not for any routers, etc., then it’s feasible to taint the nodes.
The next steps then are
- the configuration of encryption that I leave here disabled, so neither using encryption of all the data on the OSD (aka, “data at rest encryption”) nor a per-PV encryption for encrypting PVs individually with different keys per namespace,
- the selection of the network scheme that I will use as it is which means to use the same network as with all the other workloads as well as the connections for the cluster internal functionality,
- then, the review of the configuration settings,
- and finally the start of creation of the ODF cluster.
After hitting the Create Storage System button, the screen will be switched as shown in the next picture:
Currently, there will be only one storage cluster listed but only one storage cluster is currently supported. Also, since there is no monitoring available yet and no traffic can be there, the numbers show essentially 0 for everything. Selecting the entry for the storage system from the table shows the Data Foundation overview page:
Initially, there is not much listed. One can see the green check mark for the successful deployment of the Data Foundation base. As there is no capacity listed yet, the configuration of the internal Ceph cluster is not yet ready. Switching to the Storage Systems tab displays another view which gives more information regarding the status of the ODF cluster. This is more specific and shows actual status, some recent alerts, the inventory of the cluster and other information:
At this stage without object storage being deployed yet, only Block and File screen is available. Later on towards the end of the installation process of ODF, the next screen shows that there is an additional tab available for Object. The internal Ceph cluster has been deployed successfully which is shown by the green check marks for the status of Storage Cluster and Data Resiliency.
Switching over to the Object tab, it should show the status of Object Service and Data Resiliency also as green.
Some last checks I can carry out: Checking on block pools, there should be one block pool configured, named ocs-storagecluster-cephblockpool. Now, I can see also the used capacity as well as the redundancy configured.
Selecting the Storage Classes entry from the Storage menu left hand in the OCP menu tree shows that there are new storage classes available.
The final review of the Data Foundation overview screen shows now some traffic happening on the storage cluster. This results from the Multi-Cloud Gateway component that is using a database and hence a PV backed by the Ceph cluster internally to ODF.
With this last check, I can consider the first ODF cluster installation finished and been successful.
Further work is needed to get a fully operational stretched cluster: nothing is yet installed for metrics and logging – these services need to be deployed, beside the integration with Single Sign On (SSO) and other services needed which I skip here for now.
In my environment, the registry is currently installed but is running on a RHV provided storage. To get a real failover capability, rather this should run on ODF as well. So, the PV used actually should reside on a RWX PV. Since we cannot relocate data from the actual used internal registry, we’ll gonna create a new PV and start using this. Because during this process we’ll also lose the data in the registry volume, this should be done before bringing real workloads onto the cluster.
The completion of a base installation for a fully functional OCP cluster using ODF is carried forward in the next article. Thanks for reading so far!