[12 min read]
The installation of OpenShift is thoroughly described but for the sake of completeness of the showcases, OpenShift installation will be the starting point in the lab environment. Because RHV is a supported environment, the installation would be an IPI installation – this would ease the deployment. The provisioning of block devices for OSDs of ODF is not affected by this decision and could either use the default storage class or use other resources that then need to be provisioned differently.
I’ll sketch only the basic installation steps for this environment, especially for the directly depending components. Where applicable and well explained somewhere else, I’ll jump over the details. Please, reach out to me for further details and wishes.
Configure DHCP and DNS for the OCP cluster
It’s kind of a no-brainer for experienced OCP admins and is mentioned elsewhere, already. The DNS and DHCP config might perhaps be needed for understanding the examples and so it’s covered here.
Since I’m more familiar with the old BINDv9 for DNS, the example uses a RHEL VM running BIND v9 for named. In this environment, the DNS server is a central one, serving all the different parts of the example environment. The snippet below shows the used config for this cluster named ocp31. Keep in mind that the reverse lookup config is not shown here, is needed for some of the components, but must not be configured for the nodes itself.
;
; ocp31 - 172.21.51.0/26 (0..63)
;
router31-int IN A 172.21.51.1
;
ocp31-jump IN A 172.21.51.2
ocp31-dhcp IN A 172.21.51.3
api.ocp31.dslab.local. IN A 172.21.51.6
*.apps.ocp31.dslab.local. IN A 172.21.51.7
ocp31-console IN A 172.21.51.8
;
;; ocp31-n1 IN A 172.21.51.11
<-- content omitted -->
;; ocp31-n18 IN A 172.21.51.28DHCP is needed for the installation of the OCP cluster. The configuration of the DCHP server, running on the jumphost, is:
[root@ocp31-jump ~]# grep -v '^#' /etc/dhcp/dhcpd.conf |tr -d '^$'
option domain-name "dslab.local";option domain-name-servers router00.dslab.local;
default-lease-time 600;
max-lease-time 7200;
authoritative;
log-facility local7;
subnet 172.21.51.0 netmask 255.255.255.192 {
range 172.21.51.11 172.21.51.28;
option domain-name-servers router00.dslab.local;
option domain-name "dslab.local";
option routers 172.21.51.1;
option broadcast-address 172.21.51.63;
default-lease-time 600;
max-lease-time 7200;
}
[root@ocp31-jump ~]#Installing the OCP cluster
The jump host is also the location for kicking off the installation. This is a manual deployment using an OCP supported environment and so I can use the openshift-install tool to prepare the OCP cluster.
First, one needs to download the tool and the CLI tool. Both are available at the Red Hat site. In addition, one can download the pull secret from the same site. Ensure to use the tools matching the OS and architecture used to deploy the OCP cluster from, in my case being Linux on x86_64 architecture for the jump host.
[root@ocp31-jump ~]# curl -o openshift-install-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-install-linux.tar.gz
<-- output omitted -->
[root@ocp31-jump ~]# curl -o openshift-client-linux.tar.gz https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz
<-- output omitted -->Once download succeeded, unpack all, move the CLI tools into a path enabled for execution (here, picked from PATH variable). I leave the openshift-install tool within the directory, since it’s only used for the cluster initial installation.
Then it’s time to create the configuration for the OCP cluster deployment. I need to adjust the installation config to reflect my special setup of the nodes. The picture below shows the desired configuration with the sizing of the worker nodes changed. Note that the exact node names will be configured by the installation tool; only the masters have the name of the exact role applied, like *-master-0.
Right from the start, the worker nodes should be configured for use with ODF, requiring much more powerful CPU and memory configuration than the default settings provide. To apply those changes, I create first the install configuration in order to change those settings.
[root@ocp31-jump ~]# ./openshift-install create install-config --dir=ocp411
? SSH Public Key /root/.ssh/id_rsa.pub
? Platform ovirt
? Cluster Default
? Storage domain Data2
? Network net_211
? Internal API virtual IP 172.21.51.6
? Ingress virtual IP 172.21.51.7
? Base Domain dslab.local
? Cluster Name ocp31
? Pull Secret [? for help] *******
<-- truncated -->In the lab environment, there is only a single RHV host. To be able to properly deploy the OCP cluster not being spread across virtualisation hosts, I need to tweak the default configuration to not enforce the affinityGroups, for both the compute and the worker nodes.
[root@ocp31-jump ~]# vi ocp411/install-config.yaml
<-- content omitted -->
platform:
ovirt:
affinityGroups:
- description: AffinityGroup for spreading each compute machine to a different
host
enforcing: false
name: compute
priority: 3
- description: AffinityGroup for spreading each control plane machine to a different
host
enforcing: false
name: controlplane
priority: 5
<-- content omitted -->After adjusting this, the manifests for the OCP installation needs to be created to allow adjusting the settings for the node sizes of the worker nodes. Control plane nodes doesn’t need to be modified for this small cluster but the configuration could be changed for those the same way.
[root@ocp31-jump ~]# ./openshift-install create manifests --dir=ocp411
INFO Consuming Install Config from target directory
INFO Manifests created in: ocp410/manifests and ocp410/openshift
[root@ocp31-jump ~]#
[root@ocp31-jump ~]# vi ocp411/openshift/99_openshift-cluster-api_worker-machineset-0.yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
<-- output omitted -->
spec:
replicas: 3
<-- output omitted -->
spec:
<-- output omitted -->
providerSpec:
value:
<-- output omitted -->
cpu:
cores: 16
sockets: 1
threads: 1
memory_mb: 65536
<-- output omitted -->At this point, it might be a good idea to save the content of the configuration directory used to avoid redoing the previous configuration and changes if things got misspelled or are incomplete.
[root@ocp31-jump ~]# cp -rp ocp411/ ocp411--configNow it’s time to finally install the cluster. I prefer to switch on the logging to be able to identify the steps the installation fails at if something wasn’t set properly. Also, redirecting the input and decoupling from current terminal connection might be a good idea, since the installation could take 30 minutes and cannot complete if the controlling session gets disconnected.
[root@ocp31-jump ~]# nohup ./openshift-install create cluster --dir=ocp411 --log-level=debug &After the installation finished, the OCP cluster should be available for login and should have 3 control plane nodes and 3 worker nodes.
[root@ocp31-jump ~]# oc login -u kubeadmin https://api.ocp31.dslab.local:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y
Authentication required for https://api.ocp31.dslab.local:6443 (openshift)
Console URL: https://api.ocp31.dslab.local:6443/console
Username: kubeadmin
Password:
Login successful.
You have access to 67 projects, the list has been suppressed. You can list all projects with 'oc projects'
Using project "default".
[root@ocp31-jump ~]#
[root@ocp31-jump ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
ocp31-5zmws-master-0 Ready master 29m v1.24.6+5658434
ocp31-5zmws-master-1 Ready master 29m v1.24.6+5658434
ocp31-5zmws-master-2 Ready master 29m v1.24.6+5658434
ocp31-5zmws-worker-8rx58 Ready worker 20m v1.24.6+5658434
ocp31-5zmws-worker-hsbzq Ready worker 20m v1.24.6+5658434
ocp31-5zmws-worker-mg88r Ready worker 17m v1.24.6+5658434
[root@ocp31-jump ~]# Applying the failure domain information for ODF
The OpenShift cluster itself has no preference for the topology by default, neither is it aware of the layout of the nodes if not deployed into a structured environment like into zones of availability within a cloud. As I’m not in a cloud but the OpenShift cluster should be deployed across three different zones, I will need to apply the topology information manually to the nodes. This would also be applicable to Bare Metal installations or any other kind of environment without the information applied automatically.
[root@ocp31-jump ~]# oc label node ocp31-5zmws-master-0 topology.kubernetes.io/zone=DC1
node/ocp31-5zmws-master-0 labeled
[root@ocp31-jump ~]# oc label node ocp31-5zmws-master-1 topology.kubernetes.io/zone=DC2
node/ocp31-5zmws-master-1 labeled
[root@ocp31-jump ~]# oc label node ocp31-5zmws-master-2 topology.kubernetes.io/zone=DC3
node/ocp31-5zmws-master-2 labeled
[root@ocp31-jump ~]# oc label node ocp31-5zmws-worker-8rx58 topology.kubernetes.io/zone=DC1
node/ocp31-5zmws-worker-8rx58 labeled
[root@ocp31-jump ~]# oc label node ocp31-5zmws-worker-hsbzq topology.kubernetes.io/zone=DC2
node/ocp31-5zmws-worker-hsbzq labeled
[root@ocp31-jump ~]# oc label node ocp31-5zmws-worker-mg88r topology.kubernetes.io/zone=DC3
node/ocp31-5zmws-worker-mg88r labeledLabelling the control plane nodes is optional since by default ODF pods will not be deployed to control nodes.
Preparing the manually provisioned devices for ODF
In a production environment, the devices backing the OSDs could be standard local media or some type of fast storage. In virtualised environment, one could use any kind of local media, virtualised disk, or raw device mappings from an enterprise storage. In bare metal, usually one would use plain devices, like a whole SSD or NVMe, but also the use of existing enterprise storage systems is supported.
In cloud environments, the choice is usually limited for suitable storage flavours and it might depend on the type of node, too. In virtualised environments with traditional on-premises infrastructure, more options could be available, like a directly attached storage locally consumed in the hypervisor, a SAN based enterprise storage consumed with distinct volumes per OSD, the same with a shared used volume provided by the enterprise storage system, or a locally mounted media. There are dependencies for the infrastructure in terms of reliability, complexity, handling, maintenance, and of course performance and latency, and cost.
When using enterprise storage as the backing resource of the OSDs, one should keep in mind that there might be multiple challenges: multi-pathing configuration and support, concurrency with foreign workloads, performance dependency with a high-density traffic pattern, and perhaps single systems for backing the 3 replicas for redundancy. All these have an influence on the performance and reliability and might be discussed further in a later blog article.
In my lab environment, I use locally provided devices based on NVMe internal to the hypervisor host. Because of the nature of local storage I need to manually provision the devices to the VMs. Those carved out of the NVMe as a portion of the capacity to allow other workloads later on using it as well.
The successful attachment of the devices can be checked on the nodes directly. Using the debug option of the oc command, commands can be issued to the underlying host operating system. I expect on every worker node an additional 100 GB device, being unused and available for the ODF cluster.
[root@ocp31-jump ~]# oc debug node/ocp31-5zmws-worker-8rx58 -- chroot /host bash -c 'lsblk'
<-- output omitted -->
Starting pod/ocp31-5zmws-worker-8rx58-debug ...
To use host binaries, run `chroot /host`
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 120G 0 disk
|-sda1 8:1 0 1M 0 part
|-sda2 8:2 0 127M 0 part
|-sda3 8:3 0 384M 0 part /boot
`-sda4 8:4 0 119.5G 0 part /sysroot
sdb 8:16 0 100G 0 disk /var/lib/kubelet/pods/c993dd6a-4715-446a-b638-a8d121ad73ab/volumes/kubernetes.io~csi/pvc-c4707f14-b9fc-469a-81a2-5aa27cf7671d/mount
sdc 8:32 0 100G 0 disk
sr0 11:0 1 374K 0 rom
sr1 11:1 1 1024M 0 rom
Removing debug pod ...
[root@ocp31-jump ~]# One can see that there is a 100 GB device available, shown as /dev/sdc. It also shows another disk of 100 GB capacity being in use. The latter one is a device deployed by the installer to host the PVC for the internal registry. This is only on one host and thus other worker nodes will not have an additional device and will show my unused 100 GB device as /dev/sdb. This is important for the next step.
In some virtualised environments, the virtualised devices might not show up with the proper type of the media. In my case, while having an NVMe as the underlying device and being clearly a flash media, it’s not recognised as such by the operating system:
[root@ocp31-jump ~]# oc debug node/ocp31-5zmws-worker-8rx58 -- chroot /host bash -c 'cat /sys/block/sdc/queue/rotational'
<-- output omitted -->
Starting pod/ocp31-5zmws-worker-8rx58-debug ...
To use host binaries, run `chroot /host`
1
Removing debug pod ...
[root@ocp31-jump ~]# It’s clear that the operating system found it to be a rotational device, so an HDD. While there will be no difference in terms of capacity, first the scheduler used will be different and second the Ceph code tree being used will not be the one tuned for higher performance with flash media. This would be a huge difference that one would quickly recognise during self-heal and rebalancing operations in the ODF cluster if not already suffering from the high latency with applications. The reason for not recognising the flash media properly is based on miscommunicated device capabilities by the underlying hypervisor and its framework. However, changing this in the OS is only possible by changing the flag if the virtualisation solution cannot provide the correct information upon provisioning.
To correct this, I will change the setting for these devices to be rotational from now on. This is done here manually per node. For production deployments with machine configuration applied, there is a KCS describing the workaround.
[root@ocp31-jump ~]# oc debug node/ocp31-5zmws-worker-8rx58 -- chroot /host bash -c 'echo 0 > /sys/block/sdc/queue/rotational; echo ACTION==\"add\|change\", SUBSYSTEM==\"block\", KERNEL==\"sd[b-z]\", ATTR{queue/rotational}=\"0\" > /etc/udev/rules.d/90-non-rotational.rules'
<-- output omitted -->
Starting pod/ocp31-5zmws-worker-8rx58-debug ...
To use host binaries, run `chroot /host`
Removing debug pod ...
[root@ocp31-jump ~]#This changed setting must be applied on all worker nodes with a change to the name of the device being /dev/sdb on the other nodes in my cluster. Note that I chose to apply this setting for all current and future devices based on /dev/sd* pattern, except for the boot device on /dev/sda. The device for the registry in the first of the worker nodes used for the registry will be perhaps removed – at least not being used anymore – when all internal services as the registry, metrics and logging being moved to use ODF based PVs. However, in my lab environment this wouldn’t hurt anyway because all devices based on flash media and thus I could even change the pattern to include /dev/sda as well.
For proceeding, I configured additional devices that now should show up as flash devices.
This concludes the installation of the OCP cluster for the ODF deployment that will be covered in the next blog article coming soon.