[11 min read]
Placing all the required instances with respect to failure domains and redundancy might become complex if the number of failure domains is reduced. In this article I continue a series of examples for collocation with respect to the failure domains, reflecting the recent versions of Red Hat Ceph Storage up to version 6. Newer versions of Ceph might slightly change the rules and will need double checking of possible enrichments of placement options.
The example of Case G (from Case B but across 2 failure domains)
Changing the environment compared to the previous one in “Collocation example – A complex example with 3 failure domains and replication of 3” using 3 failure domains to only 2 failure domains is the way to explore how most European customers might need to plan. The scenario can be also valid for cases in the US or other countries with local gravity within a city limit when wishing to have a local redundancy.Like in any of the more complex examples going forward, the simple formula for Sum_nodes_min from the initial article “Collocation and number of nodes in a Red Hat Ceph cluster” cannot be used. If there would be plenty of nodes available per failure domain, the distribution of the roles across the cluster nodes would be fairly easy. Like in the real world with limited resources, the task is here to find a way of minimizing the number of nodes required but also again to properly distribute the roles for redundancy.
The list of roles and configurations is as follows:
- 5 MONs,
- CephFS file system 1 with 1 active and 1 passive MDS,
- CephFS file system 2 with 2 active and 1 passive MDS,
- 2 different sets of redundant iSCSI gateways,
- 2 zones for RGW realm One with 2 instances each,
- 3 zones for multi-site replicated RGW realm – two with different networks and each one 4 RGW for higher throughput.
- The dashboard should be hosted by a VM.
Explaining the requirements and dependencies
In this special case with only two failure domains, we need to plan for a special configuration. Since we can lose one half of the configured instances with only losing one failure domain or the connectivity to one of the failure domains, all roles should have enough redundant peer instances in the other failure domain. The monitoring cluster should be able to withstand a DC outage but also needs to make sure that the accessible part of the cluster is also the one that is functioning properly while blocking for any access and changes of the remainder of the cluster.
Maintaining a proper quorum of the monitors is key for a 2 failure domain structure. For this to automatically determine the “right” and the “wrong” part of the cluster after a failure of one of the 2 failure domains, a 5th monitor is used as a tie-breaker instance. This instance needs to go into a special site with direct network connections into the other 2 failure domains. While stating that this is a configuration with only 2 failure domains, it means more precisely to have 2 main failure domains with all the OSDs and the functional roles residing …. but with an additional failure domain that must be independent of the infrastructure and location of the 2 main ones. As the 3rd location doesn’t carry any data nor provides it any access path to the cluster data nor allows access to the cluster through this 5th monitor for any client, the 5th monitor can reside wherever the requirements are met, perhaps also within a public cloud. The only additional requirement that must be met is that the network connection should be available to the remaining part of the cluster in the case of a failure domain outage.
The number of monitors required from the description above is 5 monitors. This perfectly matches the requirements for such a stretched cluster with 2 main failure domains: 2 monitor instances are required as minimum within the network perimeter of any of the main failure domains. If one of the failure domains becomes unavailable, the remainder of the monitors with 2 monitor roles in the other main failure domain plus in the 3rd failure domain can still form a proper quorum.
In contrast, aiming for a lower number of monitor roles like only 3 would be not supported: The monitors in this configuration would be configured based on the actual failure domain hierarchy of the OSDs, mirroring the OSD CRUSH map for the monitors. The essential dependency of this configuration is that the local monitor instances within a main failure domain are the ones caring about all the OSDs in this part of the cluster while the other instances in the peer part would only care about the others. In turn, if all monitors inside the same failure domain would be unreachable, all OSDs would be considered to be down and no data will be exchanged anymore from these until any of the monitors got repaired or replaced. Having only one per main failure domain would fail the half of the cluster if this instance would fail. Two monitor instances instead would provide redundancy.
As with only 3 failure domains, a set of only 2 failure domains wouldn’t provide a good environment for using Erasure Coding as the redundancy scheme for any data. The natural match in such a case would be the use of replica 4. The use of a replica 2 in such a layout is dangerous because with a failed main failure domain, only one replica is left and subject to failure with a simple media or node failure. A replica 3 would lack a proper redundancy for probably half of the data with no good prediction possible on which data would be affected by a failure domain outage. Replica 4 gives enough redundancy to withstand a main failure domain outage and an additional media or node failure: even then the single replica would be enough to be able to recreate at least 2 replica of the data, assuming that there will be another node available to take the additional data replica and it would have enough free capacity on the OSDs.
The only special roles are the iSCSI gateways that would require a single node per instance. Since we need to plan separate nodes for those, this would be the starting point for finding the proper distribution.
The MDS instances required per file system with two and three per file system can be simply distributed across both main failure domains. With the 3 MDS, two being active and one being passive, the active ones should reside in two different failure domains to allow backing of both active MDS.
Also the RGW instances, being here 2 or 4 per zone, can be easily placed. It looks like it might become an easy design.
Calculation for the failure domains
Using 5 monitor instances as described above, those could be distributed as in the table below. The FD3 is lined out in parentheses which should be the warning that this is not a fully equipped failure domain and should not take any other instances than the one of the 5th monitor as the tie-breaker.
| role type | FD1 | FD2 | (FD3) |
| MON | 2 | 2 | 1 |
Next will be the iSCSI gateways, since those will require individual nodes. Distributing the 4 instances in pairs across 2 failure domains is ending best with having one for each of the pairs in one of the main failure domains.
| role type | FD1 | FD2 | (FD3) |
| MON | 2 | 2 | 1 |
| iSCSI | 2 | 2 | – |
Moving over to the RGWs and MDSs as the last instances to place: This would require to evenly distribute the sets across FD1 and FD2 plus for MDS one in any of both failure domains for the passive MDS.
| role type | FD1 | FD2 | (FD3) |
| MON | 2 | 2 | 1 |
| iSCSI | 2 | 2 | – |
| RGW | 2+2+2+1+1 | 2+2+2+1+1 | – |
| MDS | 1+1+1 | 1+1 | – |
As all the instances are already placed, the calculation would be:
FD1: Sum_nodes_min = 2 + 0 +
max ( ROUNDUP(
( 2 + [2+2+2+1+1]
+ [1+1+1] + 0
) / 2),
max ( 2, (2+2+2+1+1), (1+1+1) )
)
= 2 + max ( ROUNDUP (13/2), max (2, 8, 3) )
= 2 + 8 = 10
and is pretty much the same for FD2. FD3 will have only one node as discussed before:
| role type | FD1 | FD2 | (FD3) |
| MON | 2 | 2 | 1 |
| iSCSI | 2 | 2 | – |
| RGW | 2+2+2+1+1 | 2+2+2+1+1 | – |
| MDS | 1+1+1 | 1+1 | – |
| sum | 10 | 10 | 1 |
Resulting configuration
The minimum number of nodes per failure domain is mainly dictated by the number of RGW required, because they need to run with one instance per node. If the environment could provide VMs for this task instead, the number of required nodes per main failure domain would come down to only 4 nodes.
In this article a more complex configuration was covered with only 2 main failure domains. Although it might be seen as a more complex example, the distribution of the instances is nearly as easy as in a single failure domain. The only important aspect to remember is the need for replica 4 for all the data.
Stating only the number of failure domains for a configuration is not the key to understanding the complexity of a design. Other configurations will be discussed in the coming articles, so be patient. Thanks for reading so far and stay tuned!