[8 min read]
Many deployments assume that the failure domains can be equally treated. In reality, very diverse failure domains may exist in the same environment. Those have been built and changed over time and because of different influences those might not be that independent anymore. This article will walk through another example of collocation placement rules but with a scenario of unequal failure domains.
In this potential environment, three main failure domains have been identified. After further investigation, 2 of the failure domains reveal that there are additional structures. The main failure domains might be data centers but those are structured into different racks, at least for two of them.
The actual layout was designed for a 3 main DC layout and might have been established already. During the installation, two racks in the data center were found to be connected in a different networking structure and also the power grid was different for both.
The example of Case E moved to Case L
A simple cluster with 3 MONs, 2 MDS, 2 iSCSI gateways, but 2 different zones for RGW should be created, and the dashboard server should be installed on the cluster nodes. Because the two different RGW zones should be supported with redundant instances, we would need 4 instances, two for each zone. Each of the zones would represent a use case for us. This is exactly the example called Case E from “Collocation examples – Simple one: 3 failure domains – replication of 3“.
The result for 3 failure domains originally was:
| role type | FD1 | FD2 | FD3 |
| MON | 1 | 1 | 1 |
| iSCSI | 0 | 1 | 1 |
| RGW | 1+1 | 1 | 1 |
| MDS | 1 | 1 | 0 |
| dashboard | 0 | 1 | 0 |
| sum | 2 | 3 | 2 |
With the approach for 3 failure domains, in this case, the roles can be easily distributed using the same DCs but only using a single rack per DC. The actual number of nodes might fit into the free space properly. Something that should be avoided would be to distribute the nodes of e.g. FD2 and FD3 into different racks without further redesign.
In this example, checking the FD2 servers for their actual roles hosted, those could also be distributed across the racks in FD2 because they all run independent roles. Also for the FD3, the one server with the iSCSI gateway could easily live inside the other rack.
For the data redundancy and correct placement of the replica, the rack infrastructure must be treated as additional sub-level failure domains. Hence, the distribution of the replica should be based on the top-level failure domains. If one of the racks fails, the remaining rack continues but serves different data. Both racks further belong to the same DC and in the case of a failure, the DC would be the failure domain having an outage but the racks are included as well.
An alternate configuration and still being correct could then look like this:
The example of Case E moved to Case M
In contrast to the above example of case L, in another incarnation, the two DCs with the additional rack structure were found to depend on the same power grid and cooling. While the network being on separate paths connected and thus could be treated as independent, at least because of the power dependency those should be treated as a single failure domain. This changes the whole redundancy discussion: There would be only two top-level independent failure domains: FD1 and the combination of FD2 and FD3.
For the reason of fire separation, one could try to treat FD2 and FD3 as failure domains. However, if roles would be placed in FD2 and FD3 only, there would be no true redundancy. Having replica 3 as the redundancy scheme for the data would, however, allow for spreading the data properly but would also risk data unavailability if another failure would occur on the node or media in FD1 in case of a failure of the power or cooling for the combined set of FD2 and FD3.
The proper design for this environment would be with 2 main failure domains plus a 3rd site for placing the tie-breaker. It’s not a solution to ignore the situation but to place the tie-breaker monitor then into one of the FD2 or FD3 DCs. Instead, there would be a strong need for an additional failure domain, at least one that could host the 5th monitor, and a reliable network connection too.
The configuration would then need a change as well: First and foremost, there should be 5 monitor instances compared to the actually planned 3. The other roles need to be redistributed as well. A possible configuration is calculated below.
| role type | FD1 | FD2 | (FD3) |
| MON | 1+1 | 1+1 | 1 |
| iSCSI | 1 | 1 | – |
| RGW | 1+1 | 1+1 | – |
| MDS | 1 | 1 | – |
| dashboard | 0 | 1 | – |
| sum | 4 | 4 | 1 |
Comparing this result with the result for case L, the changed environment of failure domains will not only change the overall configuration of the cluster but will lead to important changes to the overall behavior. First of all, more servers are needed. The introduction of a 2 main DC concept will change the role of the local monitors and the dependency on those for the whole failure domain. This is not a challenge for this tiny configuration example but might be more interesting in larger deployments. Because of the data redundancy layout change, the requirement for 4 replicas will drive the costs up. In addition, seeing RGW as a use case, the default data redundancy scheme of Erasure Coding is not an option. This might be an impact or might be not, depending on the major object size range of the objects stored: for smaller objects, replication would be better to avoid capacity waste, but for larger objects, Erasure Coding would be better to save on capacity. Last but not least, the 3rd location might introduce additional challenges since it’s not only about a location to provide but also all the connectivity and access that must be sorted out.
The check of the failure domains might reveal the need for a different approach of properly designing the Ceph cluster layout than assumed at a first glance. Failure domains might be fairly tricky if complex infrastructure dependencies exist.
For getting the round-up, make sure to check also the next article that will conclude the series for now. Thanks for reading so far and stay tuned!