[13 min read]
In the series of articles about the collocation rules for placing the Ceph roles properly across failure domains, previously only examples of clusters with a redundancy scheme of replication were discussed. This article now takes it to the next level, exploring the constraints of using Erasure Coding as a valid redundancy scheme for some use cases and workloads.
The example of Case K (from Case B but across 7 failure domains)
This case is a variation of Case B, introducing Erasure Coding, the 7 failure domains, and additional layout information. The configuration information is as follows:
- 5 MONs,
- CephFS file system 1 with 1 active and 1 passive MDS,
- CephFS file system 2 with 2 active and 1 passive MDS,
- 2 different sets of redundant iSCSI gateways,
- 2 zones for RGW realm One with 2 instances each,
- 3 zones for multi-site replicated RGW realm, two with different networks and each one 4 RGW for higher throughput,
- the dashboard should be hosted by a VM,
- the clients for iSCSI reside in the failure domains 1 and 2,
- where supported, Erasure Coding 4+2 should be used to save on space but also a reliable configuration should be created,
- the second zone of the replicated RGW realm is accessed by clients only residing in failure domains 3-5 and has a different network.
Explaining the requirements and dependencies
Analyzing the types of roles and additional hints for the configuration I’ve got reveals that
- I don’t need to take the dashboard into account
- 7 failure domains are available
- the iSCSI gateways are special and in sum there are 4 gateways – those should go into failure domains 1 and 2 because the clients would then possibly leverage local access with lower latency and wouldn’t add traffic to the network connections more than the traffic between the OSDs or between the iSCSI gateway and the OSDs
- Erasure Coding is supported for non-archival workloads for the object storage data pools. All other pools probably will get a replica 3 configuration. Since there is no special requirement for the media to place, the assumption is that those will be equally distributed across all failure domains.
- for RGW, there are 5 sets of RGW to provide, with different numbers of required instances per set.
- for one zone of the replicated realm, there is a locality hint given. The one set of RGW could be placed into the failure domains 3-5 which I can check to be possible.
- I’ve got two separate CephFS file systems to create = 2 use cases for MDSs. In sum, 5 MDS need a location.
The MONs could be distributed across 5 failure domains. The 2 other failure domains would have no MON to host. This is totally ok and could be leveraged to use the 2 failure domains without a MON placement to host some other roles, like the iSCSI gateways, perhaps.
The CephFS file systems have active and passive MDS. The 2 active MDS of filesystem 2 should not be placed into the same failure domain, already from the configuration given with only one passive MDS covering two active MDS. I should make sure that only one of the MDS will need a stand-in in case of the unavailability of one of the failure domains with the active MDS.
Both CephFS filesystems MDS could be in the same failure domains as the other’s ones. From a reliability perspective and performance point, two MDS serving different file systems is possible. So, I don’t have any additional restriction from these two independent file systems and can freely assign the MDS to any of the failure domains.
The iSCSI gateways should serve clients in failure domains 1 and 2 and the best latency probably they will get when the gateways and clients reside in the same failure domains. The goal of the distribution should be to place the iSCSI gateways in pairs into those failure domains. Having two sets leaves no further reliability requirement open – in case all 4 gateways would serve the same set of volumes, perhaps one would rather distribute the half of the iSCSI gateway instances across additional failure domains. In the case with 2 independent sets there would be no additional benefit to do so but only would add additional latency if the access to the instance would be across failure domains.
The RGW realm One has two zones, each one with two RGW instances. Those could go into any of the failure domains, distributed in pairs. Even the RGW from both zones can go into the same failure domain, since there is no additional benefit for the realm. The access is based on zones and also the administrative access is performed through only the master zone.
For the replicated realm with the 3 zones, one zone is special and could be placed into the used failure domains 3-5. Given that this zone also has one of the different networks to serve, the association into those failure domains might make sense – in the assumption that this special network is available there. The rest of the RGW instances can be placed in any failure domain, also into failure domains 3-5, since those serve different zones. However, all RGW instances of a given zone might rather be distributed across the failure domains for redundancy.
Erasure Coding 4+2 scheme would require 7 independent failure domains to provide an additional failure domain over the required 6 ones which is possible here. In addition, there should be an additional node per failure domain to the number required for the capacity. In this configuration, no number of nodes based on capacity is given and so I only note the option that an additional node might be needed.
Calculation for the failure domains
For placing the roles, different options are available. Because there are some possible optimizations, this could be an iterative process:
I start with the iSCSI gateways, since those will require individual nodes. Since no other roles would have special benefit to land in the failure domains 1 and 2, all others potentially could go into the failure domains 3-7.
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
Next, there is a set of RGW instances that could potentially serve one zone better from being in the failure domains 3-5. Since there are 4 RGW instances required, those could now be distributed with 2 pairs going into FD3 and FD4. However, the special network is available also in FD5 as the clients are accessing those instances from this failure domain. Distributing the RGW instances over all three failure domains might provide one hard and one soft benefit: The hard benefit would be that the access from this special network can be validated always if all failure domains would have an active instance serving traffic there. The soft benefit would be some potential portion of local traffic. The 4th node might be placed in any of the three failure domains if I would stick with these.
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 0 | 0 | 2 | 1 | 1 | 0 | 0 |
Next one could be the MON placement. Since failure domains 1 and 2 already need to provide at least 2 nodes each without allowing additional roles on those nodes and I need to distribute 5 MON instances, those could go into the remaining failure domains:
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 0 | 0 | 2 | 1 | 1 | 0 | 0 |
| MON | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
Before I can distribute the remaining RGW instances, MDS instances would need to be cared about, since those have some dependencies but only for redundancy.
MDS instances are in sum 5, with no special affinity to any failure domain. The table actually shows FD1 and FD2 with two exclusive nodes and FD3 having the most instances assigned from the scale-out daemons. FD3 has not yet filled 2 nodes whereas FD4 – FD7 have fewer role instances to deploy. I go with placing the MDS into the failure domains 3-7, exactly giving me room for full redundant distribution of the MDS instances:
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 0 | 0 | 2 | 1 | 1 | 0 | 0 |
| MON | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| MDS | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
The remaining roles to distribute are only RGW instances. Those are 2 pairs for the realm One and two zones for the replicated realm with 4 instances each. This sums up into 2+2+4+4=12 instances. I can do a random distribution but could also check for the number of nodes actually required already and check the best distribution to limit the number of required nodes per failure domain. So, I’ll do the computation for the actual values now. For illustration, I pick FD1 and FD3 and add all the results into the table following it. The remainder of the calculations for the other failure domains will follow the same formula used before.
FD1: Sum_nodes_min = 2 + 0 +
max ( ROUNDUP(
( 0 + [0]
+ [0] + 0
) / 2),
max ( 0, (0), (0) )
)
= 2 + max ( ROUNDUP (0/2), max (0, 0, 0) )
= 2 + 0 = 2
FD3: Sum_nodes_min = 0 + 0 +
max ( ROUNDUP(
( 1 + [2]
+ [1] + 0
) / 2),
max ( 1, (2), (1) )
)
= 0 + max ( ROUNDUP (4/2), max (1, 2, 1) )
= 0 + 2 = 2
The actual required number of nodes has been added in the below table as the last row:
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 0 | 0 | 2 | 1 | 1 | 0 | 0 |
| MON | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| MDS | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| sum | 2 | 2 | 2 | 2 | 2 | 1 | 1 |
The additional RGW instances might be added first and foremost to FD6 and FD7, then FD4 and FD5 as those have only 3 scale-out daemons to host and would have one “slot” unused. So, I start by filling up those failure domains with RGW instances: 2 go into FD6, 2 into FD7, and one each into FD4 and FD5. This way I can place 6 RGW instances and still have 6 instances to place.
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 0 | 0 | 2 | 1+1 | 1+1 | 2 | 2 |
| MON | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| MDS | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| sum | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
Those remaining 6 instances I could place into any of the failure domains now. Adding one per failure domain would leave one failure domain with 2 nodes only. For the equal capacity distribution, all failure domains should have the same number of nodes, especially in smaller clusters. Spreading the RGW instances across all the failure domains would raise the number of required nodes to 3 nodes for all, but one. Alternatively, I could pick only 3 failure domains to place the roles there – but with some caveats:
- It might make it harder to pick the instances of a zone from the right failure domains and maintain it over time.
- Those picked 3 failure domains would be exhausted with the available “slots” for collocated scale-out daemons.
Leaving a “slot” empty on most of the failure domains would provide more flexibility to place a replacement instance. In addition, spreading the RGW instances evenly would leave one failure domain with an empty node. This one could be used to add a replacement instance for an iSCSI gateway, still. My final recommendation for the placement of roles would be to have 3 nodes per failure domain as the minimum number of nodes:
| role type | FD1 | FD2 | FD3 | FD4 | FD5 | FD6 | FD7 |
| iSCSI | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| RGW | 1 | 1 | 3 | 3 | 3 | 3 | 2 |
| MON | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| MDS | 0 | 0 | 1 | 1 | 1 | 1 | 1 |
| sum | 3 | 3 | 3 | 3 | 3 | 3 | 2 |
Resulting configuration
The graphical representation of distribution of the roles is now more complex but the following picture shows the configuration of all the 7 failure domains. If more nodes would be required for providing the desired capacity, a more relaxed distribution of the roles across the available does per failure domain would result – but keep in mind that it’s always good to leave one of the nodes always empty of collocated roles to be available for the ones that need to live alone.
For the reason of an equal capacity distribution to be able to withstand a node failure and still being able to accommodate all the capacity from the failed node, the number of nodes should be equal across all failure domains. This is incorporated in the picture below.
This article covered a more complex configuration with Erasure Coding in the scope but also additional infrastructure information that was adopted into the design proposal. Because of the more complex dependencies to respect, the calculation is based on an interactive process that is common for more complex use cases and infrastructure characteristics. The 7 failure domains can be understood as 7 different racks in a data center which is easy to find in the real world, illustrating that such designs might not be that artificial as it initially sounds.
Going forward, additional sub-level failure domain examples and restricted environments will be discussed in the coming articles. Thanks for reading so far and stay tuned!