[15 min read]
In this series of articles, minimizing footprint of Ceph clusters while respecting the collocation rules and the failure domains in enterprise infrastructures was explained. While reducing the material effort for providing a reliable data layer is of special interest for most teams because of physical space, network ports, power, and cooling, some other factors might be of interest to design for robustness of data and runtime effort.
In limited environments too, alternate approaches can help to still leverage the power of Ceph while taking compromise on availability. In large environments, the effort of maintaining the hardware influenced work might be of special interest. This article explores some additional suggestions for those challenges.
Using replication and Erasure Coding with 3 DCs
Erasure Coding is an easy way for saving capacity, power, and cooling and can drive down the costs for storing data. Replication has its benefits on the reliability side but also the downside of the costs. In the discussion of the role and data placement in the article Advanced Collocation – Data Placement Constraints, it was explained that there are additional requirements for using Erasure Coding in a secure way. With respect to failure domains, the need of having more failure domains was explained. In conjunction with the use cases, additional dimensions might influence the system design.
Storing data using a replication would create a full replica of any data object. If any of the replicas become unavailable, the remaining ones provide the full data without any dependency on other media or nodes. With an accurate placement of the replicas across independent failure domains, the ability to read the data back is given even with a double failure of let’s say a top-level failure domain outage and a subsequent failure to a node or media or really the second failure domain becomes unavailable: the data is still accessible – with the assumption that there is still a working monitor quorum. However, even if the cluster becomes unresponsive, if the monitor quorum can be reestablished in any way, the access to the data is possible. No data loss would occur.
Erasure Coding is adding coding chunks with a computed subset of the data based on the idea to be able to reconstruct any missing data chunks without creating full copies of it. During writes, the data is split into a number of data chunks and the coding chunks are added. Under normal circumstances, all the chunks will be written to a set of top-level failure domains. Those should provide enough space to store the chunks within the regular configuration. With 3 DCs and the use of EC 4+2 scheme, this would create ideally 2 chunks in each DC. If one of the DCs would become unavailable, the other DCs would still provide a sufficient number of chunks to recreate the data and provide the data to the client. Also, the 4 chunks would be sufficient to recreate the missing chunks for full data redundancy. In the discussion in the earlier article, it was mentioned that recreating the missing chunks would need a location to store it. With only 3 failure domains this wouldn’t be possible. This inability to find a place to store the missing chunks would not only affect the ability to self-heal the data but would also affect the ability to write new data at all. By writing only 4 chunks out of 6 desired it would be possible to create the full data in the cluster, however, without any redundancy. Losing a node or a media is then not an option but also losing data while waiting for the repair of the missing failure domain will be just a matter of time. Since the cluster pretended to the client to have stored the data, the base assumption on the client side is that the data is there – losing anything later on is not an option. Instead, with such missing redundancy, the upload or write of data should be stopped until the data redundancy can be guaranteed, at least for a first failure.
A dramatic relief for using EC in 3 DC environments would be the use of the upcoming EC 2+2 scheme. It’s not about the ability to use it in 2 DC environments: If one half of the chunks is missing due to network connectivity issues, only a media or node failure would provoke a data loss. While this is similar to the event of losing a media or a node in a EC 4+2, the difference would be that any of the 2 DCs would have a full set of the data. This is also similar to the replica 2 but with the benefit that losing 2 media or nodes out of the 4 used to store the data would still provide the full data. While replica 2 will hit hard if both used media or nodes would fail at the same time, the EC 2+2 scheme would be not affected at all with the same likelihood for this kind of event. It would be more safe.
The best benefit of using EC 2+2, however, would be that it would provide a very robust use of Erasure Coding within a 3 DC environment. With the need of only writing 4 chunks, 2 chunks could be kept together at any time in a single failure domain, leaving the third failure domain then as the alternate location for recreating or writing the needed chunks. Of course, the free capacity to do so must be available in the right amount needed.
Example of picking a set of main failure domains in the lack of sufficient number of DCs
In many IT infrastructures, non-sufficient numbers of equal DCs are available but still the data redundancy should be given across the different failure domains. The best way would be then to leverage the built-in mechanisms for replicating data into another cluster. While this might be the recommended approach, the desire might still be to have a stretched cluster covering at least the simple issues without data loss. This is the time when careful selection of creating virtual failure domains might be a solution while accepting that a failure to the main part of the structure might still provide the data but might block any changes until sufficient fixing is applied.
An example user might have only 1 main DC with fully redundant everything within this top-level failure domain but also could provide two other (minor) DCs that cannot be utilized the same way because of limited space, cooling, or power, etc. In the assumption that this main DC is really redundant in any way and could be considered mature enough for their risk understanding, at least a deviation from the standard might be used for the EC based pools.
Replication should be done using replica 3 and thus all data based on replica should be stored in all the 3 DCs. Since we need to assume that either a media, node, or a data center might fail in this scenario, using replica 2 only would not be a good idea. Alternatively, when using replica 3, also 2 replicas could be stored into the main DC whereas providing the data availability in case of a disaster in the main DC with the additional replica in one of the other minor DCs. Even if the main DC would fail in this case, the full data objects are still available in one of the other minor DCs.
For the Erasure Coding based pools, one could instead consider the following: if one could place the majority of the chunks needed into the main DC but also place sufficient number of chunks in any of the minor DCs, one could always write new data and recover from a minor DC failure. With EC 4+2, 4 chunks could go into the main DC and 2 could go either into minor DC A or minor DC B. If any of the minor DCs fail, the additional chunks could be created in the other remaining minor DC. The caveat with EC is here that if the main DC fails, no data can be provided until sufficient repair happens and the minimum number of actual chunks can be accessed again.
Obviously, this example configuration is not something that should be the norm and would require a support exception
Working towards a fully capable cluster design
In all the articles about the collocation rules, the aim was to find a minimum number of nodes to provide the required services. It was the foundational assumption that the listed number of instances per role was sufficient for the use cases. In most of the cases, this would not be an issue with regards to redundancy of the service if a failure domain would have gone down.
In designs involving more failure domains, like the one of the case K with 7 failure domains, the randomness of independent failures must be taken into account. In the case K example, there were two zones with only 2 RGW instances. The two instances give the redundancy in the event of a failure of any of both failure domains they live in. With 7 failure domains, however, the likelihood of experiencing a failure of exactly those two ones is higher than in a 3 failure domain environment. This should be reflected in the design and adding more instances across more failure domains for both the zones would be a good way to mitigate the failure likelihood.
In large clusters, a lot of components may fail. While servers might be repaired, network cards might be replaced, one of the most exposed components is the media itself. One cannot repair a failed media, usually. Instead, a replacement part will be used but the data itself is gone. Ceph cares for this kind of data loss and will recreate the missing redundant parts – without the part being replaced – in other media or nodes or after the replacement of the part. With lots of media used, many of those can fail at any time. The hardware vendors have a curve for the likelihood of failures over time and this gives the highest likelihood of a failure during the first to 1.5 years after start of the use and towards the end of the lifetime of the media. Perhaps, replacing the parts during these times of elevated exposure might be a time consuming job and might be challenging. In HPC environments, this has been the norm and one of the unneeded workloads for the data center people. Also in terms of redundancy, having many failures during a short time frame, it’s a challenge when it comes to part shortage.
To mitigate the impact of frequent media failures, a good design, while being more costly, could incorporate additional media right from the start. Additional nodes would be perhaps required for this already, but adding nodes for providing more “spare” components would ease the handling in case of any failure. Especially in large environments, this could simplify the action plans if one could deem the Ceph cluster safe because of a headroom for accommodating failures. The failed nodes could be replaced over time and also nodes could be replaced only when too many media failed – for new models or just in one effort to “repair” those nodes by replacing media in one rush.
Respecting failure domains with regards to redundant roles is just one part of the proper design. While collocation rules might give some restrictions to minimize the number of nodes in a Ceph cluster for production, additional constraints might be observed by the structure and dependencies on different kinds and layouts of the infrastructure for IT environments. For a good design, this will often dictate the minimum number of nodes but also labor and maintenance efforts might be taken into consideration.
In this series of articles with examples, many possible decision points were touched. With this article covering some more high level thoughts that cannot be counted by numbers always, the series concludes. If anything should be added to enhance the picture for collocation rules in practice, please reach out to me with your suggestions. Also I’m welcoming your contributions to the blog.