This topic describes determining requirements for infrastructure failure modes and recovery.
Before devising a strategy, you must first have a set of requirements to evaluate possible solutions. What failures are you trying to protect against, and what are your recovery goals in the event of failure?
Failure points
Before devising a strategy, you must first have a set of requirements by which the resulting solution can be evaluated. What failures are you trying to protect against, and what are your recovery goals in the event of failure?
Physical server failure
The Delphix Engine runs as a virtual machine on an underlying compute host. Failure of that physical host affects the Delphix Engine and any other virtual machines running on that host. The failure is isolated to that host and is not the result of a larger, site-wide failure.
-
Recommendation: Platform clustering or high-availability features
Storage failure
The Delphix Engine uses platform-attached storage devices, such as LUNs, managed disks, or equivalent block volumes. The storage system may include redundancy for disks or controllers, but the Delphix Engine can still be affected by an array-wide failure, storage path failure, or failure of storage devices assigned to the Delphix Engine.
-
Recommendation: Replication
Site failure
When an entire site or datacenter goes down, all servers, storage, and infrastructure are lost. This will affect not only the Delphix Engine but any production databases and target servers in the datacenter.
-
Recommendation: Replication
Administrative error
If an administrator mistakenly deletes a VDB or takes some other irreversible action, there is no method of recovery built into the Delphix Engine.
-
Recommendation: Snapshots
Recovery objectives
Once infrastructure fails, some amount of work is required to restore the Delphix Engine to an operational state. Clients won’t have access to the Delphix Engine during this time, and the point to which the system is recovered is dependent on the mechanism being used. These qualitative aspects of recovery can be captured by the following metrics. As these metrics are often directly associated with cost, it is important to think not just about the desired metrics, but also the minimum viable goals.
Recovery point objective (RPO)
The RPO is the acceptable amount of data that can be lost in the event of a failure. For example, if backups are taken once a day, then at most 24 hours of data will be lost if the system fails immediately before a regularly scheduled backup.
Recovery time objective (RTO)
The RTO is the time required to restore the system to an operational state after a failure. For example, recovery may require restoring data from a backup, followed by some number of manual steps to recreate the configuration in the new system. RTO is equivalent to the downtime experienced by clients.
Recovery time granularity (RTG)
The granularity of the recovery time is the specificity by which you can select a particular point in time from the past to restore the system. For example, VM snapshots taken every hour provide no way to restore to a point in time between those snapshots.