Configure replication

This topic describes how to configure data replication between Delphix Engines. Replication is configured with Replication Profiles that contain options such as the replication schedule, the hostname of the target Engine, and the selected objects that will be replicated.

Requirements

• Version requirements: the replication target can be on the same or newer version than the replication source.

• Engine communication: The target Delphix Engine must be reachable from the source Engine.

• Storage allocation: The target Delphix Engine must have sufficient free storage to receive the replicated data.

• Privilege requirements: The user in the replication profile must have administrative privileges on the source and the target engines.

Configure the network

Delphix Replication uses a private network protocol to communicate between two Delphix Engines. You may specify a network interface to run replication by configuring routing to direct traffic over a particular interface. 

The replication network protocol uses TCP port 8415. If there is a firewall between the source and target that is blocking this port, then there are two possible solutions:

1. Enable port 8415 on the firewall in order to allow connections to this port from the source to the target.

2. Replication can connect through a SOCKS proxy if one exists. Configure the SOCKS proxy address and port by connecting to the command-line interface (CLI) as a system administrator user and navigating to "service proxy" to update the SOCKS configuration.
   

Replication can connect through a SOCKS proxy if one exists. Configure the SOCKS proxy address and port by connecting to the command-line interface (CLI) as a system administrator user and navigating to "service proxy" to update the SOCKS configuration. Example:

Example of a SOCKS Proxy

dlpx-engine> service proxy
dlpx-engine service proxy> update
dlpx-engine service proxy update *> set socks.enabled=true
dlpx-engine service proxy update *> set socks.host=10.2.3.4
dlpx-engine service proxy update *> set socks.username=someuser
dlpx-engine service proxy update *> set socks.password=somepassword
dlpx-engine service proxy update *> commit
dlpx-engine service proxy> get
    type: ProxyService
    https:
        type: ProxyConfiguration
        enabled: false
        host: (unset)
        password: (unset)
        port: 8080
        username: (unset)
    socks:
        type: ProxyConfiguration
        enabled: true
        host: 10.2.3.4
        password: ********
        port: 1080
        username: someuser

Configure the replication source Delphix engine

1. On the source Delphix Engine, click System, then Replication.

2. In the left-hand navigation section, click Create Profile.

3. Enter the following required fields:

   1. Name of the replication profile

   2. The hostname or IP address for the target Delphix Engine.

Replication profile options

There are several configuration options for your replication profiles. These give you more granular control on options such as when replication will run, how much bandwidth it may use, and which objects are replicated. Details for each option are described below.

• Automatic replication: A policy to automatically run replication. With this option, you can set up replication based on the schedule you need. Automatic replication allows you to define a policy to automatically run replication. By default, automatic replication is disabled, meaning that you must trigger replication updates manually. To enable automatic replication, click the Enabled checkbox. With this setting, you can enter the frequency and time for replication updates to the target Delphix Engine. Automatic replication uses Quartz, a job scheduling tool (http://www.quartz-scheduler.org/), for scheduling, which can be configured via the Advanced option.

• Traffic Options: Various traffic and bandwidth options are available. For example, you may want to enable encrypted traffic or limited bandwidth during replication updates. 

  • Encrypting traffic: By default, replication streams are unencrypted, which provides maximum performance on a secure network. However, this setting allows you to encrypt traffic during replication.

    Encrypting Replication Encrypting the replication stream will consume additional CPU resources and may limit the maximum bandwidth that can be achieved. During replication, the Delphix Engine will negotiate an SSL connection with its server peer to use TLS_AES_128_GCM_SHA256 as the cipher suite and TLSv1.3 as the protocol.
    

  • Network connections: Allows setting the number of underlying network connections that can be used by replication.

  • Limiting bandwidth: By default, replication runs at the maximum speed permitted by the underlying infrastructure. In some cases, particularly when you use a shared network, replication can increase resource contention and may impact the performance of other operations. This option allows you to specify the maximum bandwidth that replication can consume.

    When you replicate multiple datasets, your actual bandwidth usage may exceed the configured limit, potentially by up to 4x, due to system-level parallelization. To mitigate this, consider setting your bandwidth limit to a lower value than your network’s maximum threshold, especially when bandwidth contention is a concern.

• Objects Being replicated: Select the objects you wish to be replicated from the source engine to the target engine. In the right-hand column, under Objects Being Replicated, you can select the objects you want to replicate. Some selected objects may have dependencies – other objects that will be pulled into replication because they share data.
  

1. When replicating a group, all dSources and VDBs currently in the group, or added to the group at a later time, will be included.

2. Regardless of whether you select a VDB individually or as part of a group, the parent dSource or VDB (and any parents in its lineage) are automatically included.

   • This is required because VDBs share data with their parent object.

   • In addition, any environments containing database instances used as part of a replicated dSource or VDB are included as well.

3. When replicating individual VDBs, only those database instances and repositories required to represent the replicated VDBs are included. Other database instances that may be part of the environment, such as those for other VDBs, are not included.

4. Non-data objects (Delphix users, roles, permissions, authorizations, policies, database configuration templates) that are associated with selected objects will be automatically included during replication. They will not be shown on the selection interface.

Removing objects from a replication profile

You can remove individual objects (for example, a VDB) from an existing replication profile without impacting other objects that remain in the profile. Removing an object does not affect the replication state or retention of any other dataset included in the profile.

When an object is removed:

1. The change is applied to the replication specification on the source.

2. The object is removed from the replication target engine only after the next replication job completes.

3. Remaining objects continue to replicate normally and are not re-transferred or modified.

4. If you later add the object back, the full dataset must be replicated again because the object is treated as new to the profile.

If a VDB on the target depends on data from an object removed from the profile, that data can remain on the target as held space after the next replication job removes the object. If you later add the object back, the entire object is replicated again; the new copy cannot be stitched or merged with data retained from the previous copy. For an explanation of this held-space case, see Source-side changes to replicated data.

You cannot remove the final object from a replication profile. If the last object must be removed, delete the replication profile instead.

Configure the target Delphix engine

Additional configuration on the target engine is not needed. Replicated objects will appear in an alternate received replica (or namespace) that mirrors the original object layout. 

To view replicated objects from the Delphix Engine:

1. Click System, then select Replication.

2. Look under Received Replicas. All replicated objects are read-only until the replica is failed over. For more information about managing replicas and how to activate a replica, see the topics Replicas and Failover and Controlled Failover.

You can create and manage objects on the target server without affecting subsequent updates, though this can cause conflicts on failover that require additional time to resolve. For disaster recovery use cases, it is recommended to keep the target passive and not create any local objects. This will avoid conflicts and guarantee a smooth failover operation.

Multiple sources can replicate to the same target, allowing for the flexible geographical distribution of data. This is not a recommended practice for disaster recovery, because it increases the probability of conflicts on failover and may oversubscribe resources on the target if multiple replicas are failed over and there is insufficient infrastructure to support the combined workloads. 

Delete received replicas

The following steps apply to unlocked network replication replicas on engines with access to the Continuous Data management UI. For offline replication, first remove the associated receive spec as described in Offline replication (air gap). Locked Continuous Vault replicas cannot be deleted; see Continuous Vault.

1. On the replication target engine, click System, then Replication.

2. Under Received Replicas, click the replica you want to delete.

3. Click the delete icon (trash can).

Re-establish replication after replica deletion

If you delete a received replica and then run its existing source profile, the replication update fails. The error can also occur after a target factory reset:

An incremental replication update could not be sent because the target is out of sync with the source. The target may have been factory reset or the namespace may have been deleted.

The following procedure uses the Continuous Data management UI. If those menus are unavailable on your installation, use DCT's replication-policy interface for policy creation, execution, and deletion; see Managing Replications in DCT for its prerequisites and procedures.

To re-establish network replication after deleting the received replica on an engine with management UI access:

1. Create a new replication profile on the source engine for the required objects and target. See Create new replication profile wizard.

2. Run the new profile by clicking Replicate Now on the source engine, and verify that the replication job completes successfully.

3. Once replication with the new profile succeeds, delete the obsolete source profile when it is no longer needed.

After a test-failover, use failback and preserve the original source profile if you want to resume updates to the existing replica. For that workflow, see Failback.

Delete replication profiles

Delete an unlocked replication profile on the replication source engine when it is no longer needed. Preserve the profile while failback remains an option, because the target needs it to receive updates after failback. Locked Continuous Vault profiles cannot be deleted.

On engines with access to the Continuous Data management UI:

1. On the source engine, click System, then Replication.

2. Under Replication Profiles, click the profile you want to delete.

3. Click the delete icon (trash can).

For the CLI procedure, see Delete replication specs.