Creating a dSource with ingestion type AWS S3

The following procedure provides details on creating a dSource with AWS S3 ingestion type.

Prerequisites

• The source and staging instances must meet the host requirements as described in Requirements for CockroachDB Staging/Target hosts.

• The accessibility of the backup location within the AWS S3 bucket from the Staging host must be verified prior to ingestion.

Creating dSource

1. Login to Delphix Management application.

2. Click Manage > Datasets.

3. Click on Add dSource.
   

4. In the Add dSource wizard, select the CockroachDB source config you just created on the staging host.
   

5. Select the dSource type from the drop-down available on dSource wizard. Refer different dSource Types.

6. Enter the CockroachDB specific parameters for your dSource configuration.

   1. dSource Type Type of dSource.

   2. dSource (AWS_S3) Details

      1. Mount Path The path on the host where Delphix will mount the dSource files via NFS.

      2. IP Address / Hostname of the Staging Cluster IP Address / Hostname of the interface for the staging Cluster (default 127.0.0.1).

      3. Staging CockroachDB Port Number Staging DB port for cluster’s inter-node communication.

      4. Staging CockroachDB Console Port Number Staging Console port for HTTP requests.

      5. CockroachDB Backup’s Subdirectory Path CockroachDB Backup’s Subdirectory Absolute Path where backups are located in S3 Bucket.

      6. CockroachDB Backup’s Bucket or Container Name Storage bucket or container name including absolute path where CockroachDB backups are located.

      7. Authentication Option for AWS S3 Storage Buckets

         1. Specified Authentication Need to specify the AWS access key ID and secret access key in the dSource configuration fields:

            1. Account Name or Access Key Specify an account access name or key associated with object storage holding CockroachDB backups.

            2. Account Secret Access Key Specify the account secret access key associated with the account name field. This is essentially the password that gives access to CockroachDB backups.

         2. Implicit Authentication Store the AWS credentials as environment variables.

            1. When using implicit authentication, ensure that aws credentials are configured on staging host using aws configure command.

            2. Configure an IAM role attached to the EC2 instance with the necessary permissions (e.g., s3:GetObject, s3:ListBucket, s3:PutObject) to access the S3 bucket where the backups are stored.

            3. Implement an S3 bucket policy to grant access to specific AWS accounts or IAM entities that need to restore the backup stored inside the S3 bucket. For example:

               {
                   "Version": "2012-10-17",
                   "Id": "AWS_S3_Policy",
                   "Statement": [
                       {
                           "Sid": "ExampleStatement01",
                           "Effect": "Allow",
                           "Principal": {
                               "AWS": "arn:aws:iam::22420070000:user/delphix"            },
                           "Action": [
                               "s3:PutObject",
                               "s3:GetObject",
                               "s3:ListBucket"            ],
                           "Resource": [
                               "arn:aws:s3:::dlpx-crdb-bucket",
                               "arn:aws:s3:::dlpx-crdb-bucket/*"            ]
                       }
                   ]
               }

      8. Skip Checking Localities Select the checkbox to skip checking the localities of a cluster before a restore.

      9. Secure Mode Option to run the CockroachDB server in secure mode

      10. Certificate directory path The absolute path to the certificate directory, it must have valid certificates containing the IP address or hostname for the CockroachDB server.

          • You must specify the input “Certificate directory path” when “Secure Mode” option is selected.

          • For more detailed guidance on CockroachDB certificate generation, refer to the official documentation: CockroachDB Certificate Generation.

      11. CockroachDB Cluster Node Count Number of nodes that consists of a Staging CockroachDB cluster. Minimum: 3 and Maximum: 9.

      12. CockroachDB Parameters These are node-level configuration parameters passed as command-line arguments when starting the CockroachDB process.

          Example:

          --max-sql-memory=0.25, --cache=0.25

          1. Parameter Name CockroachDB Cluster configuration parameter name.

          2. Parameter Value CockroachDB Cluster configuration parameter value.

      13. CockroachDB Cluster Parameters These are cluster-wide settings applied after the cluster is initialized using the SET CLUSTER SETTING command.

          Example:

          SET CLUSTER SETTING kv.bulk_io_write.restore_node_concurrency = 8

          1. Cluster Parameter Name:CockroachDB cluster-level configuration parameter name.

          2. Cluster Parameter Value: CockroachDB cluster-level configuration parameter value.

      14. Additional Nodes: Specify the node configuration for each node in the CRDB cluster.  

          1. Environment : Select environment on which node has to be created. 

          2. Environment User: Select environment user corresponding to the environment. 

          3. IP Address / Hostname: IP Address / Hostname of the interface for the staging node(default 0.0.0.0) 

          4. CockroachDB Port Number: Staging DB port for node’s inter-node communication 

          5. CockroachDB Console Port Number: Staging Console port for HTTP requests 

          6. Certificate directory path: The absolute path to the certificate directory, it must have valid certificates containing the IP address or hostname for the CockroachDB server.

          7. Append cluster CRDB parameters Check with box if the cluster configuration parameters are applicable for the node. 

          8. Additional Node Configuration Parameters: Additional CockroachDB configuration parameters to start the node.

      15. Enable PITR: Select the check box to enable ingest backups up to a provided timestamp.
      16. End time for ingestion (PITR): Timestamp upto which the backups would be restored.
      17. Enable only full backup ingestion: Select the check box to ingest only full backup from a series of incremental data available in the backup location.

7. Then click Next to proceed further.

8. Provide the details for dSource Name and Target group on the dSource configuration page and click Next.

9. On the Data management page, select the following and click Next.

10. Staging Environment: This will be your staging host where source config was created.

    • User: Database OS user with required privileges for linking the dataset.

    • On the next screens, configure a policy, hooks, and review the configuration, then click on Next button to view the summary.

11. Click the Submit button which will initiate the linking process

12. Once dSource is created successfully, you can review the datasets on Manage > Datasets > dSource Name.