Installation and setup (Mongo profiler)

Pre-requisites

1. Download the latest version of the MongoDB profiler from the Delphix download page.

2. Ensure that the host running the profiler has podman installed.

3. Set up the Hyperscale MongoDB Connector and add the required ConnectorInfo details.

Procedure

1. Untar the profiler downloaded from Delphix’s download page. It should contain the container images for the profiler and the podman-compose.yaml file to run the profiler.
   

   tar xzvf mongo-profiler-<platform>.tar.gz

2. Create db_connection_conf.yaml using the following template and populate the required information.

   cp db_connection_conf.yaml.template db_connection_conf.yaml
   
   cat db_connection_conf.yaml
   #
   # Copyright (c) 2023, 2025 by Delphix. All rights reserved.
   #
   
   source_mongodb_uri: "__MONGODB_URI__"
   source_username: "__SOURCE_DATABASE_USER__"
   
   # List of databases and their collections to process
   # Each collection can have its own sample_size, sample_percentage, and batch_size
   databases:
     - name: "__SOURCE_DATABASE_NAME_1__"
       target_database: "__TARGET_DATABASE_NAME__"
       collections:
         - name: "__SOURCE_COLLECTION_NAME_1__"
           target_collection: "__TARGET_COLLECTION_NAME__"
           sample_size: 1000
           sample_percentage: 10
           batch_size: 500
   
           #
           # Optional size parameters
           #
           # sample_size: Integer. The number of documents to be sampled in a given collection.
           # sample_percentage: Integer. The percentage of total number of documents to be sampled in a given collection.
           # batch_size: Integer. Number of documents processed in each batch (optional, for performance tuning).
           #
           # Note:
           # Specify either sample_size or sample_percentage—not both. If both are provided, sample_size will be used. Do not include the % symbol when using sample_percentage. for example, Use 10 instead of 10%.
   
         - name: "__SOURCE_COLLECTION_NAME_2__"
           target_collection: "__TARGET_COLLECTION_NAME__"
           sample_size: 2000
           sample_percentage: 30
           batch_size: 1000
     - name: "__SOURCE_DATABASE_NAME_2__"
       target_database: "__TARGET_DATABASE_NAME__"
       collections:
         - name: "__SOURCE_COLLECTION_NAME_A__"
           target_collection: "__TARGET_COLLECTION_NAME__"
           sample_percentage: 5
           batch_size: 200
         - name: "__SOURCE_COLLECTION_NAME_B__"
           target_collection: "__TARGET_COLLECTION_NAME__"
           sample_size: 500
           sample_percentage: 2
           batch_size: 100
   
   crawling_mode:  "random"
   data_profiling: false       # data_profiling is a costly operation, enable only if it is required.
   
   #
   # Other parameters
   #
   # crawling_mode: String in double quotes. Strategy for data traversal during sampling (Options:- "random" or "linear"). When set, this parameter would be applicable for all the databases and collections.
   # data_profiling: Boolean. Enable (true) or disable (false) data-level profiling. When set, this parameter would be applicable for all the databases and collections.

    

   For example:

   # Example:
   
   source_mongodb_uri: "__MONGODB_URI__"
   source_username: "testUser"
   
   databases:
     - name: "sales_db"
       target_database: "target_sales_db"
       collections:
         - name: "customers"
           target_collection: "target_customers"
           sample_size: 1000
           batch_size: 500
         - name: "orders"
           target_database: "target_sales_db"
           target_collection: "target_orders"
           sample_percentage: 10
           batch_size: 1000
     - name: "hr_db"
       target_database: "target_hr_db"
       collections:
         - name: "employees"
           target_collection: "target_employees"
           sample_size: 50
   
   crawling_mode: "random"
   data_profiling: false

    

3. Check the help of profiler.sh.

   ./profiler.sh [OPTIONS]
   
   Create masking inventory file.
   
   Options:
     -o OPERATION          Operations
                             masking-inventory   Generate masking inventory JSON file.
                             data-sets           Execute API /data-sets using the
                                                   generated masking inventory file.
   
     -c DB_CONN_CONF       Path of db_connection_conf.yaml
     -y ENTITIES_FILE      Optional path to entities.yaml to extend domain mappings
   
     [-u UNLOAD_SPLIT]     Refer Hyperscale docs for more details.
                           Required for "data-sets" operation only.
   
     [-e HSENGINE]         Hostname or IP of HyperScale Engine
     [-k KEY]              API Key for Controller
     [-d CONNECTOR-ID]     Connector ID
     [-help]               Show this message and exit.
     [-v]                  Enable verbose/debug logging
   
   
     Examples,
   
     #./profiler.sh -o masking-inventory -c DB_CONN_CONF.yaml
   
     #./profiler.sh -o masking-inventory -c DB_CONN_CONF.yaml -y entities.yaml
   
     #./profiler.sh -o data-sets -c DB_CONN_CONF.yaml -e HSENGINE.com -k APIKEY -d ID -u UNLOAD_SPLIT