Executing a profiler task (mongo profiler)

To generate a masking inventory file, execute profiler.sh and select the appropriate options from the following list:

Configure sampling percentage
Adjust the sample_percentage setting in db_connection_conf.yaml to improve sampling randomness. For example, setting sample_percentage: 100 processes (parses) all documents in the specified collection.

Enable data level profiling
To enable data level profiling, set data_profiling to true in db_connection_conf.yaml.
Data-level profiling is a resource intensive operation and will impact system performance. By default, data_profiling is set to false.
Enable data_profiling: true only when necessary.

Define profiler parameters
You can run the profiler across multiple databases and collections. Parameters such as sample_size, sample_percentage, and batch_size (documents per batch) can be customized. The default batch_size is 1000. For example:
```
databases:
  - name: "sales_db"
    target_database: "target_sales_db"
    collections:
      - name: "customers"
        target_collection: "target_customers"
        sample_size: 1000
        sample_percentage: 30
        batch_size: 500
      - name: "orders"
        target_database: "target_sales_db"
        target_collection: "target_orders"
        sample_size: 1000
        sample_percentage: 30
        batch_size: 2000
  - name: "hr_db"
    target_database: "target_hr_db"
    collections:
      - name: "employees"
        target_collection: "target_employees"
        sample_size: 1000
        sample_percentage: 30
        batch_size: 5000
```
Define custom domains and algorithms

You can define custom domains and algorithms in the entities.yaml file using the format shown below. These custom definitions are applied when generating the masking inventory. Custom definitions are helpful when you need domains or algorithms that are not included in the default list, or when implementing custom pattern matching.

Custom definitions in entities.yaml override the default configurations during masking inventory generation.

entities:                          # Required: List of entities
  - name: <name_of_entity>           # Required: Entity name which refers to domain name in Continous Compliance
    algorithm_name: <name_of_algo>   # Required: Algorithm name which refers to algorithm used for masking in Continous Compliance
    patterns:                        # Optional: Regex patterns for detection
    - name: <internal_name>          # name to identify various patterns for the same entity
        regex: <regex_pattern>       # string wrapped in double quotes (e.g. "(\\bbiometric\\b)")
        score: <score_value>
    supported_entity: <entity_name>  # Optional: For recognizer
    supported_language: <language>   # Optional: For recognizer  
    context:                         # Optional: List of context keywords 
      - <keyword1>
      - <keyword2>
exclusions:                          # Optional: Exclusion keywords
  - <exclusion1>
  - <exclusion2>

This is an example custom definition:

entities:
  - name: ZIP_CODE
    algorithm_name: dlpx-core:CM Alpha-Numeric
    patterns:
    - name: ZIP_CODE 1
      regex: "(\\b\\d{5}(?:\\-\\d{4})?\\b)"
      score: 0.9
    supported_entity: ZIP_CODE
    supported_language: en
    context:
      - zip
      - code
  - name: SALUTATION
    algorithm_name: RandomValueLookup
    patterns:
    - name: SALUTATION 1
      regex: "(?i)^(mr|mrs|ms|dr|miss)\\.?"
      score: 0.9
    supported_entity: SALUTATION
    supported_language: en
    context:
      - title
      - salutation
      - honorific
exclusions:
  - "_id"
  - "password"
  - "token"
  - "credit_card_number"

To include entities.yaml when generating the masking inventory, run either of the following two commands:

./profiler.sh -o masking-inventory -c db_conf_temp.yaml -y entities.yaml

./profiler.sh -o masking-inventory -c db_conection_conf.yaml

You can now execute profiler.sh to generate a masking inventory file:

./profiler.sh -o masking-inventory -c db_connection_conf.yaml

==================== PROFILER START [masking-inventory] 2025-11-14 08:07:38 ====================
Source Database Password:
...
[2025-11-14 08:07:41][INFO] Selected docker runtime.
...
2025-11-14 08:08:00,981 [INFO] (mongodb_connector:37) - Connected to MongoDB successfully!
...
2025-11-14 08:08:03,964 [INFO] (masking_inventory_creator:407) - Masking inventory created and saved to /app/test_db_test_collection_masking_inventory.json
...
[2025-11-14 08:08:06][INFO] Copying masking inventory file[test_db_test_collection_masking_inventory.json] to current directory.
[2025-11-14 08:08:06][INFO] =================================================================================
[2025-11-14 08:08:06][INFO]  NOTE:
[2025-11-14 08:08:06][INFO]  Recommending to review the masking inventory file[test_db_test_collection_masking_inventory.json]  
[2025-11-14 08:08:06][INFO]  in the current directory. Validate it and fix the algorithms, if required.
[2025-11-14 08:08:06][INFO] =================================================================================

Review the masking inventory file and modify it if necessary.
Masking inventory files are generated in the format <db-name>_<collection-name>_masking_inventory.json. One file is created for each collection. Each entry in the masking inventory includes:

The JSONPath profiled for masking
Its associated domain
The algorithm name

{
  "field_name": "$[*]['country']",
  "domain_name": "COUNTRY",
  "algorithm_name": "dlpx-core:Countries SL"
},

Execute profiler.sh to generate a dataset using the specified masking inventory file.

$ ./profiler.sh -o data-sets -c db_connection_conf.yaml -e hsc-host-example.com -k <API Key> -d <connector id> -u <unload split>[2024-04-29 13:50:58][INFO] Getting source database details from [db_connection_conf.yaml].
==================== PROFILER START [data-sets] 2025-11-17 01:52:10 ====================
[2025-11-17 01:52:10][INFO] Payload for API /data-sets is stored in .payload_data_sets.json.20251117015210.
[2025-11-17 01:52:10][INFO] Executing single API /data-sets POST for all databases/collections.
[2025-11-17 01:52:12][INFO] Successfully executed API /data-sets for combined payload.
[2025-11-17 01:52:12][INFO] Newly generated data-sets id:46

Last updated

Oct 9, 2026