Compliance jobs process data using a parallel execution model designed to improve performance when masking large datasets.
During job execution, the system reads data from the source system, applies the configured masking algorithms, and writes the transformed data to the target location.
To improve efficiency, the job engine processes data using multiple execution components.
Streams
Streams allow the engine to process multiple tables simultaneously.
When a job starts, the workload is divided into streams that can run in parallel. Each stream processes a portion of the rule set workload.
As streams complete their assigned tasks, the number of active streams may decrease until all tables have been processed.
Increasing the number of streams may improve performance when processing large datasets.
Threads
Each stream internally uses separate threads for different processing stages:
-
reading data from the source
-
applying masking algorithms
-
writing masked data to the target system
This separation allows the engine to efficiently process large volumes of data.
Memory allocation
Each job requires memory resources to process data streams and maintain intermediate results.
Administrators can configure minimum and maximum memory limits for a job to control resource usage.
If sufficient resources are not available, jobs may be queued until system resources become available.
Job queue
If the system is already running the maximum number of concurrent jobs or does not have enough available memory, the job is placed in a queue.
Queued jobs automatically start when resources become available.