A data cleansing algorithm is used to standardize varied spellings, misspellings, and abbreviations for the same name. For example, "Ariz", "Az", and "Arizona" can all be cleansed to "AZ". Use this algorithm if the target data needs to be in a standard format prior to masking.
Configuration options
| Options | Description | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case sensitive lookup |
Determines whether the lookup comparison is case sensitive. If enabled, the input value must match the case of the value in the lookup file. For example, if the lookup file contains Arizona=AZ:
|
||||||||||||
| Trim whitespace | Removes leading and trailing whitespace from the input value before lookup is performed. his allows a single value=replacement in the lookup file to cleanse data containing extraneous leading and trailing whitespace. This option is recommended for fixed-width files and fixed-length database types such as CHAR or NCHAR. |
||||||||||||
| Lookup file | File containing the value and replacement pairs used for cleansing. Each pair should appear on a new line in the format {value}{delimiter}{replacement}. | ||||||||||||
| Lookup file delimiter | Specify a Lookup File Delimiter (value and replacement separator) up 50 characters long. Character or string used to separate the value and replacement in the lookup file. The default delimiter is =. You can change this to match the lookup file. |
Example lookup file
The lookup file contains newline-separated value and replacement pairs. It does not require a header. Make sure there are no spaces or returns at the end of the last line in the file. The following is sample file content:
NYC=NY
NY City=NY
New York=NY
Manhattan=NYCreate an algorithm using this framework
To create an algorithm using this framework, follow the steps in Create an algorithm and select Data Cleansing during the Configuration step.