Free Text Redaction framework

The Free Text Redaction Algorithm Framework is designed to effectively remove sensitive information from unstructured text fields, such as "Notes" columns in databases. Utilizing this framework involves configuring the algorithm to recognize and mask sensitive data embedded within text, which requires some expertise.

To identify the sensitive information, the algorithm relies on a list of predetermined lookup words associated with the type of data to be masked. For instance, to redact addresses, users might configure the algorithm to search for key indicators like “St,” “Cir,” or “Blvd.” Additionally, the framework supports pattern matching to flag potentially sensitive data patterns. A common example is the pattern "123-45-6789" used to detect Social Security Numbers.

Both lookup words and regular expressions employed by the algorithm focus on matching entire words within the text. Consequently, even if a regular expression is set to match a part of a word, the algorithm is programmed to redact the entire word, ensuring that partial data elements are not inadvertently exposed. This method guarantees that the sensitive data is fully obscured, enhancing the security of the masked text.

Example

An example is provided below to help clarify how the information above works.

In this example, the redacted (masked) value is field_2 in a delimited file.

field_1,field_2,field_3
1,000123000,last
2,000123,000123,last
3,123000,last
4,000 123 000,last
5,000 123,last
6,123 000,last
7,123,last

Making it easier to read - I have only shown field 2 below.

Example1:

• RegEx: '123'

• Redacted with: 'xxx'

orig         redact         Comment
0000123000   0000123000      < Not redacted - 123 is part of the string.
0000123      0000123         < Not redacted - 123 is part of the string.123000       123000          < Not redacted - 123 is part of the string.
000 123      000 xxx         < Redacts the 'word' 123 to 'xxx'
000 123 000  000 xxx 000     < Redacts the 'word' 123 to 'xxx'
123          xxx             < Redacts the 'word' 123 to 'xxx'
000 123.     000 xxx.        < Redacts the 'word' 123 to 'xxx'

Example2:

• RegEx: '123.*'

• Redacted with: 'xxx'

orig         redact         Comment
0000123000   0000123000      < Not redacted - 123 is part of the string.
0000123      0000123         < Not redacted - 123 is part of the string.
123000       123000          < Redacts the 'word' 123000 to 'xxx'
000 123      000 xxx         < Redacts the 'word' 123 to 'xxx'
000 123 000  000 xxx 000     < Redacts the 'word' 123 to 'xxx'
123          xxx             < Redacts the 'word' 123 to 'xxx'
000 123.     000 xxx.        < Redacts the 'word' 123 to 'xxx'

Example3:

• RegEx: '.*123'

• Redacted with: 'xxx'

orig         redact         Comment
0000123000   0000123000      < Not redacted - 123 is part of the string.
0000123      0000123         < Redacts the 'word' 000123 to 'xxx'
123000       123000          < Not redacted - 123 is part of the string.
000 123      000 xxx         < Redacts the 'word' 123 to 'xxx'
000 123 000  000 xxx 000     < Redacts the 'word' 123 to 'xxx'
123          xxx             < Redacts the 'word' 123 to 'xxx'
000 123.     000 xxx.        < Redacts the 'word' 123 to 'xxx'

As can be seen - if we match a value ('word') the complete 'word' is redacted.

The Free Text Redaction can't redact a segment of characters in a word.

You can use a Free Text Redaction Algorithm Framework to show or hide information by displaying either a "DenyList" or an "AllowList".

Configuration options

Option	Description
Redact type	Determines how matching values are handled. Supported options include Deny list and Allow list. DenyList – Designated material will be redacted (removed). For example, you can set a deny list to hide patient names and addresses. The deny list feature will match the data in the lookup file to the input. AllowList – ONLY designated material will be visible. For example, if a drug company wants to assess how often a particular drug is being prescribed, you can use an allow list so that only the name of the drug will appear in the notes.
Lookup file	File containing words that should be matched during redaction. Each entry must appear on a new line.
Lookup file redact value	Value used to replace words that match entries in the lookup file.
Regular expressions	List of patterns used to detect sensitive information within text.
Regular expression redact value	Value used to replace matches detected by the regular expressions.

Limitations

• The maximum number of supported Regular Expressions is 50. Exceeding this number will lead to the Component Configuration exception.

• The maximum number of supported words in the Lookup File is 1000. Exceeding this number may affect the algorithm performance.

• The Lookup File format must be txt.

• Every entry in the Lookup File must be a new line separated. Phrases are not supported. Case sensitive.

• The maximum allowable size for an input text, which can include multibyte characters, to be masked is 32768 bytes. Exceeding this number will lead to the Non-Conformant data exception.

Examples

Input:

The customer Bob Jones is satisfied with the terms of the sales
agreement. Please call to confirm at 718-223-7896.

Algorithm configuration:

1. The Redact Type is DenyList.

2. Lookup File entries:

   1. Bob

   2. Jones

   3. agreement

3. The Lookup File Redaction Value is XXXX

4. Regular Expressions entry:

   [0-9]{3}-[0-9]{3}-[0-9]{4}

   1. The Regular Expression Redaction Value is YYYY

Masking result:

The customer XXXX XXXX is satisfied with the terms
of the sales XXXX. Please call to confirm at YYYY.

"Bob", "Jones", "agreement" and the phone number are redacted.

Create an algorithm using this framework

To create an algorithm using this framework, follow the steps in Create an algorithm and select Free Text Redaction during the Configuration step.