ASDD features and support

Automated Sensitive Data Discovery (ASDD) provides advanced detection capabilities for identifying sensitive data during discovery operations.

ASDD uses a classifier-based detection model to evaluate database metadata and data values. This approach enables more accurate identification of sensitive data compared to traditional pattern-based discovery methods.

ASDD supports discovery across multiple data formats and data sources.

Supported data sources

ASDD discovery only supports Databases for the current release.

ASDD features

• The ASDD discovery uses classifiers rather than search and type expressions. Classifiers support more features and configuration options than expressions. 

  • The TYPE classifier framework uses standard Java SQLType values to identify data types, which should provide broad support for type detection across all database variants.

  • The PATH classifier supports exact matching and can be configured to consider table name in addition to column name when matching.

  • The REGEX classifier supports the following checksums for data-level recognition:
    
    

    • LUHN: Luhn checksum for credit card numbers
    • MOD10_ABA: Modulus 10 checksum for ABA Routing numbers (USA).
    • MOD11_NHS: Modulus 11 checksum for 10-digit NHS (National Health Services) numbers.
    • MOD11_TFN: Modulus 11 checksum for 9-digit TFN (Australian Tax File Numbers).
    • MOD97: Modulus 97 checksum for IBAN (International Bank Account Numbers).
    • MOD9_JPN_CORP: Modulus 9 checksum with weights for 13 digit Japanese corporate numbers.
    • MOD11_IBM_JPN_DRIVERS_LICENSE: Modulus 11 checksum for 12 digit Japanese drivers licenses.
    • MOD9_IBM_JPN_MY_NUMBER: Modulus 11 checksum for 12 digit Japanese 'My Number'.
      

• The ASDD standard discovery policy contains data level logic by default, allowing some columns containing sensitive information to be identified even if the column names are not meaningful.

  • New or improved REGEX classifiers for Zip Code and Email Address data classes.

  • New LIST classifiers are present for First and Last Name, Full Name, US City, US State, and Country data classes.

• Assignment threshold can now be set on a per discovery policy basis.

ASDD sampling

The ASDD discovery provides better matching when the number of rows in a table is less than the target number of rows for discovery, and in general, provides more nuanced confidence value in discovery results.

The ASDD discovery supports statistical sampling for Oracle, SQL Server, and SAP ASE databases, so the data sampled will better reflect the full range of values for each column across the entire table.

The ASDD discovery attempts to retrieve more data values when a large fraction of the data values for a column are null or empty. The threshold to trigger an additional query is controlled by the application setting ASDD/DefaultNullFilterThreshold – more information can be found in the Application settings section. This additional query performs a full table scan for non-null values and may run for an extended period depending on table size. To prevent this, a timeout (via DefaultTableSampleLimitSeconds) is applied to the query, triggering an execution event if exceeded. If the discovery fails to retrieve sufficient data, the user can adjust the application settings by decreasing DefaultNullFilterThreshold (risking lower-quality sampling) and/or increasing DefaultTableSampleLimitSeconds (potentially extending job runtime) to suppress warnings. Alternatively, the user can ignore the warning and manually assign algorithms to the affected columns.

Limitations

Some limitations may apply depending on the data source or connector used.

Current limitations include:

• Discovery of XML or JSON documents stored within database columns may not be supported.

• Discovery results may vary for some extended connectors depending on database compatibility.

If discovery is not supported for a particular connector or data source, the system will report an error when attempting to run discovery.