Eight Ways To Discover Personal Data For GDPR Compliance

Unfortunately, one of the major difficulties encountered by organisations engaged in the consent process due to GDPR is to complete the seemingly simple task of locating and categorising Personal Data held across the enterprise. Often this is referred to as an Information Audit or Data Readiness exercise. For those who have been using a large ERP or CRM application packages to store and process data, meeting this challenge is even more acute.

Silwood Technology recently conducted research into five of the largest and most widely used information application packages. This revealed that the job facing organizations who are still trying to locate Personal Data in the coming weeks is significant. In SAP there are more than 900,000 fields, 140,000 in JD Edwards and 100,000 in Microsoft Dynamics AX 2012 that may (or may not) contain personal information that requires detection and risk assessment. In in short, businesses that are not well-advanced in data discovery or are undertaking manual discovery processes will not be ready on time for GDPR.

Many organisations are addressing the Personal Data challenge through software such as Information or Data Catalogue solutions within their overall Governance or Compliance programme, which often incorporate some form of scanner or crawler that connects to many sources, identifies the metadata and imports it automatically. Others may be using spreadsheet or more home-grown solutions to try to record Personal Data locations and understand how data flows through their organisation.

These solutions can be very effective for some IT systems. However, they will not be as successful for organisations running enterprise CRM or ERP applications from SAP, Oracle, Salesforce, Microsoft or other large application packages unless they incorporate specialist discovery software designed for the task. This is due to the size, complexity and level of customisation of the underlying data landscape of these systems as evidenced by Silwood Technology’s research.

Here we explore the eight main strategies available for organisations identifying Personal Data when starting the GDPR compliance process. Unfortunately, many are extremely time consuming and rely on extensive manual interrogation of databases and systems, a luxury that is not available to any enterprise that is not well advanced in the process.

1. Looking for documentation

Looking for documentation may seem a natural first port of call when trying to understand find Personal Data items in an application. However, even if the data models do exist in this static way, they will be of only limited use in anything but smaller, perhaps home-grown applications with simple data structures.

For those with large scale ERP and CRM packages, the task of navigating documentation to find individual tables and attributes from amongst thousands will be a significant challenge and of course any useful information cannot be shared easily with other tools as re-keying will be required.

2. Manual investigation

This typically involves someone tasked with scouring the relational database (RDBMS) system catalogue for any information which might provide clues as to what data the tables contain, what attributes and fields they include and crucially the relationships between tables.

This is a perfectly acceptable approach for small database systems, where a package is limited in scope or has been developed in-house, but is very labour-intensive in larger systems with a great many tables which do not have useful business names or descriptions.

3. Turning to application or technical specialists

Specialists are likely to have the most familiarity with the application and its underlying data model. They are also most likely to have access to any technical tools which are provided by software vendors which can be used in an attempt to locate the information required. However, their knowledge of the business context of a request for Personal Data may not necessarily be complete, and such specialists are often in very short supply and busy.

4. Hiring external consultants

Another common approach is to engage external consultants. Obviously, they may provide an expert resource, however there can be a significant cost as well as time to familiarise themselves with the data landscape and its customisations. In addition, this can contribute to lower in-house knowledge levels in the long term.

5.  Metadata driven software approach

Using software to identify the metadata associated with Personal Data across an organisation’s IT ecosystem can make the discovery process considerably faster and more effective. Many data catalogue and governance products have facilities to connect to source systems and import their metadata directly so that it can be investigated more fully.  Automating this process reduces the opportunity for error as there is only very limited manual intervention.

This approach does not work for large CRM and ERP systems because of the size, complexity and level of customisation of their data landscapes. There are a few advanced self-service metadata discovery tools, such as Silwood Technology’s Safyr, which provide a view into their metadata and allow users to navigate and search for Personal Data attributes and subset them into appropriate categories. That information can then be shared with Data Catalogue or Governance products or even used with Excel.

Metadata-based solutions can accelerate Personal Data discovery considerably, especially when compared to entirely manual or semi-automated processes.

6. Internet search

Using the internet to locate Personal Data attributes is only really of any value when the data models are in a format that can be published either by vendors or by customers. It would not make much sense to publicly exhibit data models of one’s own in house developed system.

However, it is possible to find metadata definitions for example for well-known social media platforms and occasionally data models from popular ERP and CRM packages which might point you in the right direction of the Personal Data you are seeking. This is often seen as a viable, low cost option, but is labour intensive and also questionable in accuracy terms.

There are also risks. The published information is unlikely to represent the system as implemented by the seeker either through version differences or individual customisations. In addition, it is often necessary to ask a technical specialist to interpret the model and augment it with relevant information from the application itself.

7. Best guess and hypothesis testing

When faced with the problem of Personal Data discovery, many companies use guesswork or hypothesis testing methods to try to find tables and attributes they need. They rely on data observation, insight and on trying to find an appropriate start point from which to launch a search – a strategy that can be frustrating, time consuming and potentially inaccurate.

8. Turning to software vendors

Data Modelling tools offer a good solution for finding Personal Data based on their ability to reverse engineer RDBMS and create a data model from the tables, fields and relationships they find. From there an analyst can try to find the items needed for GDPR.

Data Profiling software can also be useful as it provides the ability to look at data formats to determine if they are likely to contain Personal Data. Sometimes this uses a form of machine learning or other analysis techniques to surface what may be relevant.

ERP and CRM package vendors do have tools which can be used by technical specialists for more traditional database and metadata tasks.

However, the particular challenge of trying to locate Personal Data in large packaged CRM and ERP applications is not adequately met using these approaches.  This is because of the lack of meaningful metadata in their database schema, the size and complexity of the model and the numbers of attributes to be investigated.

GDPR is not a one-time event

For large organisations really struggling to find Personal Data for GDPR, it is too late to employ consultants or redirect staff away from their normal tasks to get through the work. Instead, it is time to look to software tools to automate as much of the Personal Data discovery work as possible.

Of course, Personal Data discovery is just one step towards GDPR compliance, however it is a vital one. It is also worth remembering that GDPR compliance is not a one-time event, maintaining compliance will be an essential business process in the future. A manual, intensive approach today may simply delay the inevitable data reckoning – can you afford to ignore the future?