UX: Data Lineage (2020-22)

Design Brief

Design the data lineage for the Cloud Data Governance and Catalog application that is simple, flexible and efficient. A simplified version of lineage for business users and a seamless transition to detailed lineage for technical users.

Design Challenges

Data lineages can get gnarly and complex with data flows across thousands of nodes. Design has to work for scale while at the same time showing the pulse of the flow in a condensed form that helps with comprehension without being an information overload.

Scale: Horizontal and vertical scalability ( 1 table with 20,000 columns (vertical) and number of connected systems(horizontal)
Display relevant level of details to business and technical users Ex.Technical user would like to view all data transformations across the lineage. Ex. A standardization transformation -Standardize country codes (US,USA to US) was to be applied across, but in some tables, it still shows up as USA. Need to identify why USA is still showing up – trace where the standardization transformation was not applied. Business user would like a simplified view that shows the business flow and hides all the intermediary technical transformations.
Work within the technical constraints of implementation/canvas framework.

Design Approach

Strive for simplicity, flexibility and efficiency
- Simplicity: start with just enough information, allow to show more
- Flexibility: granular control over the level, amount and type of information to display
- Efficiency: single click vs right click, tooltips vs pop up vs overlays
Start with use cases and then focus on canvas management

How to tackle scale

Level of details (summary to details)
Isolating and highlighting paths
Expanding/Collapsing nodes
Filtering content
Pagination
Line treatments
- Minimize lines
- Group lines when possible
- Do not show all second level object lines by default
- Maximize node space
Customizing and saving views

Intelligent Defaults

Show source and target and the seed and collapse the rest with knobs to expand as needed.

Process

What is data lineage?

Key use cases

1. Root Cause Analysis

User

Data Steward, Data Analyst, Line of Business User

Scenario

The profit metric doesn’t look right in the Sales Report. Why?

Is it picking up data from the right data source? Are the calculations that compute the metric data correct? Are there any data quality issues (completeness of the data)?

After diagnosing the issue, send notifications to the stakeholders to fix the issues (rule, DQ,etc).

2. Impact Analysis (What if Analysis)

Users

Data Engineer, Data Analyst, Data Architect

Scenario

New Standards: As per the new standard, the gender attribute needs to be updated to include the third gender. User creates a new gender attribute(column) in the Database.

In the application, user searches for existing gender attributes. Looks at the impacted attributes, associated process, policies, glossaries, etc and their stakeholders.

Notifies/logs a change request with instructions to point these attributes to the new gender attribute that was created.