Notify downstream consumers of impactsMake consumers aware of potential problems in advance.
Do the following whenever a data store changes.
- Refresh metadata: Crawl the data store to retrieve its updated metadata.
- Detect changes: Compare the new metadata against the previous metadata. Identify any changes that could cause an impact — the addition or removal of columns, for example.
- Find dependencies: Use lineage to find who uses the data store. These could include analyzing transformation processes, other data stores, BI dashboards, and so on.
- Notify consumers: Notify each consumer through their preferred communication channel — Slack, Jira, etc.
This process could also be incorporated as part of the testing phases of changing a data store. For example, the CI/CD process that changes the data store itself could trigger these steps. Orchestration can then notify affected consumers before production systems change.
Consumers of data are aware of potential problems in advance.