Dynamic data pipeline optimization

🔄 Tasks

1. Collect metrics:
   1. Collect runtime metrics from processing engines, including when, how often, and for how long data processing occurs.
   2. Collect metrics from each data store (per data asset), including how often the data changes and is accessed.
2. Make recommendations:
   1. Avoid processing data that changes or is accessed less often than it is processed. For example, avoid daily reprofiling of a data source whose data changes by less than 1-2% per week.
   2. Stagger the processing schedule for better resource utilization.
3. Auto-schedule: Apply the recommended scheduling changes to the data processing environment.

🎉 Outcome

Reduce unnecessary data processing and improve resource utilization.

Related

COST OPTIMIZATION

Purge stale or unused assets

Maintain a clean data landscape.

COST OPTIMIZATION

Allocate compute resources dynamically

Improve resource utilization and reduce processing delays.

COST OPTIMIZATION

Clean up data landscape by removing duplicate assets

Prevent duplicate assets to reduce costs.