
Data Pipelines: Efficient Data Movement and Transformation
What is a data pipeline: Uber's cascade starts at 3:47 AM so every dashboard shows yesterday by 7. ETL versus ELT, and why it flipped.
The exact change
A 2021 study by Monte Carlo Data, conducted across 150 data engineering teams, found that data downtime (periods when data is missing, inaccurate, or otherwise unreliable) affected organizations for an average of 10 hours per week. | The study found that approximately 60 percent of data incidents were not caught by the data team but were instead reported by business users who noticed wrong numbers in reports. This lag
A 2022 survey of 300 data professionals, conducted by Wakefield Research for Monte Carlo, found that they spend about 40 percent of their time checking data quality, a measure of how much data downtime (periods when data is missing, inaccurate, or otherwise unreliable) costs a team. | Data incidents are often caught first by business users who notice wrong numbers in reports, not by the data team. This lag
Suggested change
Corrected 2 figures to the published source.
Why this is better
Each study and survey figure in the article was checked against the published source. Figures that differed from the source were corrected to the source's own numbers, dates and attribution.
Everyone who worked on this page
More by Emir Baycan in Data Analytics Insights
- Correction The Net Promoter Score (NPS): Does It Work?
- Correction What Is Data Analytics: Turning Data Into Decisions
- Correction What Is a KPI: Key Performance Indicators Explained
- Correction Analytics vs Data Science: Key Differences Explained