Is It Time to Say Goodbye to Data Engineers? Is It Time to Say Goodbye to Data Engineers?

Metadata

• Author: SeattleDataGuy
• Full Title: Is It Time to Say Goodbye to Data Engineers? Is It Time to Say Goodbye to Data Engineers?
• URL: https://seattledataguy.substack.com/p/is-it-time-to-say-goodbye-to-data

Highlights

• Today, we are going to dive into an issue I’ve noticed that seems to oscillate in the data world every few years. That is the removal of data engineers. After all, we tend to get in the way and slow things down, right? (View Highlight)
• Ever since tools like SSIS came onto the scene, vendors and business leaders have been on a mission to remove what they see as the biggest roadblock to data-driven decision-making: data engineers. Or their counterparts—DBAs, ETL Developers, and Data Architects. (View Highlight)
• To some extent, I get why the idea is appealing. When a leader requests a report, a software engineer wants to modify an application table, or a data scientist wants to explore a new dataset, who’s the one slowing down the project? The data engineers. With their rules. Their governance. Their insistence on building robust, scalable data pipelines instead of quick fixes. They really slow down the workflow! So, sure, from that perspective, I understand the desire to move faster. But before you go all-in on the “let’s remove the data engineers” strategy—or if you’re currently fighting a business leader who’s hellbent on it—let’s take a look at what actually happens when companies try to cut out data engineering. Spoiler alert: it rarely ends well. (View Highlight)
• I’ve walked into multiple projects where data engineers were cut from the equation, and I’ve heard of plenty of horror stories about similar situations. At first, it seems like a win. The short-term goal is moving fast. (View Highlight)
• Before long, your data warehouse (Snowflake or whatever you’re using) is cluttered with an endless mess of tables, views, dashboards, and other artifacts that all tell slightly different stories. There’s no standardization, no governance, and no one treating data like a product. Eventually, someone—probably an overwhelmed analyst or a frustrated executive—suggests bringing in more structure. More governance. More control. And just like that, the pendulum swings in the other direction and the cycle repeats. (View Highlight)
• Airbnb went through this where as they put it:

  For several years, Airbnb did not have an official Data Engineer role. Most data engineering work was done by data scientists and software engineers who were recruited under a variety of different monikers. - Data Quality at Airbnb This worked great at first because it was fast, but eventually they ran into issues around: • Data quality • Ownership • Standards • Governance The truth is that, speed alone isn’t the answer to your data accessibility challenges. (View Highlight)

• If your data strategy is built on nothing more than unstructured data swamps, CSVs dumped into Azure Blob Storage, and a tangle of Python and SQL scripts, it’s only a matter of time before it all collapses. (View Highlight)