What Do Data Engineers Actually Do?

Data engineering is a branch of software engineering, and it gets more attention every year as companies need more people to work on data. But the title can mean quite different jobs at different companies. Below I explain what data engineers (DEs below) do, how the job changes between companies, and then answer some questions people usually have before they switch to this role.

In one sentence, a DE solves the company’s problems around storing and processing data. The tools changed a lot over the years, from relational databases to distributed databases, and then to cloud data warehouses and data lakes. The job title changed with them, from database administrator and ETL engineer to data engineer. Today when people say DE, they basically mean someone who works on big data processing.

The work has two big parts:

• Data platform: building and maintaining the platform, so the company can store, transform, and analyze data
• Data business logic: turning business logic and processes into data pipelines in code, and managing the whole lifecycle of the data on the platform

In a real job, you may do only one of them, or both. It depends a lot on how the company is organized and what tools it chose.

At Meta and Uber, DEs work almost only on the business side. They spend most of their time calculating business metrics and building data flows on a quite mature internal platform. So their main tool is SQL. The platform itself is built and maintained by other teams of software engineers.

Many small and mid-sized companies are different. They run on a public cloud like AWS or GCP and buy ready-made data services from the cloud provider, which makes the infrastructure part much easier. A DE there builds the platform a bit like playing with Lego, picking the services that fit best. These engineers sit somewhere between DE and SDE (software development engineer), and they often own both the platform and the business logic.

Expedia, where I used to work, was in the middle. A team of SDEs maintained a set of open source data lake components, and the data team of each business line deployed them on AWS by themselves, to build and run a data platform just for their own department.

Data Platform Work

Building a data platform includes work like:

• Planning a stable and scalable storage and processing architecture. Is the incoming data mostly batch or streaming? Does the output need to be real-time, or is a scheduled update enough? How fast is the data growing? The design should come from these real needs.
• Picking an orchestration tool and building reliable data pipelines. The most popular tool is probably Airflow.
• Providing tools to support and manage the platform: data catalog, data lineage, metadata management, access control, monitoring, and alerting. You can build them in-house or buy them from a vendor.
• Providing data quality tools that monitor freshness, accuracy, and completeness.
• Providing query engines, query services, or interfaces, so downstream teams can get data in the way they need.
• Automating operations with tools like Terraform and Ansible, or the cloud’s own tools like AWS CloudFormation.
• Choosing cloud services. Knowing what the cloud providers and popular SaaS data tools offer, and deciding which ones are worth bringing in.

How much work this is depends a lot on the company. A company running its own data center probably has to build its own version of almost every part, and each part can be a whole team. A company already on the cloud can cover all of it with a few people.

Business Logic Work

Once the platform exists, DEs on the downstream teams run the real data jobs on it. Turning business needs into data jobs includes work like:

• Designing and building metrics. Metrics usually come from analysts. But if you understand how a metric is calculated and why the business needs it, it is much easier to turn it into code.
• Data modeling. Designing models that fit both the input data and the output needs.
• Handling source data in all kinds of formats, and then cleaning, standardizing, and transforming it after it lands.
• Tuning jobs and queries. Mostly SQL tuning, and how you tune depends on the query engine. Tuning processing jobs also depends on the tools, like Spark, Hive, Flink, and Kafka.
• Monitoring jobs. For example, whether the upstream database is stable, whether the platform services are healthy, and whether the data quality is good.

In real work there is no clear line between the two sides. In big and small companies, a DE’s actual job can be any mix of the points above.

Common Questions

That is the general picture, and it is probably too rough if you are really thinking about switching. So below are some questions people usually ask.

Does a Data Engineer Need Strong Coding Skills?

It depends on which side the job is on.

If the team builds parts of the data platform by itself, it expects about the same coding skills from a DE as from a software engineer. What these engineers ship is a service or software that people inside the company use to work with data.

But most DE jobs on the market are on the business side. There, the coding bar is a bit lower than for software engineers, but the SQL bar is much higher. Typical coding work is writing data processing jobs and orchestration code. DE jobs also usually ask for big data experience, because many projects need the team to understand distributed tools, and that is closely related to how well you read and write code.

What Is On-Call Like for a Data Engineer?

On the platform side, you maintain a set of data tools for internal users, so on-call means keeping these tools working and handling issues people report. Internal users are much easier to talk to and test with than external users, and they rarely send strange inputs to your software. So the pressure is relatively low.

On the business side, it depends on where your data sits in the company. A company with many business lines usually has a central data team upstream that manages the core data, and each business line has its own team for its own data downstream. The core data team has much more pressure. It has to meet strict SLAs, like data freshness, so the downstream data can be ready on time. When something breaks, they need to find a workaround quickly and backfill the data before the SLA is missed.

For downstream teams, the pressure depends on who uses their data. A BI team serves internal analysts, for example people building dashboards in Tableau. These users usually don’t need real-time data and are more tolerant of delays, so I would say this team has the easiest on-call. But if the users are external, for example a government regulator, a delay or missing data can lead to a fine or even losing a license. Then the team needs enough people to keep the data available, and the on-call pressure is quite high.

What Career Paths Does a Data Engineer Have?

As a branch of software development, the DE career path is not very different from an SDE’s. Engineers who mostly build platforms can go deeper into architecture and become a data platform architect, or move to SDE roles closer to infrastructure. Engineers on the business side can also move to data infra SDE roles, or go deeper into one use of data and become a machine learning engineer, a data visualization engineer, or a data scientist.

If you want to move into management, in my opinion a data platform team is a good place to start. As more companies make decisions based on data, the platform team is becoming a core team in many of them. Compared with business teams, it is usually more stable, and its work often has a bigger impact.