How to Build a Data Platform: The Six Key Components

The Six Components

In my previous post, I wrote about why and when a company needs a data platform. The next question is: if you decide to build one, what goes inside it?

A data platform is a set of tools and services, not one piece of software you install. So two platforms can look very different. A startup with a few engineers and a global company with thousands of employees will pick very different tools, because their data volume, budget, and security rules are different. But when I put the tools aside, I almost always see the same six parts.

The top row is the path of the data. It comes in through ingestion, lands in storage, gets cleaned and modeled in transformation, and then people and systems use it through consumption. The two layers at the bottom, metadata management and governance, are not steps. They sit under everything, and they decide if people can find the data and trust it.

I will go through them in this order. For each one I also name a few well-known tools. Tools change every few years, so please don’t take the lists too seriously.

Data Ingestion

Ingestion brings data into the platform: from business databases, events from apps and IoT devices, log files, and SaaS tools through their APIs. Nothing else works until data comes in, so this is usually where a data platform starts.

The first decision is batch or streaming. Batch loads data in chunks on a schedule. Streaming moves data continuously. In practice you end up with one of three designs:

• Batch only. A scheduled job loads the data. If the business only needs yesterday’s numbers, reading the source once a day is enough, even if the source itself produces data in real time.
• Streaming only (Kappa). For example, a CDC (Change Data Capture) tool reads the change log of a database and sends every change into the platform as a stream. When everything is handled as a stream, including reprocessing old data, it is called the Kappa architecture.
• Both (Lambda). A batch layer reprocesses all the data regularly and gives complete but slow results. A speed layer handles only the newest data that the batch layer has not reached yet, and a serving layer merges the two. The problem is that you write and maintain the same logic twice.

I would start with batch, almost every time. Add streaming only when the business really needs fresh data, like fraud detection or live operations dashboards. Streaming pipelines are much harder to build, test, and debug. To be honest, I have seen more teams suffer from streaming they didn’t need than from missing streaming they really needed.

The second decision is ETL or ELT. With ETL (Extract, Transform, Load), you transform the data before loading it, so usually the raw data is not kept. With ELT, you load the raw data first and transform it later inside the platform. For a new platform I prefer ELT. Different use cases can share the same raw data, and when a transformation has a bug, you rebuild from the raw data instead of asking the source system to send everything again. It needs more storage, but storage is cheap now.

Example tools: Fivetran and Airbyte (connectors), Debezium (CDC), Apache Kafka (event streaming).

Data Storage

Storage is where the data lives after it comes in. Today there are three main options:

• Data warehouse. A database built for analysis (OLAP), not for running an application (OLTP). Data is structured, and SQL is the main language. Cloud warehouses store data by columns and run queries in parallel on many machines, so they are fast for analytics. They also handle semi-structured data like JSON quite well now.
• Data lake. Files in their original format, in cheap object storage (in the past, on HDFS). A lake can hold almost anything: tables, JSON, logs, documents, images, videos. Engines like Spark read these files, usually after converting them to a columnar format like Parquet. Lakes and warehouses are often used together: the lake keeps the raw data, the warehouse keeps the cleaned results.
• Lakehouse. If you use a lake and a warehouse together, you keep two copies of a lot of data. A lakehouse adds warehouse features directly on top of the files in the lake, using open table formats. You get transactions, schema enforcement, updating or deleting single rows (important for GDPR deletion requests), and time travel. And because the data stays in an open format, several engines can read the same tables, so you are less locked into one vendor.

Which one to choose? If most of your data is structured and mainly used for BI, in my opinion a cloud warehouse alone is enough, and it is the simplest to run. A lake or lakehouse makes more sense when you have a lot of semi-structured or unstructured data, very large volume, or heavy machine learning work.

Whichever you choose, organize the data in layers:

• Raw: a copy of the source data, untouched, so you can always rebuild from it.
• Staging: the cleaned version, with fixed data types, consistent names, and duplicates removed.
• Modeled: where the business logic lives. Most teams I have worked with use dimensional modeling here, which means fact tables for business events like orders, and dimension tables that describe them, like customer and product. Analysts understand it easily, and it is fast to query.
• Serving: summary tables, metrics, and APIs made for specific users, so their queries stay simple.

Example tools: Snowflake and Google BigQuery (data warehouses), Databricks (lakehouse platform), Apache Iceberg, and Delta Lake (open table formats).

Data Transformation

Transformation is cleaning, joining, aggregating, and applying business logic. In my opinion, this is where most of the long-term cost of a data platform is, because the logic grows with the business and almost never gets smaller. I have two strong opinions here.

The first one is SQL. Write as much business logic in SQL as you can. Modern SQL can express most of it, with window functions, arrays, JSON functions, and so on. The main benefit is that more people can read it. When a number looks wrong, an analyst can open the SQL and help find the bug, instead of everyone waiting for the one engineer who wrote a complicated program. For machine learning features or complex streaming logic, you still need Python, Java, or Scala, and that is fine.

The other one is orchestration. Data jobs depend on each other: this table must be ready before that report runs. An orchestration tool schedules the jobs, manages these dependencies, retries failures, and shows what is running. Please use a real one. A homemade scheduler looks easy in the first week. Then it needs retries, then backfills, then alerts, then a UI, and one day you realize you are maintaining your own small Airflow.

Example tools: dbt (SQL transformations), Apache Spark (batch processing), Apache Flink (stream processing), Apache Airflow, and Dagster (orchestration).

Metadata Management

Metadata is data about your data: which tables exist, what each column means, who owns it, where it comes from, and how fresh it is. When the platform is small, all of this is in a few people’s heads, and that works fine. When the platform grows, it stops working. In growing companies, the complaint I hear most is not about slow queries. It is that people cannot find the data they need, or they don’t know which of three similar tables is the right one.

Two tools help here:

• Data catalog. It is like a search engine for your data. People search for a table, read what it and its columns mean, and see who owns it and when it was last updated. Tags help too. If you tag the columns with personal information, governance rules can apply to them automatically.
• Data lineage. It shows where each table comes from and where it goes. When a dashboard number looks wrong, lineage tells you which upstream tables to check. Before you change a table, it tells you which tables, reports, and teams downstream will be affected. You don’t need to draw it by hand. With OpenLineage, an open standard, tools like Airflow, Spark, Flink, and dbt send an event for each job run with the datasets it read and wrote, and a backend builds the graph from these events.

Example tools: DataHub and OpenMetadata (data catalogs), Databricks Unity Catalog (catalog built into a platform), OpenLineage (lineage).

Data Governance

Governance sounds like a boring word, and some parts of it are boring. But it is about very practical things: the right people can use the right data, they can trust it, and the platform does not cost more than it should. Below are the parts I think matter most.

Ownership. Every important dataset and pipeline needs a clear owner, usually a team, not one person. The owner answers questions, approves changes, and fixes the data when something breaks. Please record the owner in the data catalog, so anyone can find who to ask. Tables without an owner are the ones nobody dares to change or delete, so they just stay there forever. When people change teams or leave, hand over their ownership on purpose.

Access control. Encrypt data in storage and in transit. Give people read access, and let only pipeline service accounts write to production tables. Know where the personal data is, and mask or restrict it. Grant access by groups and roles, not person by person. Otherwise after a year nobody can tell who has access to what.

Data lifecycle. Storage is cheap, but that is not a reason to keep everything forever. Decide how long each kind of data is kept, move old data to cheaper storage, and delete what you don’t need. Also make sure you can delete one person’s data across the whole platform when a privacy regulation like GDPR requires it.

Cost. In the cloud, the bill can grow faster than the data. Usually a few things take most of the money: some heavy queries, dashboards that refresh much more often than anyone looks at them, old pipelines still running for tables nobody uses. So first make the cost visible. Tag compute and queries by team and pipeline, set budgets with alerts, and shut down idle compute. Showing each team its own cost helps more than I expected. Once people see their numbers, many of them start to optimize by themselves.

Data Quality

From my experience, data quality decides whether a data team is trusted or not. One wrong number in front of the leadership team can undo months of trust, and it takes a long time to get it back.

“Good data” is too vague to measure, so I break it into a few dimensions and decide which ones matter for each dataset:

To actually track them:

• Test before publishing. Load new data into a staging table, run the checks, and only move it to production if they pass. This is often called write-audit-publish. Bad data should stop here, not show up silently on a dashboard.
• Monitor over time. A job can succeed and still produce wrong data. For important tables, track a few numbers like row count, freshness, null rate, and totals, and alert when they move far from normal. This catches problems nobody thought to write a test for.
• Catch problems at the source. Many issues come from upstream, like a renamed column or a field whose meaning changed. Agree with the producing team on what the data should look like. Some teams write this down as a data contract and check it during ingestion.
• Keep alerts meaningful. Decide which failures stop the pipeline and which only warn. If the team gets hundreds of alerts a day, they will ignore all of them, including the important one.

For the most important datasets, like executive dashboards, regulatory reports, and anything customer-facing, set an SLA (Service Level Agreement). It is a specific promise, for example “the daily orders table is complete for the previous day by 7 AM Eastern time”, with a named owner. Then nobody argues about what “on time” means. Core upstream tables need stricter SLAs than the reports built on them, since everything depends on them. And alert before the SLA is missed. If a source is late at 5 AM, you want to know at 5 AM, not at 7.

When bad data gets through anyway (it will), I handle it in this order:

1. Stop it from spreading. Pause the pipeline, or move the few bad rows to a quarantine table.
2. Tell the people who use it. Use lineage to find who is affected, and tell them before they find it themselves in a meeting.
3. Fix the root cause. It is often upstream, so you may need the source team’s help.
4. Repair the data. Rerun the affected period from raw data. This is only safe if your jobs give the same result when run twice, so design them like that from the beginning.
5. Prevent it next time. Add a check that catches the same problem.

When I am not sure if the data is right, I hold it and tell people it is late. Most people are OK with a late report. A decision made on a wrong number is much harder to fix.

Example tools: Great Expectations, Soda, and dbt tests (data quality checks), Monte Carlo (data observability).

Data Consumption

This is where the platform finally creates value: BI dashboards and reports, ad-hoc SQL and notebooks, machine learning, data products and APIs, and sending data back into business tools like the CRM (often called reverse ETL).

Two things make this part much smoother:

• Define key metrics once. Put metrics like revenue or active users in one shared place, such as a semantic layer, and let every dashboard use it. That is how you avoid two teams showing two different revenue numbers in the same meeting.
• Make data easy to reach. Data that only exists in log files or a folder nobody can query is almost the same as having no data. Sometimes a federated query engine, which can query several sources in one SQL statement, is easier than copying everything into one system.

Example tools: Tableau, Looker, and Apache Superset (BI).

Where to Start

After six sections this looks like a lot of work, and it is. But you don’t need all of it on day one, and I don’t recommend trying.

A good first version only needs the top row. Ingest the most important sources, store them in a cloud warehouse or lake with a raw layer and a modeled layer, transform them with SQL on a real orchestration tool, and put a BI tool on top. Even at this stage, add basic data quality checks and access control. They are much harder to add later.

After that, add the rest when you feel the pain. Add a catalog when people keep asking which table is the right one. Add lineage when a change breaks a dashboard and nobody knows why. Add SLAs when other teams start to build on your data, and more governance when an audit or a regulation asks for it.