Cloud Data Engineer Roadmap: Skills, Stack, and Career Arc

Cloud data engineer vs cloud engineer vs data scientist: clearing up the confusion

These three roles are routinely confused by people entering the field, and sometimes by organisations advertising for them. The distinctions are real and matter for career planning.

Cloud engineer

A cloud engineer builds and manages the infrastructure that runs software — compute, networking, storage, security, deployment pipelines. Their primary concern is infrastructure availability, reliability, and cost. They work with application teams and platform teams. Data is something that lives in databases and object storage in their systems; they manage it as infrastructure, not as a subject of analysis.

Data scientist

A data scientist uses statistical methods and machine learning to extract insights and build models from data. Their primary tool is Python (typically with pandas, scikit-learn, or PyTorch) or R. They work on model development, feature engineering, and statistical analysis. Infrastructure is mostly invisible to them — they work in notebooks and pipelines that other people build. Data scientists care deeply about the data itself; they care about infrastructure only when it limits their analysis.

Cloud data engineer

A cloud data engineer builds the infrastructure that moves, transforms, and stores data at scale. They are the people who build the pipelines that take raw data from source systems and make it available in usable form for analysis and machine learning. They sit between the cloud engineer and the data scientist: they work closely with cloud infrastructure (using cloud storage, compute, managed data services), but their primary output is data pipelines and data systems, not general infrastructure or statistical models.

The defining characteristic: a cloud data engineer is infrastructure-competent and data-focused. They can write scalable data processing code, design a data warehouse schema, and operate the cloud services that run those pipelines — but they are not writing Kubernetes operators or designing multi-region network topologies.

The cloud data engineering stack

Data engineering has a larger and more fragmented tooling ecosystem than cloud engineering or DevOps. Part of the career map is understanding which tools matter and which are context-dependent.

Processing frameworks

Apache Spark is the dominant large-scale data processing framework. It runs distributed computations across a cluster, making it possible to process datasets that would take days on a single machine in minutes. Most large organisations running data pipelines at scale use Spark in some form — whether through EMR (AWS), Dataproc (GCP), Azure HDInsight, or Databricks (which runs on all three major clouds).

Understanding Spark means more than using its Python API (PySpark). It means understanding partitioning, shuffle operations, broadcast joins, and execution plans — the concepts that explain why one Spark job takes 2 minutes and another takes 4 hours on the same dataset.

Flink is the leading alternative for streaming workloads — continuous processing of data as it arrives, rather than processing it in batches. Many organisations use Spark for batch and Flink (or Kafka Streams) for streaming. Flink expertise is less common and therefore more valued in roles with streaming requirements.

Orchestration

Apache Airflow is the most widely used pipeline orchestration tool. It defines data workflows as directed acyclic graphs (DAGs) — a series of tasks with dependencies between them. Airflow is imperfect (it has scaling limitations and complex operational overhead) but it is so widespread that knowing it is essentially required for data engineering roles.

Prefect and Dagster are modern alternatives that address some of Airflow’s limitations. They are gaining adoption, particularly at organisations starting fresh rather than inheriting an Airflow setup. Understanding Airflow first, then learning one alternative, is the most practical approach.

Transformation

dbt (data build tool) has become the standard tool for transforming data inside a data warehouse. It uses SQL (with Jinja templating) to define transformations as models, with built-in testing, documentation, and lineage tracking. dbt is remarkable because it brought software engineering practices (version control, testing, code review) to the analytics SQL layer.

Cloud data engineers who know dbt well are in strong demand. It is one of the tools where moderate competence — being able to write models, tests, and macros, and understanding how compilation and materialisation work — is often enough to be very valuable.

Data warehouses and lakes

BigQuery (GCP), Snowflake (multi-cloud), and Redshift (AWS) are the three dominant cloud data warehouses. Understanding at least one deeply — its query optimisation, partitioning and clustering, cost model, and loading patterns — is expected in data engineering roles.

Delta Lake, Apache Iceberg, and Apache Hudi are open table formats that bring ACID transaction semantics and time-travel queries to data stored in cloud object storage. These are increasingly important in modern data architectures (“lakehouse” patterns) and are growing rapidly in adoption.

Messaging and streaming

Apache Kafka is the dominant event streaming platform. Data engineers working with real-time or near-real-time pipelines need to understand Kafka producers, consumers, topics, partitions, and consumer groups. Managed Kafka services (Confluent Cloud, MSK on AWS, Event Hubs on Azure) reduce operational overhead.

Languages

Python is the primary programming language for data engineering — Spark, Airflow, dbt macros, pipeline logic. SQL is equally important — data engineers write a lot of complex SQL, and the ability to write efficient queries and understand query execution plans is essential. Scala is historically significant in Spark development and is still used in some organisations, but Python has largely displaced it for new development.

Cloud infrastructure knowledge for data engineers

Cloud data engineers do not need the infrastructure depth of a cloud engineer, but they need more than a data scientist. The relevant areas:

• Cloud storage: Deep understanding of object storage (S3, GCS, ADLS) — storage tiers, lifecycle policies, partitioning strategies, and cost implications of different access patterns
• IAM and data access control: How to grant data access to pipelines, analysts, and services securely — especially important when data includes PII or financial records
• Compute for data: EC2/Compute Engine/VMs, managed Spark clusters (Dataproc, EMR), serverless compute (Cloud Functions, Lambda), and how to choose the right execution environment for a pipeline
• Networking basics: VPCs, private endpoints for data services, and how to ensure data does not traverse the public internet unnecessarily
• Terraform: Provisioning data infrastructure as code — useful but not always required; many data engineering teams use managed services and rely on cloud engineers or DevOps for the IaC layer

The distinguishing infrastructure competence for data engineers is understanding how cloud data services work internally: how BigQuery’s distributed query engine processes a large join, why Spark shuffle is expensive, how Kafka consumer groups achieve parallel consumption. This is more important than broad cloud platform knowledge.

Cloud certifications relevant to data engineering

Cloud provider data certifications are well-aligned with data engineering roles:

• Google Cloud Professional Data Engineer: Highly regarded, covers BigQuery, Dataflow, Pub/Sub, Dataproc, and data architecture patterns. One of the more practically useful cloud certifications available.
• AWS Data Engineer Associate: Covers EMR, Glue, Kinesis, Redshift, and data lake architecture. Newer than the GCP equivalent but increasingly valued for AWS-focused roles.
• Databricks Certified Associate Developer for Apache Spark: Vendor-specific but widely recognised because Databricks is so prevalent in the data engineering ecosystem. Practical exam that tests real Spark coding ability.
• dbt Certification: dbt Labs offers certifications for dbt Analytics Engineers. Still newer but recognised in organisations that have adopted dbt heavily.

A practical combination for most data engineering careers: one cloud provider data certification plus the Databricks Spark cert. These two credentials demonstrate both platform knowledge and core processing framework competence.

Data engineering career stages

Junior data engineer (0–2 years)

Entry-level data engineering roles typically require Python and SQL competence, familiarity with at least one cloud platform, and ideally some exposure to a pipeline tool. The work: writing and extending data pipelines under guidance, debugging failing jobs, helping with data quality issues, maintaining existing pipelines.

The most effective way to get an entry-level data engineering role without prior professional experience: build a portfolio project that demonstrates an end-to-end pipeline. Ingest public data (government datasets, OpenStreetMap, sports APIs) into a cloud data warehouse via an Airflow or Prefect DAG, transform it with dbt, and expose it through a simple dashboard. This demonstrates the full stack in a way that certificates alone do not.

Mid-level data engineer (2–5 years)

You own data pipelines end to end — design, implementation, monitoring, and incident response when they fail. You make decisions about how new data should be ingested, what the schema should look like, and which tools to use. You work closely with analysts and data scientists to understand their requirements and translate them into pipeline designs.

At this stage, data modelling becomes important. Understanding dimensional modelling (star schema, slowly changing dimensions), modern approaches (Data Vault, wide table patterns in BigQuery and Snowflake), and when to use each — this is where senior data engineers are distinguished from mid-level ones.

Senior data engineer (5–8 years)

Senior data engineers design the data platform. They make architectural decisions: which data warehouse, how the ingestion layer works, how streaming and batch are unified, how data quality is enforced at scale. They mentor junior and mid-level engineers and represent data engineering in cross-functional technical discussions.

The scope expands at this level to include data governance: lineage tracking, PII handling, access control at the data level, and compliance with GDPR or similar regulations. Senior data engineers who understand both the technical and governance dimensions of data management are uncommon and well-compensated.

Staff / Principal data engineer or Data Architect

Staff data engineers define the data strategy across an organisation. They choose the foundational platforms, define the standards that all data teams follow, and own the long-term technical roadmap for how the organisation manages its data. The role is senior enough that it overlaps with data architect — some organisations distinguish these, others do not.

The cloud data engineer salary guide covers compensation at each of these stages in the UK and US.

Moving into cloud data engineering from adjacent roles

From general cloud engineering

Cloud engineers transitioning to data engineering need to develop the data-specific skills: Spark or Flink, pipeline orchestration (Airflow), data warehouse knowledge, and dbt. The infrastructure background is a genuine asset — you already understand how cloud services work, which many data engineers lack. Focus on the data processing and modelling layer specifically.

From software engineering

Software engineers bring strong coding skills that serve them well in data engineering — Spark jobs are complex distributed programs that benefit from good software engineering practices. The gap is typically the data domain knowledge: dimensional modelling, pipeline reliability patterns, and the practical challenges of working with messy, late-arriving, and inconsistent data.

From data analysis or analytics

Analysts with strong SQL and Python skills who want to move into engineering need to develop the infrastructure and operational layer. Building and operating pipelines — handling failures, retries, backfills, monitoring — requires thinking about reliability and automation in ways that analysis work does not. The guide to switching to a cloud career covers the general transition from adjacent technical roles in more detail.