Apache Spark on AWS: DataFrames, Lazy Evaluation, Glue vs EMR

What Spark actually is

Imagine you have a CSV file with 500 million rows. A single laptop cannot hold that in memory, and even if it could, running calculations on it one row at a time would take hours.

Spark solves this by splitting the data into chunks. Each chunk goes to a different machine in a cluster. All the machines work at the same time. Spark coordinates them, collects the results, and gives you the answer.

That is the core idea. One coordinator (the driver) plus many workers (the executors). The driver figures out what needs to happen. The executors do the actual work on their slice of data.

Spark is not a database. It does not store data. It reads data from storage (usually Amazon S3 on AWS), processes it, and writes the results back. Think of it as a very powerful transformation engine that happens to run across many machines at once.

Why AWS engineers use Spark

Spark is the most widely used tool for large-scale data processing in cloud environments. On AWS specifically:

• AWS Glue uses Spark as its execution engine. Every Glue ETL job you write runs on Spark under the hood.
• Amazon EMR is a managed Spark cluster service, giving you full control over the Spark version and configuration.
• Most data lake architectures on AWS rely on Spark to transform raw S3 data into clean, queryable formats like Parquet.
• Spark handles the workloads that single-machine tools cannot: joining two 500 GB tables, aggregating billions of events, running daily ETL across a growing data warehouse.

You do not need to understand Spark’s internals to write basic ETL code. But knowing the core concepts (lazy evaluation, partitions, shuffles) makes debugging and performance tuning much faster.

How Spark works, step by step

When you run a PySpark script, here is what actually happens:

1. Read. Spark reads data from a source, usually S3 files in formats like Parquet, CSV, or JSON. It does not load everything into memory at once.

2. Partition. Spark splits the data into partitions: smaller chunks distributed across the cluster. Each executor gets one or more partitions to work on. More partitions means more parallelism, up to a point.

3. Plan. As you write transformations (filter, join, groupBy), Spark does not execute them immediately. It records each instruction and builds a logical plan. This is called lazy evaluation.

4. Optimise. When an action (like write() or count()) is called, Spark’s Catalyst optimiser analyses the full plan. It may reorder operations, push filters earlier, or combine steps to reduce data movement.

5. Execute. Spark sends tasks to executors. Each executor processes its partitions in parallel. Results are aggregated and returned, either back to the driver or written directly to S3.

The key insight: your code describes what to do, and Spark decides how to do it efficiently across many machines. This separation is what makes lazy evaluation so powerful.

Core concepts you actually need

Driver and executors

Every Spark cluster has two roles. The driver is where your code runs. It plans the work, divides it into tasks, and coordinates the cluster. The executors are the worker nodes. They receive tasks from the driver, process their data partitions in parallel, and send results back.

This matters practically because the driver has limited memory. Calling collect() pulls all data from every executor back to the driver, which crashes if the dataset is large. Most of your data processing should happen on the executors, not the driver.

Partitions

A partition is a chunk of rows assigned to one executor. Spark processes partitions in parallel: more partitions across more executors means faster jobs, as long as you have enough data to justify the overhead.

When you read Parquet files from S3, Spark typically creates one partition per file (or per row group for large files). You can adjust this with repartition() or coalesce(). Partition count matters most for joins and aggregations where data movement between executors is involved.

DataFrames, RDDs, and Datasets

Spark has three data abstractions. In practice, you will almost always use DataFrames.

An RDD (Resilient Distributed Dataset) is the original low-level abstraction: a distributed collection of raw objects with no schema. RDDs bypass Spark’s query optimiser, so they are harder to use and slower than DataFrames for most workloads. You will rarely write RDD code in modern Spark.

A DataFrame is a distributed table with named columns and a schema. It looks like a pandas DataFrame or a SQL table. Spark’s Catalyst optimiser can analyse DataFrame operations and produce efficient execution plans automatically. This is what you use for ETL in PySpark and in AWS Glue.

A Dataset is a typed DataFrame available in Scala and Java. In Python (PySpark), everything is a DataFrame. Ignore Datasets for AWS Python work.

Lazy evaluation

Spark does not run a transformation when you call it. It records the instruction and adds it to a plan. Only when you call an action (something that needs an actual result) does Spark compile and execute the full plan.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum as spark_sum

spark = SparkSession.builder.appName("Example").getOrCreate()

# These lines build a plan. No data is read yet.
df = spark.read.parquet("s3://my-bucket/orders/")
filtered = df.filter(col("order_date") >= "2025-01-01")
by_region = filtered.groupBy("region").agg(
    spark_sum("total_amount").alias("total_revenue")
)

# This ACTION triggers reading and executing the full plan.
by_region.show()

Transformations vs actions

Every Spark operation is either a transformation (lazy, returns a new DataFrame) or an action (triggers execution, returns a concrete result).

Common transformations (lazy)

• filter() / where() — keep rows matching a condition
• select() — choose specific columns
• withColumn() — add or replace a column
• groupBy().agg() — aggregate by group
• join() — join two DataFrames
• orderBy() / sort() — sort rows
• distinct() — remove duplicate rows (triggers a shuffle, so use carefully)
• drop() — remove a column

Common actions (trigger execution)

• show() — print the first N rows to the console
• count() — return the number of rows
• collect() — pull all rows to the driver as a Python list (dangerous on large data)
• write.parquet() / write.csv() — write the DataFrame to storage
• first() — return the first row
• take(n) — return the first n rows

Shuffle

A shuffle happens when Spark needs to move data between executors. Operations like groupBy, join, and distinct all require rows with the same key to land on the same executor. To make that happen, Spark sends data across the network and rewrites it to disk before processing resumes.

Shuffle optimisation is covered in more depth in Running Spark in AWS.

Cache and persist

If you reference the same DataFrame more than once in your pipeline, for example using it in both a join and a separate aggregation, Spark recomputes it from scratch each time unless you tell it to save the result.

df.cache() stores the DataFrame in executor memory after it is first computed. df.persist() gives you more control over the storage level (memory only, memory and disk, disk only). Use caching when a DataFrame is expensive to compute and used more than once.

# Expensive DataFrame used in two places — cache it to avoid recomputation
enriched = orders.join(customers, on="customer_id", how="left")
enriched.cache()

summary = enriched.groupBy("region").agg(spark_sum("total_amount").alias("total"))
summary.show()

anomalies = enriched.filter(col("total_amount") > 10000)
anomalies.write.parquet("s3://my-bucket/anomalies/")

enriched.unpersist()  # free memory when done

When to use Spark

Spark is the right choice when:

• Your dataset does not fit in memory on a single machine
• You need to join multiple large tables (gigabytes or larger)
• You are running scheduled ETL pipelines over growing S3 data that fit naturally into a data pipeline architecture
• You need to process data faster than a single-threaded script allows
• You are building a data lake where raw files need to be cleaned, partitioned, and written to Parquet at scale
• Your team already uses Glue or EMR and you want to stay within the same ecosystem

When Spark is overkill

Spark has real overhead: cluster startup, task scheduling, serialisation, and network I/O all add cost and latency. For smaller workloads, simpler tools are faster and easier.

• Small datasets (under roughly 10 GB): pandas is faster, easier to debug, and needs no cluster.
• One-off transformations: a Python script or AWS Lambda is simpler and cheaper.
• Simple SQL aggregations: Amazon Athena queries S3 data directly with no infrastructure. See ETL vs ELT for when push-down SQL makes more sense than a full Spark job.
• Real-time event processing: Spark Structured Streaming exists, but for AWS-native event pipelines, Kinesis with Lambda or Flink on EMR is often a better fit.

Spark on AWS: Glue vs EMR vs EMR Serverless

AWS offers three main ways to run Spark. Choosing between them is one of the most common practical decisions for data engineers.

Practical PySpark examples

These patterns cover most of what you will write in real AWS ETL pipelines.

Reading from S3, cleaning, and writing back

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_date

spark = SparkSession.builder.appName("OrdersETL").getOrCreate()

df = spark.read.option("header", "true").csv("s3://my-bucket/raw/orders/")

df_clean = (
    df
    .withColumn("order_date", to_date(col("order_date"), "yyyy-MM-dd"))
    .filter(col("order_date").isNotNull())
    .filter(col("total_amount").cast("double") > 0)
    .select("order_id", "customer_id", "order_date", "region", "total_amount")
)

df_clean.write.mode("overwrite").parquet("s3://my-bucket/processed/orders/")

Joining two datasets

orders = spark.read.parquet("s3://my-bucket/processed/orders/")
customers = spark.read.parquet("s3://my-bucket/processed/customers/")

enriched = orders.join(
    customers.select("customer_id", "country", "segment"),
    on="customer_id",
    how="left"
)

enriched.write.mode("overwrite").partitionBy("region").parquet(
    "s3://my-bucket/enriched/orders/"
)

Aggregating by group

from pyspark.sql.functions import sum as spark_sum, count, avg, date_trunc

summary = (
    orders
    .withColumn("month", date_trunc("month", col("order_date")))
    .groupBy("region", "month")
    .agg(
        count("order_id").alias("order_count"),
        spark_sum("total_amount").alias("total_revenue"),
        avg("total_amount").alias("avg_order_value")
    )
    .orderBy("month", "region")
)

summary.show(50)

Why collect() is dangerous

# DANGEROUS on large data — pulls every row to the driver
all_rows = df.collect()  # crashes the driver if df has millions of rows

# Safe alternatives
df.show(20)                          # preview first 20 rows
df.write.parquet("s3://...")         # write results to storage
sample = df.limit(1000).collect()   # only pull a small sample if you must

Repartitioning for parallelism

# Check current partition count
print(df.rdd.getNumPartitions())

# Repartition triggers a full shuffle — use before a join on an uneven dataset
df_balanced = df.repartition(100)

# Coalesce merges partitions without a shuffle — use to reduce output file count
df_small = df.coalesce(10)

Writing with partitionBy(“region”) creates one output folder per region in S3. This lets Athena and downstream tools use partition pruning, scanning only the folders they need. For more on S3 partitioning strategy, see Loading Data into Redshift, which covers how Spark-generated Parquet output lands efficiently in columnar storage.