What Is AWS Glue? Data Catalog, Crawlers, Jobs & Use Cases

Simple explanation

More concretely: Glue is a serverless Apache Spark environment. When you trigger a Glue job, AWS allocates a Spark cluster, runs your PySpark or Scala code on it, and shuts the cluster down when the job finishes. You never choose instance types, patch nodes, or manage cluster capacity. You pay per second of compute used.

Alongside the compute layer, Glue provides the Data Catalog: a shared metadata store that tracks where all your datasets live and what their schemas look like. Athena, Redshift Spectrum, and EMR all read from this same catalog, so defining a schema once in Glue makes it available across services.

What AWS Glue is actually for

Glue is a good fit for these kinds of work:

• Converting file formats at scale. Raw CSV or JSON files land in S3 and you need them converted to partitioned Parquet for efficient querying. Glue handles this across thousands of files.
• Loading a data warehouse. Pull records from RDS or DynamoDB, transform them, and load the result into Amazon Redshift. Glue is one of the primary ways this is done in AWS.
• Joining datasets from multiple sources. Combine an orders table from S3 with customer records from RDS and write a denormalised output that downstream tools can query directly.
• Building a central data catalog. Register all your S3 datasets in the Glue Data Catalog so Athena can query them with SQL, without each team hand-defining schemas in multiple places.
• Incremental ETL pipelines. Run a nightly job that picks up only new records since the last run, using Glue job bookmarks to track state automatically.
• Data lake ingestion. Route raw data from multiple sources into a structured, queryable data lake on S3 with consistent partitioning and formats.
• Schema migrations and backfills. When a source schema changes, reprocess historical data to match the new structure and overwrite the processed output.

How AWS Glue works end to end

Every Glue pipeline follows some version of this sequence:

1. Data lands in a source. Raw files drop into S3, new rows appear in RDS, or records arrive from a JDBC connection.
2. Schema is discovered or defined. A Glue crawler scans the source and writes a table definition (column names, types, file format, partition structure) to the Data Catalog. For stable schemas, teams often define the table manually instead of running a crawler every time.
3. The Data Catalog holds the metadata. The catalog now knows where the data lives, what it looks like, and how it is partitioned. Athena, Redshift Spectrum, and EMR can all use this information.
4. A Glue ETL job runs. The job reads from a catalogued table, applies transformation logic in PySpark (filtering, renaming columns, joining, aggregating), and writes the result to the output location.
5. Output lands in S3. The processed data is written in an optimised format like Parquet, partitioned by date or another dimension.
6. Downstream consumers query the result. Athena runs SQL against the processed S3 data, Redshift Spectrum joins it with warehouse tables, or the next Glue job picks it up as input.

A concrete example

Your e-commerce system exports daily order CSVs to s3://raw-data/orders/YYYY-MM-DD/. Each file has around 500,000 rows and inconsistent column naming across months.

A Glue crawler runs each morning, scans the new partition, and updates the Data Catalog with any schema changes. A Glue ETL job then reads the raw orders table from the catalog, standardises column names, filters out test orders, and writes the result as Parquet to s3://processed-data/orders/year=2026/month=05/day=11/. Athena queries the processed output with a simple SELECT. The raw CSVs stay untouched.

This pattern (raw source to catalog to transform to partitioned Parquet to query) is how most AWS data teams use Glue in production. For more on how pipelines like this are structured, see Designing Data Pipelines in AWS.

Core components

Data Catalog

The Data Catalog is a central metadata registry. It does not hold your actual data. It holds descriptions of your data: where it lives, what format it uses, what the columns are called, and how it is partitioned.

The catalog is organised into databases and tables. A Glue table is not actual data. It is a pointer to an S3 location plus a schema definition. Once a table is in the catalog, you can query it with Athena, reference it in Glue jobs, or expose it to Redshift Spectrum, all without redefining the schema in each service separately.

# List all databases in the Glue Catalog
aws glue get-databases

# List all tables in a specific database
aws glue get-tables --database-name my_data_catalog

# Inspect the schema of a specific table
aws glue get-table \
  --database-name my_data_catalog \
  --name orders

Crawlers

A crawler automates schema discovery. You point it at an S3 prefix, an RDS database, or another supported source. It samples the data, infers column names and types, detects the file format, and writes a table definition to the catalog. No manual schema entry required.

Typical crawler workflow:

1. Create a crawler and configure the data source (e.g., s3://my-bucket/orders/)
2. Set a target Glue database for the discovered tables
3. Run the crawler — it samples files, infers schema, and creates table definitions
4. Query the discovered tables immediately in Athena or reference them in Glue jobs

ETL jobs

A Glue job is a unit of ETL work. You write PySpark (or Scala) code, Glue provisions a Spark cluster, runs your job, and shuts the cluster down. Glue uses a DynamicFrame: a Glue-specific Spark DataFrame wrapper that handles semi-structured and inconsistent data more gracefully than a standard Spark DataFrame. You can convert freely between DynamicFrames and regular DataFrames.

Here is a minimal Glue job that reads a catalogued table and writes the result as Parquet. This pattern covers the majority of real Glue use cases:

import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from the Glue Catalog (table backed by raw S3 CSV)
orders = glueContext.create_dynamic_frame.from_catalog(
    database="my_data_catalog",
    table_name="raw_orders"
)

# Write to S3 as Parquet
glueContext.write_dynamic_frame.from_options(
    frame=orders,
    connection_type="s3",
    connection_options={"path": "s3://my-bucket/processed/orders/"},
    format="parquet"
)

job.commit()

The job.commit() call at the end is not optional. It saves the job bookmark state, which is how Glue remembers what data has already been processed for incremental runs. Skip it and your bookmarks will not work.

Glue supports four job types:

• Spark (ETL): Full Apache Spark on a managed cluster. Best for large-scale transformations. See Apache Spark Basics for AWS Engineers for the underlying concepts.
• Python Shell: Lightweight Python environment with no Spark. Faster startup, much cheaper. Good for small data, API calls, or orchestration scripts.
• Spark Streaming: Continuous ETL using Spark Structured Streaming for near-real-time pipelines.
• Ray: Distributed Python for ML-heavy workloads that do not need Spark.

Glue Studio

Glue Studio is a visual drag-and-drop interface for building ETL pipelines without writing PySpark code. You connect source, transform, and target nodes, and Glue Studio generates the underlying PySpark for you.

It works well for prototyping pipelines quickly, standard transformations that need no custom logic, and teams without strong Spark experience. For complex pipelines with custom business logic, multi-step joins, or transformations that need to be version-controlled and tested, writing PySpark directly gives you more control and produces code that is easier to debug.

Triggers, workflows, and job bookmarks

Triggers start jobs on a schedule (cron) or in response to an event such as another job completing. Workflows chain multiple jobs and crawlers together with dependencies: when job A succeeds, start job B; if job B fails, skip job C and send an alert.

When to use AWS Glue

Glue is the right choice when:

• Your data volumes are in the GB-to-TB range and growing
• You need a repeatable, scheduled pipeline that runs nightly, hourly, or on data arrival
• You need schema management: a central catalog that Athena, Redshift Spectrum, and EMR can all use
• You are loading data into Redshift from S3 or RDS. See Loading Data into Redshift for how Glue fits this pattern.
• Your transformation involves distributed operations that Spark handles well: joins across large tables, aggregations, window functions
• You want incremental ETL without custom state management. Job bookmarks handle that for you.
• You are building a data lake ingestion layer where raw data needs to be cleaned, partitioned, and made queryable at scale

If you are deciding between ETL and ELT (loading raw data first and transforming inside the warehouse), see ETL vs ELT in AWS for guidance on when each approach makes sense.

When not to use AWS Glue

Glue is a poor fit when:

• Your data is small (under roughly 100 MB). Use a Lambda function with pandas or a Glue Python Shell job instead. Both start in seconds and cost far less.
• You need sub-second or real-time latency. Glue is a batch and micro-batch tool. Per-event, low-latency transforms belong in Lambda or a streaming platform.
• Your transform is simple row-level logic. Filtering rows, renaming a column, or adding a timestamp does not need a distributed Spark cluster.
• You are querying data in S3 with no transformation needed. If your data is already in Parquet and you just need SQL access, Athena is the right tool.
• You need fine-grained cluster control or specific Hadoop tooling (Hive, Presto, HBase). Use Amazon EMR instead.
• You already have a mature orchestration layer (Airflow, dbt, Step Functions) that handles scheduling and dependencies. Adding Glue workflows on top may duplicate concerns.

AWS Glue vs Lambda vs EMR vs Athena

These four tools come up together in most data pipeline decisions. Here is how to pick quickly:

For a deeper look at where Glue fits in larger pipeline designs, see Data Warehouses vs Data Lakes in AWS. If you are comparing Glue, EMR, and managed Spark options, see Running Spark in AWS.

Pricing and cost traps

Glue pricing is based on Data Processing Units (DPUs). One DPU is 4 vCPUs and 16 GB of memory. You are billed per DPU-second with a 1-minute minimum per job run.

• Spark ETL jobs require a minimum of 2 DPUs and start at $0.44 per DPU-hour (varies by region)
• Python Shell jobs use 0.0625 DPUs by default, making them much cheaper for small orchestration tasks
• Crawlers are also billed per DPU-hour. A crawler on a large S3 bucket with thousands of partitions can run for 30 minutes or more.

Two other cost traps worth knowing:

• Crawler frequency. Scheduling a crawler to run every hour on a large bucket accumulates significant DPU-hours even when no new data has arrived. Trigger crawlers only when you know new data has landed.
• Reprocessing without bookmarks. Without job bookmarks enabled, every job run reads the entire source dataset. On a growing S3 data lake, this cost compounds quickly. Always enable bookmarks for jobs that run on a schedule.

IAM and security basics for Glue

Every Glue job needs an IAM execution role. This is the IAM role that Glue assumes when the job runs, the same concept as an execution role for a Lambda function. The role defines what the job is allowed to do. Without the right permissions, the job fails immediately at startup with an access denied error.

The execution role needs permission for at least the following:

• Read from the source. For S3: s3:GetObject and s3:ListBucket on the source bucket. For RDS: the role needs VPC configuration and credentials via Secrets Manager.
• Write to the destination. s3:PutObject and s3:DeleteObject on the output bucket if the job overwrites partitions.
• Access the Glue Data Catalog. glue:GetTable, glue:GetDatabase, and related permissions for the databases your job reads from.
• Write logs to CloudWatch. logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents are required for job output and error logging.
• Use Glue itself. The AWS managed policy AWSGlueServiceRole covers the Glue service permissions and is a reasonable starting point, but restrict the S3 permissions to specific buckets rather than accepting the broad default.

Apply least privilege: the execution role should only have access to the specific S3 buckets, Glue databases, and Secrets Manager secrets that the job actually uses. A broad role that grants access to all S3 buckets across your account is a real security risk in production.