AWS Data Lake Architecture Explained: S3, Glue, Athena and Lake Formation

This pattern works well when you have many data sources, mixed formats, and teams with different analytical needs. Instead of loading everything into a single warehouse, you store it once in S3 and let each team query it in the format they need, using the tool that fits their workflow.

This page explains how the architecture fits together, which services do what, when a data lake is the right choice, and what mistakes to avoid. It assumes basic familiarity with Amazon S3 and focuses on AWS-native tooling.

Simple explanation

Think of a data lake like a large warehouse with open receiving docks. Deliveries (data) arrive and go straight to a receiving area (the raw zone) without being unpacked or reorganised. Workers (Glue ETL jobs) sort and repackage items into labelled bins (the processed zone). The front-of-store shelves (the curated zone) hold only what customers are likely to need today.

A label system (Glue Data Catalog) keeps track of what is on each shelf and what format it is in. Anyone with the right key card (Lake Formation permissions) can find and use what they need. Athena is the shopping assistant: you tell it what you want in plain SQL, and it goes and retrieves it directly from the shelves.

How AWS data lake architecture works

Data moves through the architecture in a predictable sequence:

1. Ingestion. Raw data lands in the S3 raw zone. Sources include Kinesis streams, application exports, database CDC feeds, third-party APIs, and file uploads. See Streaming Pipelines with Kinesis for event-driven ingestion patterns.
2. Organisation. Data is written to S3 paths partitioned by date, region, or domain. For example: raw/orders/year=2026/month=05/. Partitioning lets Athena skip irrelevant files and reduces query cost. See Partitioned Tables for how this works in practice.
3. Cataloguing. Glue crawlers scan S3 paths and register table definitions in the Glue Data Catalog, including column names, types, file format, and partition keys. Athena and Redshift Spectrum use this catalog to understand the data without reading every file.
4. Transformation. Glue ETL jobs (or Spark on EMR for larger workloads) read from the raw zone, clean and standardise the data, and write Parquet files to the processed zone. A second pass produces curated aggregations. See ETL vs ELT in AWS for when to transform before or after loading.
5. Querying. Analysts run SQL queries in Athena against the Glue catalog. Queries read directly from S3; no data is copied into a separate database. Redshift Spectrum and EMR can also query the same catalog. See Designing Data Pipelines for how querying fits into a broader pipeline.
6. Governance. Lake Formation sits above S3 and the Glue catalog and enforces who can see which tables, columns, and rows. Access decisions happen at query time regardless of which query engine is used.
7. Consumption. BI tools (QuickSight, Tableau), data science notebooks, downstream pipelines, and APIs read from the curated zone or query Athena directly.

Raw data sources
  │
  ▼
S3 raw zone  (JSON, CSV, logs — immutable)
  │
  ▼
Glue ETL / Spark on EMR
  │
  ▼
S3 processed zone  (Parquet, partitioned)
  │
  ▼
S3 curated zone  (aggregations, business-ready)
  │
  ▼
Glue Data Catalog  (metadata, schemas, partitions)
  │
  ▼
Athena / Redshift Spectrum / EMR  <-- Lake Formation (access control)
  │
  ▼
BI tools / data science / downstream APIs

Core AWS services in this architecture

Three-zone data lake layout

The standard AWS data lake pattern divides S3 storage into three zones, each with a specific purpose and set of users. All three zones live in S3, usually as top-level prefixes in one or two buckets.

Data flows one direction only: raw → processed → curated. Transformation jobs promote data between zones but never write back to an earlier zone.

s3://my-data-lake/
  raw/
    orders/year=2026/month=05/day=11/orders_20260511.json.gz
    clickstream/year=2026/month=05/day=11/clicks_20260511.log
  processed/
    orders/year=2026/month=05/day=11/
      part-00000.parquet
      part-00001.parquet
  curated/
    daily_order_summary/year=2026/month=05/day=11/
      summary.parquet

Cataloguing data with Glue crawlers

Data in S3 is just files. Athena does not know what columns they contain or how they are partitioned until you register a table in the Glue Data Catalog. Glue crawlers automate this step by scanning S3 paths, sampling files, inferring schemas, and writing table definitions.

This example creates a crawler that scans the processed orders zone on a daily schedule and registers the table in the data_lake database:

# Create a Glue crawler for the processed orders zone
aws glue create-crawler \
  --name orders-processed-crawler \
  --role arn:aws:iam::123456789012:role/GlueServiceRole \
  --database-name data_lake \
  --targets '{"S3Targets": [{"Path": "s3://my-data-lake/processed/orders/"}]}' \
  --schedule "cron(0 1 * * ? *)"

# Run manually on demand
aws glue start-crawler --name orders-processed-crawler

After the crawler runs, the data_lake.orders table appears in the catalog. Athena can immediately query it with standard SQL:

SELECT
    product_category,
    COUNT(*) AS order_count,
    SUM(amount) AS total_revenue
FROM data_lake.orders
WHERE year = '2026' AND month = '05' AND day = '11'
GROUP BY product_category
ORDER BY total_revenue DESC;

For a full breakdown of the Glue Data Catalog, crawlers, and Glue Studio, see the AWS Glue Overview.

Transforming data with Glue ETL

Glue ETL jobs run PySpark or Python Shell scripts on a fully managed Spark environment. They are the standard way to move data from raw to processed in an AWS data lake: no cluster to configure, and you pay only while the job runs.

This job reads raw JSON from S3, cleans it, and writes Parquet to the processed zone partitioned by date:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext
from pyspark.sql.functions import col, to_timestamp, trim

args = getResolvedOptions(sys.argv, ['JOB_NAME', 'source_path', 'target_path'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read raw JSON from S3
raw_df = spark.read.json(args['source_path'])

# Clean: dedup, filter bad rows, standardise types
clean_df = raw_df \
    .dropDuplicates(['order_id']) \
    .filter(col('amount') > 0) \
    .withColumn('placed_at', to_timestamp(col('placed_at'))) \
    .withColumn('customer_email', trim(col('customer_email')))

# Write as Parquet partitioned by date
clean_df.write \
    .partitionBy('year', 'month', 'day') \
    .mode('overwrite') \
    .parquet(args['target_path'])

job.commit()

For teams running very large transformation jobs or needing custom Spark configurations, running Spark in AWS covers the tradeoff between Glue and EMR in detail. If you are deciding whether to transform before or after loading, see ETL vs ELT in AWS.

Governance with Lake Formation

As your data lake grows and more teams need access, raw S3 bucket policies become difficult to manage. Lake Formation adds a fine-grained permission layer on top of S3 and the Glue catalog without moving data.

With Lake Formation you can:

• Column-level access: allow an analyst to query the orders table but hide the customer_email column
• Row-level filtering: a regional team sees only orders from their region, not the full dataset
• Table-level grants: grant access to a Glue table without giving direct S3 bucket access, so queries always go through the catalog
• Cross-account sharing: share a table with another AWS account via the catalog without copying data to a second bucket

Lake Formation permissions apply consistently whether users query through Athena, Redshift Spectrum, or EMR. The same policy covers all query engines.

When to use this architecture

A data lake on S3 is a strong fit for:

• Centralising logs and event data. Application logs, clickstream events, and IoT feeds arrive continuously and vary in schema. S3 handles this naturally; a relational database would not.
• Mixed structured and semi-structured data. If you have relational tables alongside JSON API responses and CSV exports, a lake stores them all and lets you join them at query time.
• Analytics without pre-loading. Athena queries data directly in S3. You do not need to load it into a warehouse before you can ask questions, which is useful during exploration and early-stage analysis.
• Long-term low-cost retention. S3 storage is cheap, especially with lifecycle policies that move older data to Glacier tiers. A data warehouse charges for compute and storage together even if the data is rarely queried.
• Cross-team shared datasets. Multiple teams can query the same S3 data through Athena or Redshift Spectrum without anyone owning a separate copy. Lake Formation controls what each team can see.

When not to use this architecture

A data lake is not the right first choice when:

• You need an OLTP database. A data lake is an analytics pattern. If your application is reading and writing individual records at high frequency, you need RDS, Aurora, or DynamoDB, not S3 and Athena.
• Your reporting needs are simple. If you have one database and a few dashboards, a data lake adds operational complexity without benefit. A direct Redshift connection or an RDS read replica is simpler and faster to operate.
• Your workload needs strict relational modelling from day one. If you know your schema will be stable and query patterns are well-defined, a warehouse like Amazon Redshift will be faster and easier to govern. See Data Warehouses vs Data Lakes for the comparison.
• Your team has no governance process. A data lake without naming conventions, catalog hygiene, and data ownership quickly becomes a data swamp: data accumulates in S3 but becomes impossible to trust or use.

Data lake vs data warehouse vs lakehouse

The lakehouse is the hybrid pattern: curated data lives in Redshift for fast structured queries, while raw and processed data stays in S3 and is accessible via Redshift Spectrum using the same Glue catalog. For the full comparison, see Data Warehouses vs Data Lakes.

Best practices

• Use Parquet or ORC in the processed and curated zones. Columnar formats compress better and let Athena skip columns it does not need, reducing both cost and query time.
• Design partitions around your most common query filters. If analysts almost always filter by date, partition by date. If they also filter by region, add region as a second partition key. Avoid over-partitioning: thousands of tiny files hurts performance.
• Keep the Glue Data Catalog clean. Add table descriptions and column comments when you register a table. Delete stale tables. Treat the catalog like source code: it should be accurate and up to date.
• Document owner and freshness per table. Use Glue table properties or a supplementary wiki to record who owns each dataset and how often it should refresh. Analysts need this to judge whether data is trustworthy before building on it.
• Be aware of Athena scan costs. Large unpartitioned queries get expensive quickly. Consider Athena workgroups with per-query scan limits to prevent runaway costs. See S3 Cost Optimisation for storage-side cost controls.
• Use Lake Formation for access control from the start. Retrofitting fine-grained permissions onto a lake that grew on raw S3 policies is painful. Set it up early, even if the initial policies are simple.