Data Lake Architecture in GCP Explained: Zones, BigQuery, BigLake

Simple explanation

Think of a data lake as a central place where you dump all your raw data, no matter the format: CSV exports, JSON logs, database backups, event streams, images. You do not need to clean it or define a schema before storing it. The data sits in Cloud Storage, which is cheap and scales without limits.

Once the data is there, you organise it into zones. Raw data stays untouched in one zone. Cleaned, validated data moves to a second zone. Analytics-ready data goes to a third. When you need answers, you query it with BigQuery or BigLake SQL, pulling from whichever zone has the data you need.

Why data lake architecture matters

Traditional data warehouses require you to define the schema before loading data. That works well for structured, well-understood datasets. But when data arrives in many formats, from many sources, at unpredictable volumes, forcing it into a schema upfront slows teams down and discards information.

A data lake solves this by separating storage from processing:

• Flexible ingestion. Accept structured tables, semi-structured JSON, unstructured logs, and binary files without redesigning your schema.
• Low storage cost. Cloud Storage is significantly cheaper than warehouse storage for large volumes of raw data, especially with Nearline or Coldline classes for older data.
• Analytics when ready. Apply different schemas to the same raw data for different use cases: BI dashboards, data science notebooks, ML training pipelines.
• Reprocessing safety. Keeping raw data immutable means you can always reprocess from source if downstream transformations produce incorrect results.
• Support for multiple consumers. Data engineers, analysts, and data scientists all access the same underlying data through different tools and patterns.

For most GCP teams, a data lake works alongside BigQuery rather than replacing it. Raw data stays in Cloud Storage; clean, aggregated data serves analysts from BigQuery. The two patterns are complementary.

How a data lake architecture works in GCP

A typical GCP data lake follows this flow:

1. Data sources. Application databases, SaaS APIs, IoT devices, clickstream events, log files, third-party exports. Data arrives in whatever format the source produces.
2. Ingestion. Pub/Sub handles real-time event streams. Dataflow runs batch and streaming ingestion pipelines. Cloud Storage Transfer Service pulls from external sources. gsutil or gcloud storage cp handles ad-hoc uploads.
3. Raw zone (Cloud Storage). Data lands exactly as received. No transformations. Append-only. This is your safety net for reprocessing.
4. Curated zone (Cloud Storage). Dataflow or Dataproc jobs clean, validate, deduplicate, and convert data to columnar formats like Parquet. Data engineers own this zone.
5. Consumption zone (Cloud Storage or BigQuery). Aggregated, business-ready data optimised for read performance. Often loaded into BigQuery managed tables, or kept as Parquet files queried via BigLake.
6. Query and governance layer. BigQuery and BigLake provide SQL access across all zones. Dataplex catalogues assets, tracks lineage, and enforces data quality rules.

Core zones in a GCP data lake

Well-governed data lakes organise data into zones based on how processed and trusted the data is. Each zone maps to a separate Cloud Storage bucket (or prefix) with its own IAM policies.

Raw zone

Contains data exactly as it arrived from source systems. No transformations. This is your reprocessing safety net. Raw data must be immutable once landed. Never overwrite or delete raw data. Use Cloud Storage object versioning or retention policies to enforce this.

Curated zone

Contains data that has been cleaned, validated, deduplicated, and standardised. Sensitive fields may be masked. Data is stored in columnar formats (Parquet or ORC) so BigQuery can query it efficiently via external tables. Data engineers own this zone.

Consumption zone

Holds aggregated, business-ready data that analysts and data scientists query directly. Data here often lives in BigQuery managed tables or as Parquet files queried via BigLake. Optimised for read performance, not ingestion flexibility.

Example architecture

Here is a concrete example for an ecommerce platform:

• Sources: Clickstream events from the website, order exports from the database, application logs from Cloud Logging, marketing campaign CSVs from a third-party vendor.
• Ingestion: Clickstream events flow through Pub/Sub into the raw zone. Order exports arrive via a nightly Dataflow batch job. Logs are exported from Cloud Logging. Campaign CSVs are uploaded manually.
• Raw zone: All data lands in gs://ecom-lake-raw/ in its original format, partitioned by date.
• Curated zone: A Dataflow pipeline reads raw clickstream JSON and order CSVs, deduplicates records, enforces data types, masks customer PII, and writes Parquet to gs://ecom-lake-curated/.
• Consumption zone: Aggregated daily revenue, funnel conversion rates, and customer segment tables load into BigQuery. Infrequently queried historical data stays as Parquet in gs://ecom-lake-consumption/ and is queried via BigLake.

# Create zone buckets with uniform bucket-level access
gcloud storage buckets create gs://ecom-lake-raw \
  --location=europe-west2 --uniform-bucket-level-access

gcloud storage buckets create gs://ecom-lake-curated \
  --location=europe-west2 --uniform-bucket-level-access

gcloud storage buckets create gs://ecom-lake-consumption \
  --location=europe-west2 --uniform-bucket-level-access

File formats and partitioning

The format you choose directly affects query speed, storage cost, and tooling compatibility.

CSV

CSV is acceptable in the raw zone for preserving original source data. It is human-readable and universally supported. However, CSV is row-based with no type information, no compression by default, and no column pruning. Do not use CSV in the curated or consumption zones.

Parquet

Parquet is the standard format for analytical data in GCP data lakes. It is columnar: queries reading only a subset of columns skip unneeded data entirely. Parquet compresses well and includes type metadata. BigQuery queries Parquet via external tables or BigLake efficiently because it pushes column pruning and filter predicates down to the file scan. Use Parquet in the curated and consumption zones.

Avro

Avro is row-based with strong schema evolution support. It is well suited for event data and Pub/Sub message schemas where the schema changes over time. Use Avro for streaming and event data in the raw zone when schema compatibility matters.

Hive-style partitioning

Organise files using Hive-style directory structure that mirrors your query patterns. For time-series data:

gs://ecom-lake-curated/
  orders/
    year=2026/month=03/day=10/
      orders-001.parquet
    year=2026/month=03/day=11/
      orders-001.parquet

When you create a BigQuery external table over this path with partition detection enabled, BigQuery prunes partitions during queries. A filter on year = 2026 AND month = 3 AND day = 10 scans only that day’s directory, not the entire bucket. Without partitioning, BigQuery scans all files, which increases both cost and query time.

When to use a data lake architecture

A data lake pattern is a good fit when:

• You have large volumes of raw data in multiple formats (JSON, CSV, Avro, logs, images) that you want to store before deciding how to use it.
• You need cheap storage first and modelling later. Cloud Storage costs a fraction of warehouse storage for rarely queried data.
• Multiple teams need different views of the same raw data: BI dashboards, data science exploration, ML training datasets.
• You want to keep a historical, immutable copy of all raw data for reprocessing if business logic changes.
• Your data pipelines are ELT-style: load first, transform later.

When not to use a data lake architecture

A data lake adds complexity that is not always justified:

• Small structured datasets. If your data fits cleanly into BigQuery tables and is always structured, a lake layer adds overhead with no real benefit. Load directly into BigQuery.
• Fully modelled data requirements. If every consumer needs the same clean schema immediately, a warehouse-first approach is simpler. Skip the raw zone.
• No multi-format retention needs. If you only work with structured tables and never need to reprocess from raw, Cloud Storage adds a layer you do not use.
• Small team, limited data engineering capacity. A data lake needs ongoing governance, pipeline maintenance, and format management. If you do not have the team to support it, start with BigQuery and add a lake layer when the need arises.

Data lake vs data warehouse vs lakehouse

These three patterns serve different needs. In GCP, you often combine them. For a deeper comparison, see Data Warehouses vs Data Lakes.