Google Cloud Data Engineering Hub — 55 GCP Data Engineering Projects in Python

55 production-grade, live-tested Google Cloud (GCP) data engineering projects in Python — BigQuery, Dataflow / Apache Beam, Cloud Composer / Airflow, Pub/Sub, Dataproc Serverless PySpark, Cloud Storage, Vertex AI, Gemini, RAG, Document AI and MCP. Real code, architecture diagrams and deploy.sh for every project — ideal for GCP data engineer portfolio projects, Professional Data Engineer exam prep and interview practice.

What is this?

A free, open-source collection of end-to-end Google Cloud data engineering projects — real code, real architecture, tested against live GCP environments.

Unlike snippet libraries, every project is a complete runnable system:

Modular Python code (not scripts)
ASCII architecture diagram
Sample data fixture
deploy.sh with step-by-step GCP setup

Built by Vishal Bulbule — Google Developer Expert, 12x GCP Certified, Founder @ TechTrapture.

🎬 Data Engineering on YouTube

📺 Full Playlist → Data Engineering on Google Cloud

➡️ View all Data Engineering videos →

🛠️ Stack Covered

Data Processing & Orchestration

Storage & Formats

AI & ML on GCP

Ops & Governance

⚡ Quick Start

git clone https://github.com/vishal-bulbule/google-cloud-data-engineering-hub.git
cd google-cloud-data-engineering-hub

# Pick any project
cd 01-bq-csv-ingestion-pipeline

# Set up Python environment
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Configure your GCP project
export GOOGLE_CLOUD_PROJECT=your-project-id   # ← replace
export GOOGLE_CLOUD_LOCATION=us-central1

# Run GCP setup (enable APIs, create resources)
chmod +x deploy.sh && ./deploy.sh

# Run the project
python main.py

📁 Projects

BigQuery

#	Project	What it builds
01	bq-csv-ingestion-pipeline	Load CSV from local disk, GCS, and multi-format (JSON/Parquet)
02	bq-analytics-reporting	Queries, DataFrames, dataset/table metadata listing
03	bq-upsert-merge-pattern	Partitioned + clustered tables, MERGE upsert, snapshots
04	bq-finops-cost-visibility	Dry-run cost estimates, INFORMATION_SCHEMA FinOps, scheduled queries
05	bq-ml-train-predict	Train + predict with BigQuery ML (BQML logistic regression)
06	bq-external-tables-ai	External tables over GCS, BigQuery AI embeddings via remote model
07	bq-streaming-inserts	Real-time streaming inserts with deduplication
55	bq-scd-slowly-changing-dimensions	SCD Type 1, 2 & 3 — MERGE SQLs, hash-based change detection, point-in-time queries, Dataform/dbt/Airflow integration notes

Cloud Storage

#	Project	What it builds
08	gcs-file-management	Full object lifecycle — create, upload, list, copy, rename, delete
09	gcs-signed-urls-lifecycle	Signed URL v4, lifecycle rules for storage cost optimization
10	gcs-high-throughput-transfer	Parallel transfer manager, in-memory streaming, pandas over `gs://`

Data Engineering

#	Project	What it builds
29	data-format-processing	CSV, JSON/JSONL, pandas transform, Parquet read/write with benchmarks
30	bigquery-etl-pipeline	End-to-end ETL: extract → transform → BQ load → incremental MERGE
31	pubsub-streaming-pipeline	Pub/Sub publish + pull subscribe with flow control
32	dataflow-template-launch	Dataflow managed template job launch (DRY_RUN safe)
33	dataproc-serverless-pyspark	Dataproc Serverless PySpark batch job submission
34	data-quality-validation-framework	Null, type, range, uniqueness, referential integrity checks

Cloud Composer / Airflow

#	Project	What it builds
51	cloud-composer-dag-basics	First DAG, operators, XComs, BranchPythonOperator, TaskGroups, retries
52	composer-bigquery-pipeline	GCS sensor → BQ load → SQL transform → validation → GCS export
53	composer-gcs-sensor-trigger	Event-driven pipeline on file drop, archive pattern, Pub/Sub notify
54	composer-dataflow-orchestration	DAG launching Dataflow Flex Template → monitoring → BQ validation

Apache Beam / Dataflow

#	Project	What it builds
47	beam-data-transformation	WordCount + CSV transform pipelines (DirectRunner)
48	beam-csv-to-bigquery-load	CSV → BigQuery Beam pipeline (live-tested load)
49	beam-advanced-patterns	Branching, side outputs, dead-letter, event-time windowing
50	beam-dataflow-production-submit	Submit Beam job to Dataflow with cost controls

Gemini

#	Project	What it builds
11	gemini-text-generation-basics	First API call, streaming, token counting, generation config
12	gemini-multimodal-analysis	Image, PDF, video/audio understanding
13	gemini-function-calling-tools	Function calling, structured JSON output, multi-turn chat
14	gemini-enterprise-features	Grounding, context caching, batch prediction (50% cost)
15	gemini-safety-thinking-budget	Safety settings, thinking budget (Gemini 2.5)

Embeddings & RAG

#	Project	What it builds
16	text-embeddings-fundamentals	Batch embeddings, task types, output dimensionality
17	semantic-search-engine	Cosine similarity, product catalog semantic search
18	text-clustering-multimodal-embeddings	Text clustering, multimodal embeddings, BQ vector storage
19	rag-quickstart-in-memory	Minimal in-memory RAG, 4 chunking strategies
20	pdf-rag-with-citations	PDF RAG with source citations + hallucination detection
21	vertex-rag-engine-faiss	Vertex RAG Engine, FAISS, BigQuery `VECTOR_SEARCH`
22	hybrid-rag-rerank	BM25 + dense vector hybrid search, cross-encoder reranking

Document AI

#	Project	What it builds
23	document-ocr-pipeline	OCR processor creation, PDF text extraction
24	form-invoice-extraction	Form key-value extraction, invoice entity parsing
25	batch-document-processing	Batch processing over GCS, layout-aware chunking for RAG

Imagen & Vision

#	Project	What it builds
26	imagen-image-generation	Imagen 4 Fast, aspect ratio, seed, watermark config
27	gemini-image-creation-editing	Gemini native image generation, editing, analysis
28	vision-api-object-detection	Bounding boxes, Vision API labels + OCR

Resource Management & FinOps

#	Project	What it builds
35	gcp-resource-inventory	All projects, Asset Inventory sweep, org-level resource search
36	iam-security-audit	Enabled API audit, IAM policy review, SA key age
37	compute-data-estate-inventory	Compute, BQ/GCS/Bigtable data estate inventory
38	billing-cost-recommendations	Billing accounts, Recommender API cost savings

Vertex AI Platform

#	Project	What it builds
39	vertex-model-catalog-explorer	List and filter all publisher models
40	vector-search-index-pipeline	Vector Search index, upsert, k-NN query
41	vertex-ml-training-evaluation	Batch prediction, fine-tuning, GenAI evaluation
42	vertex-pipelines-endpoint-finops	Vertex Pipelines, endpoint FinOps audit

MCP (Model Context Protocol)

#	Project	What it builds
43	connect-gcp-remote-mcp-servers	GCP native MCP servers (Developer Knowledge + BigQuery)
44	gemini-adk-with-mcp-tools	Gemini + MCP, ADK agent with MCPToolset
45	build-custom-mcp-server	FastMCP server with BQ + GCS tools
46	deploy-mcp-server-cloud-run	Deploy MCP server to Cloud Run, MCP Toolbox

🗂️ Project Structure

Every project follows the same layout so you always know where to look:

<project-name>/
├── README.md          ← architecture diagram (ASCII) + setup + run instructions
├── main.py            ← entry point — imports and orchestrates modules
├── <module>.py        ← focused logic (ingest / transform / load / etc.)
├── data/              ← sample fixture (CSV, JSON, text)
└── deploy.sh          ← gcloud commands with your-* placeholders

🔒 Cost Safety

Projects that create billable GCP infrastructure default to DRY_RUN=True:

Service	Default	To run for real
Dataflow jobs	`DRY_RUN=true`	`export DRY_RUN=false`
Dataproc Serverless	`DRY_RUN=true`	`export DRY_RUN=false`
Vertex AI fine-tuning	`DRY_RUN=true`	`export DRY_RUN=false`
Vector Search index	`DRY_RUN=true`	`export DRY_RUN=false`
Vertex Pipelines	`DRY_RUN=true`	`export DRY_RUN=false`
Cloud Composer env	Manual step	See `deploy.sh`

🏢 Building Enterprise Data Platforms on GCP?

These projects are the starting point. If your team is designing production systems — data pipelines, agentic AI, streaming platforms, FinOps automation — we architect and build them.

TechTrapture works with enterprise teams on:

Data platform architecture — BigQuery, Dataflow, Pub/Sub, Composer, event-driven at scale
Agentic AI on GCP — ADK agents, MCP-connected tools, multi-agent orchestration
Gemini Enterprise & Vertex AI — RAG, fine-tuning, evaluation, production deployments
GCP FinOps — cost visibility, BigQuery optimization, resource governance
Cloud architecture — Cloud Run, GKE, migration, greenfield builds

👤 About the Author

Vishal Bulbule — Google Developer Expert (GDE), 12x GCP Certified, AWS Community Builder.
Founder @ TechTrapture — building production-grade AI + data systems on Google Cloud and teaching engineers from real deployments.

🎓 TechTrapture Academy

Structured, hands-on courses on ADK, Gemini Enterprise, Vertex AI, BigQuery, and cloud architecture. Built for practitioners — cohort programs from real deployments, not toy demos.

📝 Data Engineering Interview Questions

Preparing for a GCP Data Engineering interview? Curated real-world questions covering BigQuery, Dataflow, Pub/Sub, Composer, Dataproc, Beam, and more — built by the same team behind this repo.

Contributing

PRs welcome for:

Bug fixes in existing projects
New GCP services not yet covered
Better placeholder defaults or setup instructions

Keep each project self-contained with its own requirements.txt.

License

Apache License 2.0 — see LICENSE.

Built with ☁️ by TechTrapture

LinkedIn · YouTube · Academy · Website · [email protected]

"Learn, work and share knowledge!"