Job Description

Role Overview

We need a senior, hands-on Lead Data Engineer with deep experience building and optimizing data & ML platforms on Google Cloud Platform (GCP) . This person will own scalable data pipelines, Spark performance tuning, ML model operationalization, and platform automation across a modern cloud-native stack.

Core Technical Stack (Must-Have Depth)

Area

Tech/Tools

GCP Data & Processing

Data Fusion, Dataproc, BigQuery

Spark tuning required.

Programming

Python (strong)

Production-quality ETL + ML support.

Big Data / Hadoop

Admin-level familiarity

Cluster behavior, resource mgmt, job tuning.

Orchestration

Cloud Composer

DAG build, scheduling, dependency mgmt.

Containers & Orchestration

Docker , Kubernetes (GKE preferred; EKS OK)

Build, deploy, scale workloads.

DevOps

Jenkins (CI/CD)

Automated build/test/deploy for data/ML code.

Serverless

Cloud Functions

Event-driven glue.

Registry

GCR

Image mgmt & promotion.

ML Lifecycle

Model training, eval metrics, deployment patterns

Understand algorithms & performance tracking.

Key Responsibilities

• Architect, build, and optimize large-scale data pipelines in GCP (batch + streaming as applicable).
• Tune Spark jobs for cost, performance, and reliability in Dataproc/managed clusters.
• Develop Python-based data frameworks , transformation logic, reusable libraries.
• Operationalize ML workflows : data prep, model training triggers, performance metrics, retraining criteria.
• Build and manage Cloud Composer DAGs integrating GCP & external services.
• Containerize workloads and deploy via GKE (or EKS experience transferable).
• Implement CI/CD pipelines for data/ML code (Jenkins + Git).
• Automate environment provisioning, artifact promotion, and job observability.

Required Qualifications

• 13+ years in data engineering / big data / platform engineering.
• Recent, hands-on GCP project work (please show dates).
• Proven Spark performance tuning experience (cluster sizing, executor config, partitioning, caching).
• Advanced Python (beyond scripting-packages, unit tests, packaging).
• Dataproc + BigQuery production usage.
• Big Data admin-level understanding (YARN, HDFS, resource queues, job lifecycle).
• Built/managed Cloud Composer pipelines.
• Kubernetes (deploying data or ML workloads; Helm/YAML a plus).
• Docker image authoring & registry lifecycle (GCR).
• Jenkins CI/CD pipelines for data/ML code.
• Understanding of ML model lifecycle : feature prep, algorithm differences, eval metrics (AUC, RMSE, recall, etc.), retrain strategy.

Nice to Have

• Data Governance exposure (catalog, lineage, DLP).
• ML Ops tooling (Vertex AI, MLflow, Kubeflow).
• Infra-as-Code (Terraform).

Observability (Stackdriver, Prometheus, Grafana).

Job Tags

Similar Jobs