Senior Data Engineer | Databricks | PySpark | Delta Lake|Remote|Contract — Senior Data Engineer – India | Contract
You’ll design and operate enterprise‑grade data pipelines on the Databricks Lakehouse. The role focuses on reusable, metadata‑driven frameworks rather than one‑off ETL jobs, giving you ownership of end‑to‑end data ingestion, optimisation, and deployment at scale.
What You’ll Do
Build reusable Python ingestion frameworks and publish internal wheels to PyPI.
Optimise PySpark jobs, troubleshoot skew, shuffle, and memory bottlenecks.
Manage GitHub workflows, enforce code‑review standards, and maintain CI/CD pipelines.
Deploy Databricks Asset Bundles across Dev, UAT, and Prod environments.
Design Delta Lake solutions with Unity Catalog, medallion architecture, and time‑travel features.
Implement scalable SCD Type 2 pipelines using MERGE and audit mechanisms.
Collaborate with platform engineers to embed infrastructure‑as‑code for automated releases.
What You Need
Strong Python experience: packaging, virtual environments, and wheel distribution.
Deep knowledge of Spark internals, Catalyst optimizer, and Adaptive Query Execution.
Proficiency with GitHub (feature branching, GitFlow, pull‑request reviews).
Hands‑on expertise with Databricks Asset Bundles and CI/CD integration.
Solid understanding of Delta Lake transaction logs, ACID guarantees, and schema evolution.
Ability to design SCD Type 2 frameworks with efficient MERGE operations.
Experience implementing data security, lineage, and access‑control at scale.
The Opportunity
This is a remote contract role offering a focused, high‑impact team environment where you’ll shape the core data platform for a growing global services firm.