Specialist Software Engineer – Data Engineer Databricks pySpark

Specialist Software Engineer – Data Engineer Databricks pySpark — Specialist Software Engineer – Data Engineer (Databricks‑pySpark) – Bengaluru You’ll design and run high‑performance data pipelines that power critical banking analytics. The role sits at the heart of Société Generale’s Global Solution Centre, where fast‑moving data teams shape the future of finance. Join a multicultural environment that values continuous improvement and offers real impact on ESG initiatives. What You’ll Do Translate business and technical requirements into scalable data solutions. Build and optimise end‑to‑end pipelines on Databricks using pySpark. Develop data flows within Hadoop tools (Hive, HDFS, Starburst, Oozie). Tune cluster performance and manage resource usage on Databricks. Apply CI/CD, unit‑ and integration‑testing practices to all pipelines. Document specifications, runbooks, and maintain data governance artefacts. Support production, troubleshoot incidents, and participate in on‑call rotations. What You Need 4+ years in data engineering or large‑scale distributed systems. Strong pySpark expertise with performance optimisation. Hands‑on experience with Hadoop ecosystem (Hive, HDFS, YARN, Oozie, Starburst). Proficiency in Python for pipeline utilities. Solid data modelling, partitioning, and governance knowledge. Git and modern SDLC practices (branching, PR reviews). CI/CD pipeline experience (GitHub Actions, Ansible, AWX). Good to Have Experience with NiFi/Kafka for batch and real‑time ingestion. Familiarity with JFrog, Cloudera Manager, and Spark SQL tuning. Exposure to Azure/AWS cloud platforms and SAFe Agile methodologies. The Opportunity You’ll work with global teams on a mature data platform, balancing rapid delivery with rigorous quality standards.