Data Engineer_Spark/Scala

Data Engineer_Spark/Scala – Bengaluru You’ll design and optimize large‑scale data pipelines that power Zorba AI’s analytics and AI models. The role combines hands‑on Spark development with Azure cloud engineering, giving you ownership of end‑to‑end data flows in a fast‑growing tech environment. What You’ll Do Design, build, and maintain Spark‑based ETL/ELT pipelines using Scala or Python. Develop batch and streaming jobs to handle terabytes of data daily. Tune Spark applications, configure clusters, and manage resources for peak performance. Orchestrate workflows with Apache Airflow, ensuring reliable job scheduling. Deploy and manage data solutions on Azure services (e.g., Data Lake, Synapse). Collaborate with data architects and business stakeholders to translate requirements into scalable pipelines. Monitor production pipelines, troubleshoot issues, and implement improvements. What You Need 5‑8 years in data engineering or big‑data roles. Deep, hands‑on expertise with Apache Spark (architecture, tuning, cluster optimization). Proficiency in Scala and/or Python for data processing. Experience building and scheduling workflows in Apache Airflow. Strong knowledge of Azure cloud data services and associated tooling. Solid grasp of ETL/ELT concepts and distributed data platforms. Proven problem‑solving and debugging abilities in production environments. Good to Have Experience with Azure Data Lake, Azure Databricks, or Azure Synapse. Familiarity with CI/CD pipelines and DevOps practices. Exposure to Agile/Scrum development cycles. Understanding of data‑warehousing and real‑time processing frameworks. The Opportunity Zorba AI is expanding its data platform to support AI workloads, offering you a pivotal role in shaping a high‑impact infrastructure while working