Join a team building next‑gen data pipelines on AWS and Databricks, ensuring reliable, scalable ingestion for financial datasets. You’ll partner with platform and offshore teams to transition, optimize and monitor enterprise‑wide data flows.
What You’ll Do
- Transition existing pipelines to the enterprise data platform with Databricks.
- Stabilize and optimize pipelines for performance, scalability, and reliability.
- Implement repeatable ingestion, transformation, testing, deployment, and monitoring patterns.
- Apply secure AWS deployment and governed data access controls.
- Design metadata‑driven pipelines using Delta Lake, Iceberg, Unity Catalog.
- Establish observability: logging, alerts, dashboards for production pipelines.
- Support batch and streaming onboarding via AWS Glue, Lambda, Kinesis.
What You Need
- 5‑7 years data engineering experience building batch or streaming pipelines.
- Hands‑on with Databricks lakehouse platforms.
- Proficient in AWS services: S3, Glue, Lambda, Lake Formation, Kinesis, IAM.
- Experience with Delta Lake, Apache Iceberg, Unity Catalog.
- Knowledge of data quality, schema management, lineage, encryption, access control.
- Provide technical guidance through code and design reviews.
- Use AI‑assisted coding tools like Claude or GitHub CoPilot.
Good to Have
- Enterprise onboarding frameworks and reusable pipeline template experience.
- Familiarity with data cataloging, metadata management, and governed data products.
- Understanding of semantic modeling, such as semantic layers or canonical models.
The Opportunity
You’ll be part of S&P Dow Jones Indices, the world’s largest provider of index‑based market benchmarks. The role contributes to platforms that support over $7 trillion in passively managed assets.
