Job Title: Data Engineer
Experience: 5+ Yrs
Must Have:
* Strong experience in Python
* Expertise in Data Engineering Frameworks
* Hands-on experience with ETL processes
* CI/CD
* Experience working on GCP (Google Cloud Platform)
1. Job Summary – highlight project details/what is exciting about the role :
As a Data Engineer for Project Storm, you will architect and build robust, scalable data pipelines and infrastructure that power advanced AI-driven market prediction and recommendation solutions for the global shipping industry. This role offers the opportunity to work with state-of-the-art cloud technologies—including Google Cloud Platform (GCP), real-time data streaming, and agentic AI frameworks—to solve complex, high-impact business challenges. You will collaborate with data scientists, software engineers, and business stakeholders to ensure the seamless flow, transformation, and reliability of data across the platform. This is an ideal position for someone passionate about designing data systems that enable innovation, automation, and actionable insights at scale.
Key Excitement Factors:
• Opportunity to deliver high-impact data engineering solutions from design to production in a dynamic, global environment.
• Work with cutting-edge tools and methodologies, including GCP, Pub/Sub, BigQuery, Dataflow, and agentic AI integration.
• Direct influence on business outcomes by enabling real-time analytics and AI-driven decision-making.
• Exposure to diverse technical challenges and the chance to pioneer new approaches in cloud-based data engineering.
2. Key Responsibilities – list what the person will be doing on a day to day basis:
• Design, build, and maintain scalable, secure, and high-performance data pipelines using Google Cloud services (e.g., Pub/Sub, Dataflow, BigQuery, Cloud Storage).
• Develop and optimize ETL/ELT processes to ingest, transform, and deliver data for analytics, machine learning, and agentic AI applications.
• Collaborate with data scientists and AI engineers to operationalize models and ensure data availability, quality, and lineage.
• Implement data governance, monitoring, and validation frameworks to ensure data integrity and compliance.
• Automate data workflows and support CI/CD practices for data infrastructure using tools like Cloud Build and Cloud Composer.
• Troubleshoot and resolve data-related issues, optimize performance, and document solutions and best practices.
• Stay current with advancements in cloud data engineering, streaming analytics, and AI integration, and promote continuous improvement within the team.
3. Job Requirements (As per the 2006 Age Discrimination Act please do not specify number of years experience. Use words like Extensive, Strong, Good, Fair)
Essential Skills:
Extensive experience designing, developing, and maintaining cloud-based data pipelines and architectures.
Strong proficiency in SQL, Python, and data engineering frameworks (e.g., Apache Beam, Spark).
Good understanding of Google Cloud Platform services (Big Query, Pub/Sub, Dataflow, Cloud Storage, IAM) and experience with large-scale, real-time data processing.
Strong knowledge of ETL/ELT best practices, data modelling, and data governance.
Experience with CI/CD for data workflows and infrastructure-as-code tools.
Excellent problem-solving, critical thinking, and collaboration skills.
Nice to Have Skills.
Experience with agentic AI or integrating data pipelines with AI/ML solutions.
Familiarity with containerization (Docker, Kubernetes) and orchestration tools (Cloud Composer, Airflow).
Good knowledge of domain-specific applications (e.g., shipping, logistics, or financial markets).
Qualifications:
Strong academic background with a degree in Computer Science, Engineering, Information Systems, or a related field.
Google Cloud Professional Data Engineer or similar certifications are a plus.
Commitment to continuous learning and staying updated with the latest advancements in cloud data engineering and AI integration.
Pay: ₹509,356.57 – ₹1,066,829.63 per year
Application Question(s):
• Do you have experience in ETL
Experience:
• Data Engineer: 5 years (Required)
• GCP: 3 years (Required)
• DevOps: 1 year (Preferred)
Work Location: Remote