Data Engineer For Python — Data Engineer – Python | Pune
Design and build high-performance Big Data architectures by developing scalable ETL pipelines and optimizing Spark frameworks. This role focuses on handling complex data products, requiring strong Python skills to solve large-scale engineering challenges. It is a great fit for engineers passionate about data optimization and system efficiency.
What You’ll Do
Design and implement Big Data product architectures.
Develop and maintain robust ETL data pipelines.
Optimize Spark job performance, parallelism, and memory usage.
Handle real-time data feeds using streaming tools.
Process complex file formats like Parquet, ORC, and Avro.
Integrate SQL and NoSQL databases into data workflows.
What You Need
Strong proficiency in Python coding and OOP concepts.
Hands-on experience with Spark RDD, DataFrame, and Dataset.
Knowledge of SQL databases like MySQL or PostgreSQL.
Experience with NoSQL databases such as Cassandra or Elasticsearch.
Demonstrated ability in performance tuning and resource utilization.
Background in developing ETL for data products.
Good to Have
Familiarity with Apache Kafka for real-time data streams.
Experience with analytical databases like Druid, MongoDB, or Hive.
Knowledge of Scala or parallel programming techniques.
The Opportunity
This role at A2Tech Consultants offers deep technical challenges in optimizing large-scale data architectures for enterprise clients.