Data Engineer – Palantir Foundry

The core responsibilities for the job include the following:

Palantir Data Engineering and Analytics;
• Design, develop, and maintain scalable, modular data pipelines using Foundry Pipeline Builder (visual and code-based).
• Ingest, integrate, and process data from diverse sources (S3 RDBMS, REST APIs, flat files, etc. ).
• Implement advanced data processing techniques: incremental processing, anomaly detection, geospatial transformations, and time series analysis using PySpark, Python, and Foundry’s no-code tools.
• Parse and process various data formats (CSV, JSON, XML, Parquet, Avro) within Foundry.
• Reuse and modularize functions and parameters for efficient, maintainable pipeline management.
• Leverage LLMs for translation, classification, and data enrichment in pipelines via Palantir AIP Logic.

Ontology and Schema Management:
• Create and manage ontologies using Ontology Manager and Object Explorer to model business entities, relationships, and data lineage.
• Implement property-level and object-level access controls for secure data modeling and compliance.

Data Quality, Validation, and Monitoring:
• Design and implement Master Data Management (MDM) and Reference Data Management solutions to ensure consistency and accuracy of key business entities across the organization.
• Lead efforts in entity resolution, de-duplication, and golden record creation within Palantir or integrated MDM platforms.
• Implement data validation, health checks, and monitoring for production-grade reliability.
• Ensure data integrity, quality, and consistency across all stages of the data lifecycle.
• Set up automated alerts and dashboards for pipeline health and anomaly detection.

Data Security and Governance:
• Enforce data privacy, security, and compliance standards (RBAC, audit logs, data masking) within Palantir and cloud environments.
• Document data lineage, transformations, and access controls for auditability and governance.

Collaboration and Best Practices:
• Work closely with business analysts, data scientists, and product owners to translate requirements into robust data solutions.
• Mentor junior engineers and analysts, contribute to code reviews, and champion best practices.
• Document technical designs, workflows, and user guides for maintainability and knowledge transfer.

Data Analysis and Visualization:
• Perform data profiling, cleaning, joining, and enrichment to support business decision-making.
• Conduct statistical and exploratory analysis to uncover trends, patterns, and actionable insights.

Dashboarding and Reporting:
• Develop and manage interactive dashboards and reports using Palantir Contour, Quiver, and other BI tools (Tableau, Power BI, Looker).
• Build pivot tables, advanced reporting, and custom visualizations tailored to business needs.
• Leverage Palantir’s visualization modules for real-time and historical data analysis.

Cloud Platform Integration
• Integrate AWS, Azure, or GCP data engineering and analytics services (Glue, Data Factory, BigQuery, Redshift, Synapse, etc. ) with Palantir workflows.
• Design and implement end-to-end data pipelines that bridge Palantir and cloud-native ecosystems.

API and Microservices Integration:
• Develop and consume RESTful APIs, GraphQL endpoints, and microservices for scalable, modular data architectures.

DevOps and Best Practices:
• Implement CI/CD pipelines for data pipeline deployment and updates (Foundry, GitHub Actions, Jenkins, etc. ).
• Apply containerization (Docker) and orchestration (Kubernetes) for scalable data processing.

Agile Collaboration:
• Work in Agile/Scrum teams, participate in sprint planning, and contribute to continuous improvement.

Requirements:
• Experience: 7-12 years in data engineering, data analysis, or related roles; 1-2 years on Palantir Foundry, Pipeline Builder, Contour, Quiver, or strong experience with AWS/Azure/GCP data engineering and analytics services.
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, Mathematics, or a related field.

Certifications (Preferred but not mandatory):
• Palantir Foundry Data Engineer/Analyst Certification.
• AWS/Azure/GCP Data Engineering or Analytics Certifications.
• Relevant BI/Visualization tool certifications.

Palantir Platform:
• Data pipeline development, transformation, and cleaning (Pipeline Builder, Code Workspaces).
• Ontology creation, management, and data lineage (Ontology Manager, Object Explorer).
• Data validation, health checks, and monitoring in production pipelines.
• Data security, RBAC, audit logging, and compliance within Palantir.
• Dashboarding and visualization (Contour, Quiver).
• LLM integration for data enrichment (AIP Logic).

Data Engineering:
• Proficiency in SQL and Python; experience with PySpark is highly desirable.
• Experience with data ingestion, integration, aggregation, and transformation from multiple sources.
• Geospatial data processing, time series analysis, and anomaly detection.
• Parsing and processing structured, semi-structured, and unstructured data.

Data Analysis:
• Data profiling, cleaning, joining, and enrichment.
• Exploratory and statistical analysis.
• Dashboarding, reporting, and advanced visualization (Contour, Quiver, Tableau, Power BI).

Cloud Platforms:
• Hands-on experience with AWS, Azure, or GCP data engineering and analytics services.
• Integration of cloud services with Palantir workflows.

General Skills:
• Strong analytical, problem-solving, and communication skills.
• Experience working in Agile/Scrum environments.
• Ability to mentor and guide junior engineers and analysts.

Preferred Skills:
• Experience with Palantir’s Pipeline Builder, Code Workspaces, and advanced data transformation modules.
• Exposure to LLMs for data translation and enrichment.
• Familiarity with data governance, security, and compliance best practices.
• Prior experience in industries such as finance, healthcare, manufacturing, or government.
• Experience with open-source data engineering tools (dbt, Great Expectations, Delta Lake, Iceberg).
• Knowledge of CI/CD, DevOps, and automation tools.