We are seeking a skilled Data Engineer to design, develop, and maintain data pipelines and processing frameworks for large-scale analytics.
The ideal candidate will have strong expertise in Python, PySpark, and modern data orchestration tools, with experience in cloud-based data platforms.
Key Responsibilities
- Develop and optimize data pipelines using Python and PySpark.
- Implement workflow orchestration using Airflow for ETL processes.
- Design and maintain scalable data solutions leveraging BigQuery and Spark.
- Collaborate with data scientists and analysts to ensure efficient data access and transformation.
- Ensure data quality, integrity, and compliance with governance standards.
- Monitor and troubleshoot data workflows for performance and reliability.
Required Qualifications
- Proven experience in Python programming for data engineering tasks.
- Strong hands-on experience with PySpark and distributed data processing.
- Expertise in Airflow for workflow orchestration.
- Proficiency with BigQuery and Spark for large-scale data analytics.
- Solid understanding of data modeling, ETL concepts, and performance optimization.
Preferred Qualifications
- Experience with cloud platforms (AWS, GCP, or Azure).
- Familiarity with CI/CD pipelines and DevOps practices for data engineering.
- Knowledge of data streaming tools like Kafka.
#J-18808-Ljbffr