We are seeking an experienced Python Big Data Engineer with expertise in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate will have extensive experience with Python, AWS cloud services, big data technologies, and data engineering best practices to build scalable, high-performance data pipelines and analytics solutions.
Key Responsibilities
- Design, develop, and maintain scalable big data applications using Python.
- Build and optimize data processing pipelines using Hadoop, EMR, Spark, and Hive.
- Develop and support cloud-native data solutions using AWS services.
- Design and implement data ingestion, transformation, and processing workflows.
- Develop and maintain integrations using APIs and SQL queries.
- Perform large-scale data manipulation, transformation, analysis, and visualization.
- Analyze large datasets to extract meaningful business insights.
- Design, implement, and support CI/CD pipelines for automated deployment.
- Collaborate with cross-functional teams to deliver scalable and reliable data solutions.
- Apply analytical and problem-solving skills to identify and resolve complex technical challenges.
Required Qualifications
- 10+ years of hands‑on Python development experience for big data applications.
- Extensive experience implementing scalable data processing pipelines using Hadoop, Amazon EMR, Spark, and Hive.
- 5+ years of development experience with AWS services including EMR, Lambda, EventBridge, Step Functions, Redshift, Amazon S3, and AWS Glue.
- 5+ years of experience developing solutions involving API integrations, SQL queries, Spark, data manipulation, data transformation, analysis, and visualization.
- Strong experience analyzing large datasets and extracting meaningful insights.
- Hands‑on experience implementing CI/CD pipelines using GitLab, Terraform, or similar tools.
- Strong analytical, critical thinking, and problem‑solving skills.
- Ability to collaborate effectively across cross‑functional and matrixed teams.
Preferred Qualifications
- Experience with infrastructure as code (IaC).
- Experience optimizing large‑scale cloud‑based data engineering solutions.
#J-18808-Ljbffr