Astyra’s client is seeking a hands-on Senior Data Engineer to support a large-scale data and technology modernization initiative. This position will focus on modernizing legacy ETL workloads, developing Spark-based data pipelines, and ensuring data accuracy and consistency as workloads transition to modern platforms.
The ideal candidate has deep hands-on experience with Spark and Scala and a strong background designing, building, optimizing, and validating enterprise data pipelines.
What You’ll Do
- Design, build, and maintain scalable data pipelines using Spark and Scala
- Support the migration of legacy ETL workloads into a modern Spark-based data framework
- Develop and enhance data transformation and processing solutions
- Perform data comparison, reconciliation, and validation between legacy and target platforms
- Identify and troubleshoot data discrepancies throughout the migration process
- Develop data quality checks and controls to ensure completeness and accuracy
- Optimize Spark jobs and pipelines for performance, scalability, and reliability
- Work closely with data engineers, architects, and other technical teams throughout the modernization effort
- Support testing, deployment, troubleshooting, and ongoing maintenance of production data pipelines
- Contribute to broader feature engineering and data modernization initiatives
What We’re Looking For
- 7+ years of experience in data engineering or a related technical discipline
- Strong hands-on experience designing and developing enterprise data pipelines
- Advanced experience with Apache Spark
- Strong development experience with Scala
- Experience with ETL development, modernization, and migration initiatives
- Strong understanding of data transformation, data quality, and reconciliation
- Experience comparing and validating data across legacy and modernized systems
- Experience developing and supporting production data-processing workloads
- Familiarity with modern CI/CD and software development practices
- Strong troubleshooting and problem-solving skills
Preferred Qualifications
- Experience migrating legacy ETL platforms to Spark-based architectures
- Experience supporting large-scale technology or data modernization initiatives
- Experience within financial services or other highly regulated environments
- Python experience
- Exposure to Generative AI or AI-assisted development and modernization tools
- Previous experience within a large enterprise engineering environment
#J-18808-Ljbffr