We are seeking a highly motivated Data Engineer (ETL) to design, develop, and maintain scalable data engineering solutions supporting enterprise data aggregation and integration initiatives. This role focuses on building high-performance ETL pipelines, developing Apache Spark applications, processing large-scale datasets, and leveraging cloud-native technologies to deliver reliable and efficient data solutions. The ideal candidate will have strong expertise in Spark, Scala, Java, cloud platforms, and big data technologies while collaborating with cross-functional teams in an Agile environment.
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines for enterprise data integration and movement.
- Develop, test, deploy, and support ETL batch jobs using Apache Spark and Amazon EMR.
- Build and maintain Spark services for large-scale data processing.
- Schedule, monitor, troubleshoot, and optimize Spark batch processing jobs.
- Develop Java clients for consuming REST and SOAP APIs.
- Develop and maintain Scala-based Spark batch applications.
- Process and transform large datasets using Apache Spark.
- Store and manage processed data in NoSQL databases such as Cassandra or Yugabyte.
- Design and implement cloud-based data warehouse and data lake solutions using Snowflake.
- Work with large-scale data storage formats including Parquet and HDF5.
- Collaborate with cross-functional teams to design innovative data solutions that address business requirements.
- Participate in code reviews, pair programming, and technical discussions to ensure code quality and best practices.
- Support the full Software Development Life Cycle (SDLC), including analysis, design, development, testing, deployment, and maintenance.
- Troubleshoot production issues and optimize data processing performance.
- Identify opportunities for innovation and continuous improvement in data engineering practices.
- Collaborate across multiple Agile teams to deliver scalable and reliable data solutions.
- Apply a customer-first mindset when designing and delivering enterprise data solutions.
Required Qualifications
- Bachelor's degree in Computer Science, Information Systems, or a related field.
- 10+ years of experience developing Spark services for large-scale data movement.
- Experience scheduling, monitoring, debugging, and optimizing ETL Spark batch processes.
- Hands-on experience developing Java clients for REST and SOAP API integrations.
- Experience developing Scala Spark batch applications.
- Experience developing, deploying, and maintaining Apache Spark and Amazon EMR ETL jobs.
- Experience processing big data using Apache Spark.
- Experience with NoSQL databases such as Cassandra or Yugabyte.
- Experience with cloud-based data warehouse and data lake platforms such as Snowflake.
- Experience managing large datasets using storage formats such as Parquet and HDF5.
- Strong understanding of software development lifecycle (SDLC) methodologies.
- Strong analytical, troubleshooting, and problem-solving skills.
- Experience working in Agile software development environments.
- Excellent collaboration and communication skills.
Preferred Qualifications
- Experience designing enterprise-scale data integration and aggregation solutions.
- Experience supporting external-facing data platforms and APIs.
- Experience working with financial services or FinTech data platforms.
- Passion for continuous learning and adopting modern data engineering best practices.
#J-18808-Ljbffr