Sr. Data Engineer

Job summary

Data Engineer is needed to perform the following duties: Build streamline pipelines leveraging various AWS services including Lambda and Glue for data processing, ingestion, transformation, feature engineering, and business analysis.

Responsibilities

• Build streamline pipelines leveraging various AWS services including Lambda and Glue for data processing, ingestion, transformation, feature engineering, and business analysis.
• Leverage AWS Event Bridge, EMR and Step Functions to create a streamline data pipeline to invoke Lambda and Glue jobs with batch based and event-based triggers, finally ingesting transformed data into Snowflake.
• Develop custom PySpark and Scala coding to perform data transformations such as filtering, cleansing, mapping, handling null values and improving data quality.
• Improve efficiency by optimizing complex SQL queries in Snowflake, implementing data partitioning, and indexing strategies for parallel processing, and reduced pipeline complexity.
• Leverage python and its multiple packages to create functions to create derived variables from existing variables, performed end-to-end testing, unit testing, integration tests, and exceeded the required sonar coverage.
• Bachelor's Degree is required in Computer Science or Computer Engineering or Information Systems or Business Analytics.
• JOB TAGS
• Senior data engineer, sr.
• data engineer
• Job Details
• Job Ref Code:
• APM-116

Skills

Degrees