Senior Data Engineer

Job summary

Overview: Summary The candidate must have a successful track record in handling complex data pipelines for high transaction volume application and reporting/analytics data with minimal supervision. The candidate will be expected to support a variety of structured and semi-structured data in streaming and batch frameworks for both migration from and synchronization with legacy systems.

Qualifications

• The candidate must have a successful track record in handling complex data pipelines for high transaction volume application and reporting/analytics data with minimal supervision.
• and Education:
• Required
• BA or BS degree in CS, Computer Engineering, Information Technology, or a related field.
• Databricks Certified Data Engineer (Associate or Professional) or equivalent experience.
• 5+ years of hands-on experience with PySpark, Python, and various Python toolkits and libraries for data processing and pipelines.
• Proven experience and deep understanding of the Medallion architecture, with the ability to define and adhere to best practices diligently.
• Hands-on experience orchestrating complex data workflows using Apache Airflow.
• 5+ years of experience and ability to create complex SQL queries and functions.
• 3+ years of work experience on AWS-based datasources such as RDS, Athena, Redshift.
• 2+ years of hands-on experience in creating ETL and data pipeline design patterns for modular and reliable ETL with robust error management and alerting.
• 1+ year of experience in Java and Linux scripting is essential.

Responsibilities

• What You'll Be Doing:
• Designs, implements, and operates data management systems using Databricks and PySpark.
• Defines and adheres to best practices for implementing the Medallion architecture diligently across all data layers.
• Orchestrates complex data pipelines and workflows using Apache Airflow.
• Works with product and application owners, data stewards, and business/data analysts to determine, create, and populate required data in an optimal, reliable, and measurable manner.
• Assists the Data Architects and technical leads in designing how data will be stored, accessed, used, integrated, and managed by different data regimes and digital systems.
• Plans, designs, and optimizes data throughput and query performance.
• Participates in the employment of backend database
• a candidate’s work experience, location, education/training, and key

Certifications

Degrees

Industry

Company size