Azure Data Engineer

Job summary

Data Engineer (Databricks) Azure Databricks Environment Austin, TX - Hybrid General Information Description: The Databricks Data and AI Agent Automation Engineer is responsible for designing, developing, and deploying intelligent agents that automate data engineering development (dev) and quality assurance (QA) tasks within the Azure Databricks environment.

Qualifications

• Education
• Bachelor's degree in Computer Science, Software Engineering, Data Science, or related field (Master's preferred)
• Experience
• 5+ years of data engineering or software engineering experience, with hands-on work in cloud data platforms (Databricks, Snowflake, Azure)
• Proven experience developing automation solutions, agents, or AI-powered

Responsibilities

• Design and implement intelligent agents (LLM-powered, rule-based, or hybrid) that automate data engineering workflows in Azure Databricks
• Develop agents for automated code generation, refactoring, and optimization of PySpark and SQL workloads
• Build agents that automate data quality validation, testing, and reconciliation tasks in dev and QA environments
• Create agents for metadata management, lineage tracking, and governance automation within Databricks
• Develop autonomous CI/CD pipeline tasks, including automated test case generation, deployment validation, and rollback logic
• Implement agents that perform intelligent monitoring, anomaly detection, and root-cause analysis for data pipelines
• Build prompt engineering frameworks and agent orchestration patterns tailored to data engineering use cases
• Collaborate with data engineers and architects to identify high-impact automation opportunities and prioritize agent development roadmap
• Establish best practices for agent development, testing, deployment, and observability within Azure Databricks
• Document agent workflows, decision logic, and operational runbooks for team knowledge sharing
• Evaluate emerging AI/agent frameworks and

Skills

Certifications

Degrees

Industry

Company size