Databricks Platform Engineer

Job summary

Con*** is seeking a Databricks Platform Engineer to build, automate, and operate the foundational services that make our enterprise Databricks platform secure, scalable, governed, and cost-efficient.

Qualifications

• The ideal candidate has strong software engineering fundamentals, hands-on Databricks experience, and a passion for building highly automated, production-grade cloud data platforms.
• 5+ years of experience in Data Engineering, Platform Engineering, Cloud Engineering, or a related field.
• 2+ years of hands-on experience supporting Databricks in production environments.
• Strong proficiency in Python and SQL with solid software engineering practices including testing, version control, and code reviews.
• Experience with Databricks Asset Bundles or equivalent deployment tooling.
• Hands-on experience with CI/CD platforms such as GitHub Actions or Azure DevOps.
• Experience building metadata-driven or configuration-based data pipelines using Spark and Delta Lake.
• Working knowledge of Unity Catalog, including governance, permissions, lineage, tagging, and policy automation.
• Experience implementing monitoring, observability, alerting, and production support for cloud data platforms.
• Experience with Infrastructure-as-Code, preferably Terraform.
• Experience with one or more major cloud platforms, including Azure, AWS, or Google Cloud Platform.
• Experience optimizing Databricks workloads, managing DBU consumption, and implementing cloud cost optimization strategies.

Responsibilities

• Design, build, and maintain governance automation for Unity Catalog, including access controls, metadata tagging, policy enforcement, and governance-as-code.
• Develop metadata-driven frameworks for scalable data ingestion, transformation, and publishing using Spark and Delta Lake.
• Build reusable platform services, libraries, templates, and integrations that accelerate development and promote engineering best practices.
• Develop and maintain automated data quality pipelines, validation frameworks, quality scorecards, and issue escalation workflows.
• Own the Databricks CI/CD framework, including repository standards, automated testing, Databricks Asset Bundles, environment promotion, and release automation.
• Implement Infrastructure-as-Code using Terraform and other modern DevOps practices to ensure repeatable, auditable deployments.
• Build and maintain monitoring, logging, alerting, and observability solutions for Databricks workloads and platform operations.
• Manage platform reliability by supporting incident response, operational runbooks, and continuous service improvements.
• Monitor Databricks usage and cloud costs, implementing optimization strategies to improve DBU efficiency and reduce operational expenses.
• Support workspace provisioning, identity integration, and access management while automating provisioning workflows where possible.
• Partner with architects, data engineers, and platform stakeholders to continuously enhance platform capabilities and developer experience.

Certifications

Degrees

Industry

Company size