Databricks Data Operations Specialist

Job summary

Position Summary Sci*** is seeking a Databricks Data Operations Specialist to support the operational stability, reliability, performance, and security of a large-scale Databricks data platform. This role will work as part of a multidisciplinary Databricks team that includes data engineers, architects, business analysts, governance specialists, and technical managers.

Qualifications

• The ideal candidate will have hands-on experience operating Databricks environments and supporting production data workloads using Apache Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring
• Detect schema drift, unexpected source-system changes, and contract violations.
• Coordinate resolution of data-contract issues with business analysts, data owners, and engineering teams.
• Execute automated and manual data-quality controls.
• Monitor data-quality rules related to accuracy, completeness, uniqueness, consistency, validity, and timeliness.
• Track exceptions and ensure data-quality issues are assigned to the appropriate owner.
• Support reconciliation of contracting, procurement, financial, and operational datasets.
• Assist with validation of data structured according to the Open Contracting Data Standard when applicable.
• Maintain operational evidence supporting data quality, lineage, audit, and governance
• Deployment and Change Management
• Support the release and deployment of Databricks notebooks, jobs, workflows, libraries, configurations, and infrastructure changes.
• Coordinate deployments across development, testing, staging, and production environments.

Responsibilities

• Databricks Platform Operations
• Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services.
• Provide day-to-day operational support for production Databricks environments.
• Monitor scheduled and event-driven data pipelines for failures, delays, data-quality issues, and performance degradation.
• Troubleshoot Databricks jobs, Spark workloads, cluster failures, notebook errors, connectivity issues, and data-processing exceptions.
• Restart, rerun, recover, or coordinate remediation of failed workflows according to documented procedures.
• Support Databricks Workflows, Lakeflow, Delta Live Tables, Auto Loader, Structured Streaming, and batch-processing workloads.
• Review driver and executor logs, Spark UI metrics, cluster event logs, audit logs, and application telemetry to identify root causes.
• Support configuration and maintenance of Databricks clusters, serverless compute, instance pools, cluster policies, and SQL warehouses.
• Monitor resource utilization and recommend changes to cluster sizing, autoscaling, workload scheduling, and compute policies.
• Help maintain consistent configurations across development, testing, staging, and production environments.
• Support platform upgrades, runtime-version changes, library updates, and configuration changes.

Skills

Certifications

Degrees

Languages

Industry

Company size

Security clearance