Companies in this catalog do not charge applicants. Never send money to apply.
Position Summary
Scicom Infrastructure Services is seeking aDatabricks Data Operations Specialistto support the operational stability, reliability, performance, and security of a large-scale Databricks data platform.
This role will work as part of a multidisciplinary Databricks team that includes data engineers, architects, business analysts, governance specialists, and technical managers. The Data Operations Specialist will be responsible for monitoring production data pipelines, responding to incidents, troubleshooting platform and data-processing issues, coordinating deployments, and ensuring that data products meet established availability, quality, and service-level requirements.
The ideal candidate will have hands-on experience operating Databricks environments and supporting production data workloads using Apache Spark, Delta Lake, SQL, Python, cloud infrastructure, monitoring tools, and automated deployment practices.
Key Responsibilities
Databricks Platform Operations
Monitor the health, availability, performance, and capacity of Databricks workspaces, jobs, pipelines, clusters, SQL warehouses, and related cloud services.
Provide day-to-day operational support for production Databricks environments.
Monitor scheduled and event-driven data pipelines for failures, delays, data-quality issues, and performance degradation.
Recommend cluster right-sizing, autoscaling, scheduling, caching, partitioning, and workload-isolation improvements.
Support implementation of compute policies, tagging standards, budgets, and cost alerts.
Produce usage and cost reports for technical and program leadership.
Work with engineers and architects to reduce unnecessary compute consumption.
Track the operational impact of optimization initiatives.
Documentation and Continuous Improvement
Create and maintain operational procedures, troubleshooting guides, escalation matrices, and support runbooks.
Document platform configurations, dependencies, schedules, service accounts, integrations, and recovery requirements.
Maintain an accurate inventory of production jobs, pipelines, data products, and system interfaces.
Identify opportunities to automate repetitive monitoring, support, recovery, and reporting activities.
Participate in operational-readiness reviews for new data products and pipelines.
Ensure new solutions include appropriate monitoring, alerts, logging, support procedures, and ownership assignments before production deployment.
Share lessons learned and common troubleshooting practices across the Databricks team.
Contribute to platform standards, operational policies, and continuous-improvement initiatives.
Required Qualifications
Bachelor’s degree in computer science, information technology, data engineering, information systems, or a related field, or equivalent professional experience.
At least four years of experience in data operations, production support, data engineering, cloud operations, DevOps, platform support, or a related technical role.
At least two years of hands-on experience supporting Databricks environments or production Databricks workloads.
Experience supporting production ETL, ELT, batch, or streaming data pipelines.
Working knowledge of:
Databricks
Apache Spark
PySpark or Python
SQL
Delta Lake
Databricks Workflows or Jobs
Cloud object storage
Data pipeline monitoring
Log analysis and troubleshooting
Experience investigating failed jobs, delayed pipelines, schema issues, data discrepancies, and performance problems.
Familiarity with medallion architecture and Bronze, Silver, and Gold data layers.
Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud.
Experience using ticketing and service-management tools such as ServiceNow, Jira, or a comparable platform.
Experience with monitoring, alerting, logging, or observability tools.
Understanding of incident, problem, change, and release-management processes.
Ability to write and maintain technical procedures and operational documentation.
Strong analytical, organizational, communication, and troubleshooting skills.
Ability to work effectively with engineers, business analysts, architects, security teams, and nontechnical stakeholders.
Ability to support multiple production priorities in a structured and timely manner.
Preferred Qualifications
Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.
Databricks Certified Associate Developer for Apache Spark.
Microsoft Azure, AWS, or Google Cloud certification.
Experience with Unity Catalog, Delta Live Tables, Lakeflow, Auto Loader, Structured Streaming, or Databricks SQL.
Experience with Azure Data Factory, Azure Data Lake Storage, Azure Monitor, Log Analytics, AWS Glue, Amazon S3, CloudWatch, or comparable cloud services.
Experience with Terraform, Git, Azure DevOps, GitHub Actions, Jenkins, or other CI/CD tools.
Experience with Splunk, Datadog, Grafana, Prometheus, or similar observability platforms.
Experience supporting APIs, Kafka, Event Hubs, Kinesis, Airflow, dbt, Snowflake, or Power BI.
Knowledge of data contracts, schema registries, metadata management, data lineage, and data-quality frameworks.
Familiarity with OCDS, procurement data, contracting data, financial data, or public-sector data standards.
Experience supporting government, regulated, or high-security environments.
Experience operating systems with formal service-level agreements and 24-hour support requirements.
Experience with disaster recovery, backup, archival, and operational-resilience testing.
Core Competencies
Strong operational ownership and attention to detail.
Ability to remain calm and organized during production incidents.
Clear written and verbal communication.
Methodical troubleshooting and root-cause analysis.
Ability to distinguish symptoms from underlying technical causes.
Strong documentation and process-discipline skills.
Ability to prioritize incidents based on business and technical impact.
Collaborative approach to working with engineering and business teams.
Commitment to automation, reliability, security, and continuous improvement.
Willingness to take ownership of issues through resolution.
