Data Engineer (Databricks)

The Data Engineer will be responsible for designing, developing, and maintaining scalable and reliable data pipelines on Databricks and cloud platforms. The role requires integrating diverse data sources, ensuring high-quality data processing, and supporting analytics, reporting, and machine learning workloads. The role involves collaborating closely with analytics, product, and infrastructure teams to enhance the company’s data platform while adhering to best practices for governance, monitoring, and reliability.

What will you do?

Develop and maintain ETL pipelines for centralized data storage systems (e.g. Delta Lake).

Integrate data from databases, APIs, log files, streaming platforms, and external providers Develop data transformation routines to clean, normalize, and aggregate data Apply data processing techniques to handle complex or inconsistent datasets Contribute to frameworks and best practices for code development and deployment Implement data governance in alignment with company standards Partner with analytics and product leaders to design and operationalize pipelines Collaborate with infrastructure leaders to advance cloud-based data platforms Explore new tools and techniques leveraging Azure, Databricks, or related platforms Monitor data pipelines to detect and resolve issues promptly Develop monitoring tools, alerts, and automated error-handling mechanisms Analyse business requirements and identify data extraction requirements Attend requirement grooming and refinement sessions with users Develop and maintain ETL pipelines for ingestion, transformation, validation, and loading Optimise performance and batch scheduling Develop dashboards, reports, scorecards, and data visualizations Perform SIT, data validation, data profiling and confirm data accuracy Validate completeness and consistency of ETL Loads Support UAT and production implementation