Data Engineer Azure Databricks

Position: Data Engineer (Azure & Databricks)
Location: Dublin North, Hybrid (3 days on site, 2 from home)
Contract Type: 12‑month Fixed‑Term Contract

About the Company
A leading provider of innovative data solutions is seeking a highly skilled and motivated Data Engineer with expertise in Microsoft Azure and Databricks. This role is an excellent opportunity for a talented professional to join a growing data analytics team and contribute to the design and delivery of robust, secure and efficient data pipelines.

The Role
As a Data Engineer, you will design and implement robust, secure and efficient data pipelines using Azure and Databricks. You will play a pivotal role in ensuring that data platforms support both real‑time and batch processing, advanced analytics and machine learning. You will work closely with cross‑functional teams - including data scientists, analysts and other engineers - to understand requirements and deliver end‑to‑end data solutions that drive business value.

Key Responsibilities

• Design, develop and implement scalable and resilient data pipelines for both real‑time and batch workloads within Azure and Databricks.

• Implement ELT processes to integrate data from various sources into the central data platform.

• Develop and maintain data quality checks, monitoring and alerting to ensure pipeline health.

• Optimise data workloads within the data platform, focusing on performance, cost efficiency, resilience and security.

• Collaborate with stakeholders to understand data needs and deliver optimal solutions from ingestion to visualisation.

• Ensure data solutions adhere to data governance, privacy and security best practices and regulations.

• Utilise Infrastructure‑as‑Code (IaC) tools such as Terraform for cloud infrastructure provisioning.

• Maintain data integrity, security, governance and compliance across all data solutions.

• Mentor junior data engineers and champion best practices and technical excellence within the team.

• Troubleshoot and resolve data-related issues to maintain data accuracy and performant pipelines.

• Stay current with the latest trends and technologies in cloud computing and data engineering, especially within Azure and Databricks.

Required Qualifications

• 5+ years of experience in data engineering, including 3+ years working within Microsoft Azure.

• Strong experience with Azure services (Data Factory, SQL Database, Storage, Key Vault, Function and Logic Apps, Cost Analysis).

• Hands‑on experience with Databricks, including data engineering, processing and analytics development.

• Strong understanding of performance optimisation, data governance frameworks (e.g., Unity Catalog) and best practices.

• Strong knowledge of ETL/ELT processes, data modelling, data warehousing concepts and Medallion Architecture.

• Experience with real‑time data processing frameworks (e.g., Apache Kafka).

• Proficient in Python programming and PySpark, SQL and working with large‑scale datasets.

• Knowledge of CI/CD and DevOps practices and tools (e.g., Git, Azure DevOps).

• Experience working with Agile delivery methods.

• Understanding of Infrastructure‑as‑Code (IaC) using Terraform (desirable).

• Knowledge of machine learning integration and MLOps (desirable).

Soft Skills

• Excellent problem‑solving and analytical capabilities.

• Strong communication and collaboration abilities across both technical and business stakeholders.

• Skilled at managing multiple priorities in a fast‑paced environment.

• Strong mentoring and team‑building abilities.

• Solid understanding of data governance, security protocols and compliance standards.

• Commitment to team values and collaborative culture.