Position: Data Engineer (Azure & Databricks)
Location: Dublin North, Hybrid (3 days on site, 2 from home)
Contract Type: 12‑month Fixed‑Term Contract
About the Company
A leading provider of innovative data solutions is seeking a highly skilled and motivated Data Engineer with expertise in Microsoft Azure and Databricks. This role is an excellent opportunity for a talented professional to join a growing data analytics team and contribute to the design and delivery of robust, secure and efficient data pipelines.
The Role
As a Data Engineer, you will design and implement robust, secure and efficient data pipelines using Azure and Databricks. You will play a pivotal role in ensuring that data platforms support both real‑time and batch processing, advanced analytics and machine learning. You will work closely with cross‑functional teams - including data scientists, analysts and other engineers - to understand requirements and deliver end‑to‑end data solutions that drive business value.
Key Responsibilities
-
Design, develop and implement scalable and resilient data pipelines for both real‑time and batch workloads within Azure and Databricks.
-
Implement ELT processes to integrate data from various sources into the central data platform.
-
Develop and maintain data quality checks, monitoring and alerting to ensure pipeline health.
-
Optimise data workloads within the data platform, focusing on performance, cost efficiency, resilience and security.
-
Collaborate with stakeholders to understand data needs and deliver optimal solutions from ingestion to visualisation.
-
Ensure data solutions adhere to data governance, privacy and security best practices and regulations.
-
Utilise Infrastructure‑as‑Code (IaC) tools such as Terraform for cloud infrastructure provisioning.
-
Maintain data integrity, security, governance and compliance across all data solutions.
-
Mentor junior data engineers and champion best practices and technical excellence within the team.
-
Troubleshoot and resolve data-related issues to maintain data accuracy and performant pipelines.
-
Stay current with the latest trends and technologies in cloud computing and data engineering, especially within Azure and Databricks.
Required Qualifications
-
5+ years of experience in data engineering, including 3+ years working within Microsoft Azure.
-
Strong experience with Azure services (Data Factory, SQL Database, Storage, Key Vault, Function and Logic Apps, Cost Analysis).
-
Hands‑on experience with Databricks, including data engineering, processing and analytics development.
-
Strong understanding of performance optimisation, data governance frameworks (e.g., Unity Catalog) and best practices.
-
Strong knowledge of ETL/ELT processes, data modelling, data warehousing concepts and Medallion Architecture.
-
Experience with real‑time data processing frameworks (e.g., Apache Kafka).
-
Proficient in Python programming and PySpark, SQL and working with large‑scale datasets.
-
Knowledge of CI/CD and DevOps practices and tools (e.g., Git, Azure DevOps).
-
Experience working with Agile delivery methods.
-
Understanding of Infrastructure‑as‑Code (IaC) using Terraform (desirable).
-
Knowledge of machine learning integration and MLOps (desirable).
Soft Skills
-
Excellent problem‑solving and analytical capabilities.
-
Strong communication and collaboration abilities across both technical and business stakeholders.
-
Skilled at managing multiple priorities in a fast‑paced environment.
-
Strong mentoring and team‑building abilities.
-
Solid understanding of data governance, security protocols and compliance standards.
-
Commitment to team values and collaborative culture.
