We are seeking an experienced Lead Data Engineer to join a high-performing data team focused on building advanced analytics solutions to support financial crime detection and prevention.
In this role, you will design and deliver scalable, secure, and high-quality data platforms, enabling stakeholders to generate actionable insights and make data-driven decisions. You will play a key role in shaping modern cloud-based data architecture using Databricks and AWS technologies.
Key Responsibilities
- Lead the end-to-end design, development, and optimisation of scalable data pipelines using Spark / PySpark (batch and streaming) on Databricks
- Define and implement lakehouse architecture (bronze, silver, gold layers), ensuring strong data governance, quality, and lineage
- Build and manage secure data ingestion frameworks (APIs, SFTP/FTPS, Apache NiFi)
- Architect and maintain AWS-based data infrastructure (S3, Glue, IAM, Lake Formation, Lambda, Step Functions, etc.)
- Develop and standardise workflow orchestration using Airflow, Databricks Workflows, and Step Functions
- Champion data quality, observability, and reliability, including SLAs, monitoring, and alerting
- Implement metadata, lineage, and governance frameworks to support audit and compliance requirements
- Drive CI/CD best practices for data pipelines and infrastructure (Terraform/CloudFormation, Git workflows)
- Optimise performance and cost across data platforms (partitioning, caching, storage strategies)
- Collaborate with cross-functional teams including data science, product, and compliance
- Lead technical design reviews, mentor engineers, and promote engineering best practices
- Support incident management, root cause analysis, and continuous improvement initiatives
Key Requirements
Essential Skills & Experience
- 10+ years' experience in data engineering
- Strong expertise in Databricks, Spark, PySpark, Python, and SQL
- Proven experience building large-scale data pipelines in cloud environments
- Deep knowledge of AWS data ecosystem (S3, Glue, IAM, Lake Formation, etc.)
- Hands-on experience with workflow orchestration tools (Airflow, Step Functions)
- Experience implementing data governance, lineage, and security controls
- Strong understanding of CI/CD and infrastructure as code
- Excellent stakeholder management and communication skills
- Experience mentoring or leading engineering teams
