Senior Data Engineer (Scala/Spark)
London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay
We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale data platforms that support advanced machine learning and recommendation systems used by millions of users globally. This is an opportunity to work on highly complex distributed systems, developing production-grade data pipelines that power critical machine learning workflows at scale.
As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets and support the delivery of intelligent, data-driven products.
This role sits at the intersection of data engineering and machine learning infrastructure, with a strong focus on performance, scalability, reliability and operational excellence.
Key Responsibilities
- Design, develop and maintain large-scale Scala and Spark data pipelines.
- Build new data processing capabilities within an established engineering framework.
- Own the performance, reliability and operational health of production data pipelines.
- Develop tooling and infrastructure that supports machine learning workflow execution and deployment.
- Optimise data processing systems for throughput, scalability and efficiency.
- Collaborate closely with Data Engineers, ML Engineers and Research teams to deliver high-quality datasets and platform capabilities.
- Identify and resolve bottlenecks, performance issues and system failures.
- Contribute to engineering best practices, code quality and platform improvements.
About You
You are an experienced data engineer who enjoys building robust, scalable systems and operating them in production environments.
You'll bring:
- Strong commercial experience developing applications and data platforms using Scala.
- Deep expertise with Apache Spark and distributed data processing.
- Proven experience designing, building and supporting production-grade data pipelines.
- Strong understanding of software engineering principles, testing and operational best practices.
- Experience troubleshooting complex performance, reliability and scalability challenges.
- Ability to work effectively within established engineering frameworks and standards.
- Excellent communication and collaboration skills.
Desirable Experience
- Experience working within large-scale data or platform engineering environments.
- Exposure to machine learning infrastructure, pipelines or production ML systems.
- Experience supporting data processing frameworks, orchestration platforms or workflow tooling.
- Familiarity with JVM-based distributed systems.
- Experience working within highly scalable cloud or distributed computing environments.
What You'll Gain
- The opportunity to work on data systems operating at global scale.
- Exposure to cutting-edge machine learning infrastructure and workflows.
- Collaboration with world-class engineers, researchers and technical leaders.
- A highly technical engineering environment focused on quality, scalability and innovation.
This role would suit a Senior Data Engineer, Scala Engineer, Spark Engineer, Data Platform Engineer or Distributed Systems Engineer looking to work on complex engineering challenges within a world-class technology environment.
