Senior Real-Time Data Engineer - Finance - Freelance

As part of the development of data products for a demanding international environment, an organisation is looking for a Senior Real-Time Data Engineer.

The consultant will join a team responsible for designing, developing and operating cloud-based data products, from event ingestion through to data delivery via APIs and streaming interfaces. The solutions must meet high standards of reliability, security, performance, observability and maintainability.

The consultant will be responsible for designing, developing and operating real-time and batch data pipelines.

Key responsibilities will include:

• Designing and developing real-time data products using Apache Flink, Apache Kafka and event-driven architectures.

• Building robust and scalable ingestion, transformation and data delivery pipelines.

• Developing REST APIs, OpenAPI specifications and microservices to expose data to consumers.

• Implementing AWS cloud services suited to streaming, processing and deployment requirements.

• Defining data models, schemas, data quality controls and validation rules.

• Industrialising software delivery through automated CI/CD pipelines.

• Implementing comprehensive observability covering logs, metrics, traces, dashboards and alerting.

• Supporting production operations, incident management, root-cause analysis and continuous improvement.

• Applying security best practices relating to authentication, authorisation, secrets management and access control.

• Producing clear architecture documentation, technical specifications, operational procedures and runbooks.

• Contributing to code reviews, engineering standards and knowledge transfer within the team.

The objective is to deliver production-ready real-time data products that meet functional, performance, reliability, security and maintainability requirements.