Senior Real-Time Data Engineer - Finance - Freelance

Context

As part of the development of data products for a demanding international environment, an organisation is looking for a Senior Real-Time Data Engineer.

The consultant will join a team responsible for designing, developing, and operating cloud-based data products, from event ingestion through to data delivery via APIs and streaming interfaces. The solutions must meet high standards of reliability, security, performance, observability, and maintainability.

Missions

• Design and develop real-time data products using Apache Flink, Apache Kafka, and event-driven architectures.
• Build robust and scalable ingestion, transformation, and data delivery pipelines.
• Develop REST APIs, OpenAPI specifications, and microservices to expose data to consumers.
• Implement AWS cloud services suited to streaming, processing, and deployment requirements.
• Define data models, schemas, data quality controls, and validation rules.
• Industrialise software delivery through automated CI/CD pipelines.
• Implement comprehensive observability covering logs, metrics, traces, dashboards, and alerting.
• Support production operations, incident management, root-cause analysis, and continuous improvement.
• Apply security best practices relating to authentication, authorisation, secrets management, and access control.
• Produce clear architecture documentation, technical specifications, operational procedures, and runbooks.
• Contribute to code reviews, engineering standards, and knowledge transfer within the team.

The objective is to deliver production-ready real-time data products that meet functional, performance, reliability, security, and maintainability requirements.

Tools & Environment

• Apache Flink, Apache Kafka
• Event-driven architectures
• REST APIs, OpenAPI specifications, microservices
• AWS services: MSK, Kinesis, Lambda, API Gateway, ECS/EKS, S3, IAM
• CI/CD tools: GitLab CI, GitHub Actions or equivalent
• Observability tools: CloudWatch, Prometheus, Grafana, OpenTelemetry

Working Conditions

• Reference: RFP-272706-1
• Senior-level consultancy role in an international finance environment