LLMOps & Observability Engineer - Freelance

Our Cutomer is establishing a dedicated AI team to bring structured AI engineering capabilities to its software delivery organization. The team is divided into two functions: a Platform team that builds shared infrastructure, and an Enablement team that collaborates with engineering and product teams to ship AI use cases and transfer the capability to operate independently. The goal is to achieve measurable AI capability across the SDLC within six months.

This role is part of the Platform team and is responsible for the infrastructure that makes AI workloads measurable, including tracing, evaluation pipelines, quality alerting, test data management, and prompt versioning.

Missions

• Own observability infrastructure: distributed tracing of LLM calls, latency, cost, and quality signals. Assess current tooling such as Langfuse and decide whether to continue using it or switch to alternatives like Langsmith.

• Develop and maintain evaluation pipeline infrastructure that enables teams to define, run, and version their evaluation suites continuously.

• Manage test data: generation, curation, versioning, and access control for evaluation datasets.

• Design and build prompt and model versioning systems, as none currently exist.

• Implement alerting mechanisms for production quality degradation.

• Develop model registry patterns as the portfolio of use cases expands.

Tools & Environment

• Observability tooling including Langfuse, Langsmith, Braintrust, and others.

• TypeScript-first technology stack (Python is not in scope).

• AWS infrastructure for deployment and operation of the observability stack.

Working Conditions

• Role within a newly formed AI Platform team.

• Focus on building scalable, production-grade AI observability and evaluation infrastructure.

• Goal-oriented with measurable success criteria within six months.