Senior Python Data Engineer - Data Lineage & Fabric - Finance - CDI/Freelance - Freelance

Context

A financial institution in Luxembourg is building its data lineage and metadata capabilities on top of Microsoft Fabric. As a Senior Python Data Engineer, you will design and build the software that harvests metadata from the data platform, computes table- and column-level lineage, exposes it through APIs and makes it visible through graph visualisations, supporting the institution's data governance operating model. You will turn loosely defined requirements into working, tested software.

Missions

• Design and develop reusable Python packages to harvest metadata and lineage from Microsoft Fabric (Spark workloads, Data Pipelines, Lakehouses, Warehouses, Semantic Models)
• Compute and store table-level and column-level lineage; enable impact analysis
• Design and consume REST APIs; work with relational databases
• Build graph / DAG visualisations of lineage (D3.js, Cytoscape.js, React Flow or similar)
• Integrate lineage standards and tools (OpenLineage events, custom facets, Marquez or similar)
• Write automated tests and clear technical documentation; contribute to a scalable, maintainable lineage solution

Environment

• Python (packaging, automated testing), REST APIs, SQL / relational databases
• Microsoft Fabric: Spark, Data Pipelines, Lakehouses, Warehouses, Semantic Models
• Azure data services (Synapse, Databricks, Purview): a plus
• OpenLineage, Marquez, OpenMetadata, DataHub, Apache Atlas: a plus
• dbt, Airflow, Spark as lineage sources: a plus
• React, D3.js, Cytoscape.js, React Flow
• Data governance frameworks: DAMA-DMBOK, DCAM

Working conditions

• Contract: Freelance, CDI or CDD: we will find a solution as long as you are the right candidate
• Location: Luxembourg, on site
• We are looking for people ready to relocate and work in Luxembourg
• Salary / daily rate: depending on profile
• Languages: English and French