Golden Dataset Pipeline — Surya Ramessh

Pipeline flowchart: extract from a synthetic case management database under a read-only login, source checks with pandera, pseudonymisation with HMAC tokens and date shifting, output checks and PII scan, deterministic train and eval split, versioned Parquet output with manifest. Re-identification keys are written to a restricted store and the salt comes from an environment variable or Key Vault.

the pipeline end to end · either check failing stops the run and publishes nothing

The problem

Plenty of organisations have a "golden dataset" that started as someone's careful manual exercise: a spreadsheet of completed cases, assembled by hand, correct at the time and never quite reproducible afterwards. It works right up until someone asks the two questions that matter. Can you rebuild it from source? And can you prove the AI you are building on it is getting better rather than worse?

Those questions are harder than they sound, because the honest answer usually involves personal data, a production system nobody wants queried heavily, and no record of which version of the data a given model was trained on.

So I built a small pipeline that takes a manual dataset of completed legal matters and turns it into something versioned, pseudonymised, quality-checked and repeatable, with the evaluation data needed to measure whether a model improves. Everything in it is synthetic.

1 command

rebuilds the dataset from source

2 gates

source and output checks, fail loudly

0 rows

published if either gate fails

100%

synthetic data, generated not collected

How it works

Extract under least privilegeA read-only login with SELECT on a single view, pulling incrementally by a completion-date watermark in small batches. The point is to put no meaningful load on the source system and to hold no more access than the job needs.
Check the source before trusting itpandera schema checks on what came out: types, ranges, valid dates, nulls. If the source has drifted, the run stops here rather than quietly producing a subtly wrong dataset.
PseudonymiseHMAC tokens for identifiers, postcodes truncated to district, dates shifted, and free-text notes scrubbed of personal detail. Re-identification keys are written to a separate restricted store, never into the output. The salt comes from an environment variable locally, and would come from Key Vault on Azure.
Check the output before publishing itSchema validation plus a PII scan over the result. This is the gate that matters most: it is the one that catches a free-text field that slipped through the scrub. Failing here also stops the run.
Split deterministicallyA reproducible train and evaluation split stratified by matter type, plus a small evaluation question-and-answer set built from the case notes. That set is the thing that lets someone say whether a model got better, instead of guessing.
Version the resultEach run writes a numbered snapshot: Parquet tables, the evaluation set, and a manifest recording row counts, schema, content hash and run time. A model trained last month can be traced back to the exact data it saw.

The design decision worth calling out is that both quality gates fail closed. A pipeline that degrades quietly is worse than one that stops, because a silently wrong golden dataset propagates into every model trained on it and nobody finds out until much later.

What it demonstrates

Least-privilege, low-impact extraction from a live source
Pseudonymisation with keys held separately from data
Quality gates that fail closed, at input and output
Versioned snapshots with a manifest for lineage
Evaluation data as a first-class output, not an afterthought
One command to rebuild, with tests running in CI

How it maps to a cloud platform

Orchestration into Data Factory or Fabric pipelines
Snapshots onto ADLS with RBAC per container
Salt and connection secrets into Key Vault
Managed Identity in place of stored credentials
Manifests feeding a catalogue for lineage
GitHub Actions promoting versions between environments

What it is not

This is a demonstration of an approach, not a production system, and it is deliberately small enough to read in one sitting. The source is a generated conveyancing dataset standing in for a case management system; a real deployment would read from whatever the firm actually runs, over ODBC or a replica rather than the live database. The PII scrubbing is regex-based and would want a proper detection library behind it. There is no retention or deletion handling, which a real dataset built from client records would need from day one.

I built it with Claude Code, which is worth saying plainly rather than leaving someone to wonder.

On the data: every record in this project is synthetic, generated with Faker for the purpose of the demo. No client, employer or case data of any kind was used, and the repository contains no real personal data.