Building a KYC Document Pipeline Where No Unmasked Aadhaar Survives

Masking a document is one API call. Building a KYC pipeline where no unmasked Aadhaar number survives is a different problem, and it is mostly about the places documents pass through rather than the place they land.

Teams usually get the masking step right on the first try. What takes longer to get right is everything either side of it: where the document arrives, what touches it in flight, and what is left behind afterwards. This is a walk through a pipeline that holds up, and the specific places the ones that don't tend to fail.

The shape of it

Five stages, and the interesting decisions are at the boundaries.

upload → validate → mask → store → purge
   |          |          |        |        |
   |          |          |        |        └─ originals gone, everywhere
   |          |          |        └─ masked copy is the record
   |          |          └─ zero-count documents diverted to review
   |          └─ rejected locally, before spending a call
   └─ never written to disk unencrypted

Stage 1: upload

The first decision is whether the unmasked document ever touches your filesystem. If you can hold it in memory from upload through masking, do — it removes an entire class of problem around temp directories, backup snapshots and cleanup that did not run.

# Express: memoryStorage, so the unmasked document never reaches disk.
const upload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 50 * 1024 * 1024 },
});

Where the file is genuinely too large to hold in memory, or the flow is asynchronous, write it to encrypted storage with a short lifecycle rule — and make the rule the mechanism rather than a cleanup job, because a lifecycle rule cannot forget to run.

Stage 2: validate before you call

Check the content type and size locally. An unsupported file costs a round trip to be told 422, and rejecting it yourself gives the user a faster and more specific message.

More usefully, this is where to catch the document that is not what the user thinks it is. A screenshot of an e-Aadhaar, a PDF containing only a QR code, a photo so dark nothing is legible — these will come back with nothing masked, and a size or dimension check catches some of them before they consume anything.

Stage 3: mask, and read the header

The call itself is the easy part. The branch after it is the whole point of the pipeline.

const res = await fetch(`${API}/mask-aadhaar`, {
  method: "POST",
  headers: { "X-API-Key": process.env.MASKAADHAAR_KEY },
  body: form,
  signal: AbortSignal.timeout(60_000),
});

const regions = Number(res.headers.get("x-masked-count") ?? 0);

if (regions === 0) {
  // 200 OK, and the body is the ORIGINAL document, unchanged. Filing this as
  // masked is how an unmasked Aadhaar number ends up in a folder marked done.
  await queueForReview(docId, "no_number_detected");
  return;
}

Three outcomes, not two: masked, not masked, failed. The middle one looks exactly like success — same status, same content type, a real document in the body — and it is the outcome that leaves live identity numbers in your store if you collapse it into either neighbour.

Stage 4: store the masked copy as the record

The masked document is what your application reads from now on. Everything downstream — the reviewer UI, the partner export, the regulatory pack — points at it. If any downstream consumer still references the original, the pipeline has not actually changed anything.

Stage 5: purge, and mean it

Delete the original in the same operation that stores the masked copy, not on a schedule. A cleanup job is a thing that can be disabled, silently fail, or be scoped to the wrong bucket. An inline delete either happens or the whole operation is retried.

await store.putMasked(docId, masked);
// Same operation, deliberately. A masked copy stored beside an unmasked original
// has reduced nothing — it has added a file.
await store.deleteOriginal(docId);

The five places documents leak anyway

These are the ones that survive an otherwise correct implementation, because none of them are in the diagram.

Logs

The most common by a wide margin. A debug log that records request bodies, or an error handler that dumps the full request on failure, writes unmasked Aadhaar numbers into your log aggregator — a system with broader access and longer retention than the document store you just spent a sprint protecting. Log the document id and the outcome. Never the bytes, and never the API key.

Error tracking

Sentry, Rollbar and similar tools attach request context to exceptions by default. If an exception fires while a document is in scope, the payload can go with it. Scrub the file field explicitly rather than assuming the default configuration is safe.

Backups and snapshots

Deleting the original from the live bucket does not remove it from last night's snapshot. If your retention policy says thirty days, unmasked documents exist for thirty days after you stopped holding them. That is often acceptable — it just needs to be a decision rather than a discovery.

The message queue

If the document body travels through a queue between upload and masking, it is at rest in that queue's storage, with that queue's retention and access rules. Passing a reference rather than the bytes avoids the problem entirely.

The review queue you built in stage 3

Worth stating plainly, because it is a consequence of doing the right thing. Documents that could not be masked are, by definition, unmasked documents — and you have now collected them all in one place. That queue needs tighter access control than the masked store, a genuine workflow to empty it, and someone whose job it is to. A review queue nobody works is just a folder of live Aadhaar numbers with a reassuring name.

Failure handling worth having

Two rules cover most of it.

Retry only what is transient. 429 rate_limited and 500 processing_failed are worth retrying with exponential back-off. An invalid key, an oversized file, an unsupported type and an exhausted quota will return the same answer however many times you ask. Note that rate_limited and quota_exceeded share a status code and mean opposite things about retrying, so branch on the JSON error field.

Never retry a zero count. It is not a failure of the service, it is a statement about the document. Retrying spends quota to receive the same answer.

What to monitor

Four numbers tell you whether the pipeline is healthy, and only one of them is about errors:

• Zero-count rate. Your review rate. A sudden rise usually means an upstream change — a new scanner, a new mobile upload path, a compression setting.
• Review queue depth. If it only grows, the workflow does not exist.
• Quota remaining. Read X-Quota-Remaining off each response into your metrics. It is the cheapest possible early warning.
• Originals outstanding. A count of documents where a masked copy exists and the original has not been deleted. This should be zero, and alerting when it is not catches the purge step breaking long before an audit does.

That last metric is the one most teams do not have, and it is the one that would have caught most of the incidents described above.

Questions

How should Aadhaar masking fit into a KYC pipeline?

Five stages: accept the upload without writing it to disk unencrypted, validate the type and size locally before spending a call, mask and branch on the X-Masked-Count header, store the masked copy as the record every downstream consumer reads, and delete the original in the same operation that stores the masked copy rather than on a cleanup schedule.

What should happen to documents where no Aadhaar number is detected?

They go to a review queue, never to the masked store. A 200 response with X-Masked-Count of zero returns the original document unchanged, so it looks identical to success. That queue then needs tighter access control than the masked store and a workflow that actually empties it — a review queue nobody works is a folder of live Aadhaar numbers with a reassuring name.

Where do KYC pipelines leak unmasked documents?

Five places that are never in the architecture diagram: application logs that record request bodies, error trackers that attach request context to exceptions, backup snapshots taken before the original was deleted, message queues carrying document bytes rather than references, and the review queue of unmaskable documents itself.

Should I keep the original document after masking?

Only where a law or regulator requires it, and then under stricter access than the masked copy. Storing a masked copy beside the unmasked original reduces exposure by nothing — it adds a file. Delete the original in the same operation that stores the masked copy, because a cleanup job can be disabled, silently fail, or be scoped to the wrong bucket.

What should I monitor in a masking pipeline?

Four numbers: the zero-count rate, which is your review rate and rises when something upstream changes; review queue depth, which reveals whether the workflow exists; quota remaining, read from the X-Quota-Remaining header into your metrics; and originals outstanding — documents with a masked copy whose original has not been deleted. That last one should be zero, and alerting on it catches a broken purge step long before an audit does.