Ten thousand documents collected before anyone thought about masking. A nightly KYC batch. An archive a regulator has asked about. This is how to run that volume through masking without melting your rate limit, and what the arithmetic actually looks like.
Most organisations get masking right on the day they turn it on. New documents arrive, they go through the API, they land redacted. The problem is everything that arrived before that day: the S3 bucket, the shared drive, the email attachments folder, the scanned box files from the branch network. That archive is usually larger than the daily flow by two or three orders of magnitude, and it is the part an auditor asks about.
A backlog is also where masking most often goes quietly wrong, for a reason that has nothing to do with the API. Live traffic is watched. A migration script run once, over a weekend, by one engineer, with the output going to a bucket nobody opens again, is not. If it writes a hundred documents unmasked, that fact can sit undiscovered for years.
The number that matters is not how many files you processed.
It is how many came back with X-Masked-Count: 0 — processed successfully,
returned unchanged, still carrying a readable Aadhaar number. A bulk run that reports "10,000
documents masked" without separating those out has not told you anything useful.
Throughput is bounded by your plan's requests-per-minute, not by your hardware. The work
happens on our side, so adding local threads past that ceiling produces
429 rate_limited responses rather than documents.
| Plan | Requests / minute | Documents / hour | 10,000 documents |
|---|---|---|---|
| Starter | 10 | 600 | ~16.7 hours |
| Growth | 30 | 1,800 | ~5.6 hours |
| Enterprise | Custom | Custom | Sized to the deadline |
Two adjustments to that table before you plan against it. Images are quick; a twelve-page scanned PDF is not, and a batch that is mostly PDFs will run below the theoretical rate because each request occupies its slot for longer. And you need headroom: running at exactly your limit means every transient blip becomes a retry, and retries consume slots that would otherwise carry new documents. Plan for about 80% of the ceiling.
The monthly quota is a separate limit from the per-minute rate, and it is the one that actually stops a backlog. Ten thousand documents against a 1,000-document monthly quota is not a scheduling problem, it is a provisioning conversation — which is why bulk runs are priced by volume and the rate limit can be lifted for a migration window.
Three rules, and the second one is where most bulk scripts go wrong.
Retry rate_limited, not quota_exceeded. Both
return 429. The first clears in seconds. The second does not clear until the
month resets, so a loop that cannot tell them apart will spin for hours and then give up
having achieved nothing. Read the JSON error field, never the status code alone.
Back off the whole pool, not the one request. If you are rate limited, so is every other worker — the limit is per key, not per connection. A script where each worker independently sleeps and retries produces a thundering herd that re-hits the limit the moment the window opens. One shared cooldown that every worker observes is both simpler and faster in wall-clock terms.
Add jitter. Without it, workers that were blocked together retry together. A random fraction added to each delay spreads them out and materially improves throughput at the ceiling.
# A shared cooldown, so a 429 pauses the whole pool rather than one worker. import random, threading, time _cooldown_until = 0.0 _lock = threading.Lock() def respect_cooldown(): while True: with _lock: wait = _cooldown_until - time.monotonic() if wait <= 0: return time.sleep(min(wait, 1.0)) def trigger_cooldown(seconds): # Jitter stops every worker waking at the same instant and re-hitting the limit. until = time.monotonic() + seconds + random.uniform(0, seconds * 0.3) with _lock: global _cooldown_until _cooldown_until = max(_cooldown_until, until)
A run over ten thousand documents will be interrupted. The machine reboots, the token expires, someone stops it to check something. Design for restarting from the middle from the beginning, because retrofitting it after a crash means either starting over — paying for every document twice — or guessing at what completed.
The mechanism does not need to be clever. A per-document state row, written before the call and updated after, is enough:
CREATE TABLE masking_progress ( document_id TEXT PRIMARY KEY, source_path TEXT NOT NULL, state TEXT NOT NULL, -- pending | done | review | failed regions INTEGER, error_code TEXT, attempted_at TIMESTAMPTZ, completed_at TIMESTAMPTZ ); -- Resuming is then just: pick up anything not finished. SELECT document_id, source_path FROM masking_progress WHERE state = 'pending' OR (state = 'failed' AND error_code IN ('rate_limited', 'processing_failed')) ORDER BY attempted_at NULLS FIRST LIMIT 1000;
Note which failures are eligible for a resume. A document that failed with
unsupported_file_type will fail identically on every future run; picking it up
again just spends time. Only the transient codes belong in that IN clause.
Writing the row before the call rather than after matters more than it looks. If the process dies mid-request, a row written afterwards leaves no trace that the document was ever attempted — and you cannot tell whether the quota was consumed.
Bulk scripts tend to be written as though each document either succeeds or fails. There are three outcomes, and conflating the middle one with either neighbour is the single most common way a bulk run leaves unmasked documents behind.
| Outcome | Signal | What it means | What to do |
|---|---|---|---|
| Masked | 200, X-Masked-Count > 0 |
Redacted document returned | Store it; delete the original |
| Not masked | 200, X-Masked-Count = 0 |
No number found; original returned unchanged | Route to human review — never file as done |
| Failed | 4xx or 5xx | Never processed | Retry if transient; otherwise report |
The middle row is the dangerous one because it looks like the first. Same status code, same content type, a real document in the response body. The only thing separating "this is redacted" from "this still has a live Aadhaar number on it" is a response header.
Expect a non-trivial share of a legacy backlog to land there. Old scans, faxes, photographs of photocopies, documents scanned at 150 dpi in 2014 — these are exactly the population that OCR struggles with, and exactly the population a backlog is made of. A review rate of a few percent on archive material is normal. A review rate of zero means you are not checking the header.
For why detection fails on particular documents, and what a scan needs to be readable, see why Aadhaar masking fails.
Concrete numbers, so the shape of the thing is clear. A housing finance company with 10,000 archived KYC files: roughly 70% JPEG scans, 30% multi-page PDFs.
Seven hours is one overnight window, which is usually the right way to run this: start it at close of business, have the report waiting in the morning. The report should say four things, and a run that cannot produce them is not finished:
unsupported_file_type
(TIFFs, as it turned out) and 14 were transient and succeeded on the resume pass.The 380 are the point of the exercise. Before the run they were unmasked and unknown; after it they are unmasked and listed. That list is what you hand to the team that re-scans them, and what you show a regulator to demonstrate the archive was actually examined rather than assumed clean.
Reconciling quota consumed against your own count is worth the five minutes. Your masked
count and X-Quota-Used should agree exactly, because those are the same documents.
A gap usually means retries that succeeded server-side but whose responses were lost —
harmless, but you want to know it rather than discover the discrepancy in an invoice.
Tell us the document count, the mix of images and PDFs, and the deadline. We will size the rate limit and quota for the window and quote on volume.
Discuss volume pricing Read the API docsRun the documents through the masking API with concurrency bounded by your plan's requests-per-minute, and record a per-document state row before each call so the run can resume after an interruption. Sort every result into three buckets, not two: masked, not masked (a 200 response with X-Masked-Count of zero, where the original came back unchanged), and failed. Bulk rate limits and quota are provisioned per organisation and can be raised for a migration window.
At 30 requests per minute, roughly 5.6 hours at the theoretical ceiling and about 7 hours planned at 80% headroom, which is one overnight window. Multi-page PDFs take longer per request than images, so a batch weighted towards PDFs runs below the nominal rate. Higher throughput can be provisioned for a migration.
They come back with HTTP 200 and X-Masked-Count: 0, and the response body is the original document, unchanged. This is the outcome bulk scripts most often mishandle, because it looks identical to success. Route those documents to human review and never record them as masked. A few percent of a legacy archive landing here is normal; a rate of zero means the header is not being checked.
Only if the JSON error field says rate_limited, which clears in seconds. A 429 with quota_exceeded shares the status code but will not clear until the month resets, so retrying it achieves nothing. Back off the whole worker pool rather than the individual request, since the limit is per key, and add jitter so workers do not all retry at the same instant.
Yes, if you write a per-document state row before each call rather than after. Resuming is then a query for anything still pending, plus failures whose error code was transient. Rows written only after a successful call leave no trace of documents that were in flight when the process died, and no way to tell whether quota was consumed.
No. Quota is consumed only when a document comes back redacted. Rejected requests — an invalid key, an oversized file, an unsupported type, a rate limit — cost nothing, and neither does a 200 response with X-Masked-Count of zero. That matters for a backlog of old scans, where the unreadable share can be several percent: you are not billed for documents the OCR could not read, only for ones it masked.