Contracts, invoices, resumes, and thirty years of scans: documents are where organizations lose track of personal data. This guide builds the full pipeline — text extraction, OCR for image-only files, chunking under the API limit, batch folder scans, and choosing between detection and redaction output.
Start ReadingDatabases hold the personal data you designed to collect; documents hold the personal data that happened to you. Every contract signed, invoice paid, resume received, claim filed, and letter scanned deposited identifiers into a file, and the file went wherever files go — SharePoint sites, network shares, S3 buckets, "Final_v3_ACTUAL" folders, and email attachments that became more files. Industry analyses consistently estimate that 80–90% of enterprise data is unstructured, and documents are its largest component. Almost none of it is inventoried at the level that matters: which files contain whose identifiers.
The risk profile differs from structured data in three ways. Documents are invisible to schema-based governance. A data catalog can tag the customers.email column in seconds; it has nothing to say about 1.4 million PDFs. Documents outlive their purpose. A CSV export is regenerated; a signed agreement is kept forever, along with the passport copy stapled to page 40. Documents concentrate risk in single files. One benefits enrollment spreadsheet or one scanned intake packet can carry more distinct identifiers than an entire application database table.
Ransomware crews understood this before most compliance teams did: modern extortion targets file shares precisely because that is where the un-inventoried PII lives, and breach notifications routinely balloon after the forensic review discovers what the stolen documents actually contained. Every one of those reviews is a document PII scan performed too late, at legal-emergency prices.
The fix is the same discovery primitive described in the PII detection pillar guide, applied to files: convert each document to text, run context-aware detection over it, and record what was found, where, with offsets and confidence. Everything else in this guide is the engineering of that sentence.
"Scan the documents" means four different engineering problems depending on what the bytes are. Classify first — the extraction strategy, cost, and accuracy expectations all follow from format.
Digitally created PDFs (exports from Word, invoicing systems, e-signature platforms) carry a machine-readable text layer, so extraction is fast and faithful — but layout is the trap. PDF stores positioned glyphs, not paragraphs: multi-column layouts, tables, headers, and footers can extract in visual rather than logical order, splitting a name from its context. Use layout-aware extractors (pdfplumber's word clustering, table extraction) rather than naive text dumps, and keep page numbers with each extracted block so findings can be traced back for redaction.
Office formats are zipped XML, so text extraction is nearly lossless — python-docx and openpyxl read paragraphs and cells directly. The PII hides in the parts extractors skip by default: headers and footers, speaker notes, comments and tracked changes (the deleted SSN is still in the file), embedded objects, and document properties naming every author. Spreadsheets deserve special respect: serialize rows with their column headers ("SSN: 512-84-1177") so the detector gets the context the grid implies.
Faxes, mailroom scans, photographed IDs, and legacy archives are pictures of text: no text layer at all, so nothing to extract until OCR runs. This is the format that silently defeats naive scanning programs — a folder scan that skips image-only PDFs reports "clean" for exactly the files most likely to contain intake forms and identity documents. Detect the case programmatically (a PDF page with no extractable text) and route it through the OCR branch covered below.
System exports masquerade as documents: CRM dumps, log bundles, saved emails, web page archives. They are trivially readable but structurally deceptive — a CSV of 50,000 customers is not "a file", it is a database that escaped. Flatten them to labeled text (header + value pairs), scan, and treat volume as the primary risk signal. Saved email files loop back to the email scanning guide; exports found in dev folders belong to test data detection.
Every document scanner — one-off script or petabyte crawler — is the same five stages. Get the boundaries right and each stage stays independently testable and replaceable.
Walk the corpus (folder, bucket, document management system), fingerprint each file by content type — not extension, which lies — and branch: text layer present, Office format, image-only, container. Record size and page count now; they drive chunking and OCR budgeting later. Skip nothing silently: unreadable and encrypted files go on an exceptions report, because "we couldn't scan it" is a finding.
Run the format-appropriate extractor; fall through to OCR when no text layer exists or extraction returns suspiciously little. Keep provenance with every block — file, page, region — so a downstream finding can say "SSN on page 12" rather than "somewhere in the file". Normalize whitespace and de-hyphenate line-broken words, which materially improves detection on both PDFs and OCR output.
POST the text — whole document when it fits, chunks when it doesn't — to the endpoint with your entity profile, threshold, and mask mode. One request shape covers every format because by this stage everything is just text; that uniformity is the whole reason to separate extraction from detection. The contract is on the API overview.
Roll chunk-level findings up to file level: entity types, counts, pages, confidence distribution. Then score files for triage — a resume with one phone number is routine; a spreadsheet with 4,000 SSN hits on an open share is a P1. Volume, sensitivity of type, and exposure of location make a serviceable three-factor risk score.
The findings feed whichever program commissioned the scan: quarantine or ACL-tightening for exposed files, redacted copies for sharing, deletion queues for retention enforcement, DSAR answers, or clean text for a RAG ingestion pipeline. Keep the audit trail entity-typed but value-free, so the report never becomes the next document you have to protect.
The two halves of the pipeline in working code: pdfplumber extraction feeding the detection endpoint, the raw cURL contract, and the OCR branch for image-only files.
# Python — extract a PDF with pdfplumber, then detect PII per page
# pip install pdfplumber requests
import pdfplumber, requests
ENDPOINT = "https://piidetectionapi.com/api/moderate.php"
def detect_pii(text, mask_mode="replace"):
resp = requests.post(ENDPOINT, json={
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": text,
"entities": ["PERSON_NAME", "EMAIL_ADDRESS", "PHONE_NUMBER", "SSN",
"ADDRESS", "DATE_OF_BIRTH", "CREDIT_CARD_NUMBER",
"IBAN_CODE", "FINANCIAL_ACCOUNT_NUMBER", "PASSPORT_NUMBER"],
"mask_mode": mask_mode,
"threshold": 0.4,
}, timeout=60)
resp.raise_for_status()
return resp.json()
findings = []
with pdfplumber.open("vendor_contract.pdf") as pdf:
for page in pdf.pages:
text = page.extract_text() or ""
if not text.strip():
findings.append((page.page_number, "IMAGE_ONLY — route to OCR"))
continue
result = detect_pii(text)
for e in result["detected_entities"]:
findings.append((page.page_number, e["type"], e["start"],
e["end"], round(e["confidence"], 2)))
for row in findings:
print(row) # (12, 'SSN', 448, 459, 0.97) — page-level provenance for redaction
# cURL — the same detection call, raw: extracted document text in, entities out
curl -X POST https://piidetectionapi.com/api/moderate.php \
-H "Content-Type: application/json" \
-d '{
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": "INVOICE 2211 — Bill to: Maria Alvarez, 14 Elm St, Springfield. Pay to IBAN DE89 3704 0044 0532 0130 00. Contact: [email protected], +1 415 555 0182.",
"entities": ["PERSON_NAME","ADDRESS","IBAN_CODE","EMAIL_ADDRESS","PHONE_NUMBER"],
"mask_mode": "replace",
"threshold": 0.4
}'
# Python — OCR branch for scanned/image-only documents, then the same detect call
# pip install pytesseract pdf2image pillow (textract works similarly for mixed formats)
import pytesseract
from pdf2image import convert_from_path
def ocr_pdf_to_text(path, dpi=300):
"""Rasterize each page and OCR it; 300 DPI is the accuracy sweet spot."""
pages = convert_from_path(path, dpi=dpi)
out = []
for i, img in enumerate(pages, start=1):
text = pytesseract.image_to_string(img, lang="eng")
out.append((i, text))
return out
for page_num, text in ocr_pdf_to_text("scanned_intake_form.pdf"):
if not text.strip():
continue
result = detect_pii(text) # same function as above — one contract
print(page_num, result["entities_detected"],
[e["type"] for e in result["detected_entities"]])
Full parameters, including exclude_entities and custom_instruction, are in the API documentation; a free key from get started is enough to run everything on this page.
The API accepts up to 50,000 characters per request — roughly 15–25 typical pages — so most documents scan in a single call. Long contracts, deposition transcripts, and serialized spreadsheets need splitting, and how you split determines whether you lose findings at the seams.
Three rules make chunking safe. Split on natural boundaries. Pages, sections, and paragraphs — never mid-line and never mid-sentence, because an identifier bisected by a chunk boundary ("SSN 512-84-" | "1177") is invisible to both halves. Page-level chunking, as in the pdfplumber example, is usually ideal: it respects layout and gives you page provenance for free. Overlap adjacent chunks. Carry the last few hundred characters of each chunk into the next, so an entity straddling the cut appears whole in at least one request; de-duplicate findings that land in the overlap by their absolute offsets. Preserve offset arithmetic. The API returns start/end relative to the submitted text — keep each chunk's base offset so findings map back to positions in the original file, which downstream redaction depends on.
Chunk size involves a mild trade-off. Larger chunks give the model more context and cost fewer requests; smaller chunks parallelize better and localize failures. In practice, page-sized to ~20,000-character chunks with 200–500 characters of overlap hit the sweet spot; going smaller buys nothing but request volume. If you are chunking for a downstream vector store as well, do detection first on the large chunks and let the embedding-oriented splitter work on masked text afterwards — the ordering argument is made in the RAG pipeline guide.
Finally, chunk deterministically: same file, same chunks, every run. Deterministic chunking makes scan results diffable across runs — the audit answer to "what changed since last quarter's scan?" — and makes incremental re-scans possible when only some pages of a file changed.
OCR converts the image-only backlog into scannable text, and it converts it imperfectly — which changes detection engineering more than any other factor on this page. A crooked fax renders "SSN 512-84-1177" as "SSN 5l2-B4-ll77"; a coffee-stained intake form loses a digit; a two-column layout interleaves lines from unrelated fields. Pattern-based scanners collapse under this noise, because a regex that demands nine clean digits refuses the "l"-for-"1" substitutions OCR loves.
Context-aware detection is markedly more resilient — the label "SSN", the form structure, and the shape of the token still signal an identifier even when characters are mangled — but recall on noisy scans is genuinely lower than on native text, and honest programs engineer for that. Three levers matter most. Rasterize at adequate resolution: 300 DPI is the OCR sweet spot; 150 DPI receipts and faxes measurably raise error rates. Lower the detection threshold on the OCR branch (0.3–0.4): degraded tokens legitimately earn lower confidence, and on a discovery scan a false positive costs a review click while a false negative costs an unreported identifier. Preprocess the images: deskew, binarize, and denoise before OCR — every point of character accuracy compounds into entity recall.
Route intelligently rather than OCR-ing everything: the classify step already knows which pages lack a text layer, so only those pay the OCR cost. For hybrid files — typed contract, scanned signature page with a driver's license photocopy — merge both branches' findings by page. And flag low-yield anomalies for human review: a 30-page scanned "patient records" file in which OCR found no identifiers at all is far more likely a failed OCR than a clean file.
Teams processing identity documents — passports, driver's licenses — live entirely in this branch, and healthcare's fax heritage makes OCR the default path for the intake archives discussed in the HIPAA PHI guide.
Every call returns two complementary artifacts, and document workflows use them differently. The detection output — the detected_entities array of type, matched text, start/end offsets, and confidence — is the inventory product. Discovery scans, DSAR searches, risk scoring, and audit reports consume this and never rewrite the file at all: the deliverable is knowledge, stored as entity types and counts per file and page.
The masked output — anonymized_text — is the production product: a rewritten text you can share, index, or archive. The mask_mode parameter picks the strategy. replace yields readable placeholders ("Bill to: [NAME], [ADDRESS]"), right for review copies and text handed to LLMs. redact removes spans outright for the strictest disclosures. hash substitutes consistent tokens so the same person remains trackable across a document set without being identifiable — invaluable when redacted contracts must still show that the same counterparty appears in clauses 3, 7, and 12.
One boundary matters for PDFs specifically: masking the extracted text is not the same as redacting the original file. Producing a court- or FOIA-grade redacted PDF means using the offsets to locate each finding's page coordinates (pdfplumber exposes word bounding boxes) and drawing opaque boxes onto the page — while also stripping the underlying text layer, the mistake behind every "copy-paste the black box" scandal. The detection API supplies the where-and-what with character precision; the PDF rewrite is a rendering step your pipeline owns.
Most programs need both artifacts from the same pass: the entity inventory to drive triage and compliance reporting, and masked text for the safe copy. Since one API call returns both, the design question is simply which artifact each downstream consumer receives — auditors get counts, the data lake gets hashes, the disclosure package gets redactions.
Different document classes carry predictably different identifier mixes. Tuning the entities parameter per class raises precision and keeps reports focused on what each document type actually risks.
| Document Class | Typical PII Payload | Key Entity Types | Scanning Notes |
|---|---|---|---|
| Contracts & agreements | Signatory names and titles, notice addresses, emails, phone numbers; banking details in payment schedules | PERSON_NAME, ADDRESS, EMAIL_ADDRESS, IBAN_CODE, ROUTING_NUMBER, FINANCIAL_ACCOUNT_NUMBER |
PII clusters in first and last pages plus schedules; hash mode preserves party consistency in redacted sets. Heavy in legal/eDiscovery and real estate. |
| Invoices & financial docs | Contact blocks, bank accounts, VAT/tax IDs, occasionally card numbers on remittance notes | IBAN_CODE, SWIFT_BIC, TAX_ID, CREDIT_CARD_NUMBER, PHONE_NUMBER |
Table layouts — use layout-aware extraction; checksum validation makes financial entities high-precision. See detecting bank accounts & IBANs. |
| Resumes & HR files | Full identity and contact data, DOB, education and employment history; sometimes photos, marital status, national IDs | PERSON_NAME, DATE_OF_BIRTH, ADDRESS, EMPLOYMENT, NATIONAL_ID, MARITAL_STATUS |
Nearly 100% PII by design — the question is exposure and retention, not presence. Core workflow for HR & recruiting. |
| Claims, medical & intake forms | The full HIPAA 18: MRNs, insurance IDs, DOBs, diagnoses, relatives — usually via fax-quality scans | MEDICAL_RECORD_NUMBER, HEALTH_INSURANCE_ID, SSN, DATE_OF_BIRTH, DIAGNOSIS |
OCR branch by default; low thresholds; the identifier list and de-identification rules live in the HIPAA guide. |
| Identity document copies | Passport and license numbers, faces, signatures, addresses — often photographed at angles | PASSPORT_NUMBER, DRIVERS_LICENSE_NUMBER, NATIONAL_ID, DATE_OF_BIRTH |
Highest per-page sensitivity in any corpus; preprocess aggressively before OCR and quarantine on detection rather than merely logging. |
Regulators stopped distinguishing between databases and documents long ago; obligations attach to personal data, wherever it sits. Under GDPR, a subject access request covers the PDFs on the file share as much as the CRM record — and Article 30's processing inventory, the erasure duty, and the 72-hour breach assessment all presume you can search documents by data subject. An organization that cannot scan its document stores answers DSARs by manual folder archaeology, at hourly rates that make an automated scan look free. The full mapping is in the GDPR PII detection guide.
Sector rules sharpen the point. HIPAA's Security Rule risk analysis must cover ePHI in scanned faxes and shared drives, not just the EHR. PCI DSS Requirement 3 makes unencrypted PANs in stored documents an automatic finding — and card numbers turn up in emailed order forms and scanned receipts with depressing regularity, as covered in cardholder data discovery. FOIA and public-records regimes flip the problem: government bodies must release documents, making reliable redaction a production workflow with legal consequences for every missed identifier.
Breach law completes the economics. When a file share is exfiltrated, notification duties key on which individuals' data elements were in those specific files — so the forensic document review happens either before the incident, calmly and once, or after it, urgently and expensively while the clock runs. Insurers and regulators increasingly ask the before question directly: show us your unstructured data inventory.
The strategic takeaway: document scanning is not a compliance project per regulation but a single capability that feeds all of them. One extract-then-detect pipeline, one entity taxonomy, one findings index — consumed by the GDPR program, the PCI assessor, the FOIA office, and the incident-response runbook alike. Deploy once via the cloud API, or on-premise when documents cannot leave your boundary; pricing covers both paths.
Everything on this page composed into one runnable scanner: walk a directory, extract per format, chunk under the limit, detect, and write a triage-ready report.
# Python — folder scanner: PDF/DOCX/TXT extraction, chunking, detection, CSV report
# pip install pdfplumber python-docx requests
import csv, pathlib, requests, docx, pdfplumber
ENDPOINT = "https://piidetectionapi.com/api/moderate.php"
MAX_CHARS = 50000 # API limit per request
CHUNK_SIZE = 20000 # practical chunk with headroom
OVERLAP = 300 # so boundary-straddling entities appear whole once
def extract_text(path: pathlib.Path) -> str:
if path.suffix.lower() == ".pdf":
with pdfplumber.open(path) as pdf:
return "\n".join(p.extract_text() or "" for p in pdf.pages)
if path.suffix.lower() == ".docx":
return "\n".join(p.text for p in docx.Document(path).paragraphs)
if path.suffix.lower() in (".txt", ".csv", ".log", ".html"):
return path.read_text(errors="replace")
return "" # unsupported -> exceptions report in production
def chunks(text: str):
step = CHUNK_SIZE - OVERLAP
for i in range(0, max(len(text), 1), step):
yield i, text[i:i + CHUNK_SIZE]
def scan_file(path: pathlib.Path) -> dict:
counts, confidences = {}, []
for base, chunk in chunks(extract_text(path)):
if not chunk.strip():
continue
resp = requests.post(ENDPOINT, json={
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": chunk,
"mask_mode": "replace",
"threshold": 0.4,
}, timeout=60)
resp.raise_for_status()
for e in resp.json()["detected_entities"]:
if base > 0 and e["start"] < OVERLAP:
continue # de-duplicate the overlap window
counts[e["type"]] = counts.get(e["type"], 0) + 1
confidences.append(e["confidence"])
return {"counts": counts, "total": sum(counts.values()),
"min_conf": min(confidences, default=None)}
with open("folder_scan_report.csv", "w", newline="") as f:
w = csv.writer(f)
w.writerow(["file", "total_entities", "types", "min_confidence"])
for path in sorted(pathlib.Path("/data/shared_drive").rglob("*")):
if not path.is_file():
continue
r = scan_file(path)
if r["total"]:
w.writerow([str(path), r["total"],
";".join(sorted(r["counts"])), r["min_conf"]])
Production hardening from here: a worker pool with rate limiting, checkpointing for resumability, the OCR branch for image-only files, and an exceptions list for encrypted or unreadable items. The report feeds triage directly — sort by total entities descending and start at the top.
Document scanning inherits every accuracy question from text detection and adds one of its own: extraction error compounds detection error. If the extractor mangles a table or the OCR drops a line, the detector never sees the identifier and no threshold setting can recover it. So evaluate the pipeline end to end — from file to findings — not the detection model in isolation. Take a stratified sample across your real formats (native PDFs, Office files, clean scans, terrible scans), label the identifiers a human finds in the original documents, and score the pipeline's output against that ground truth, per entity type and per format branch.
Expect the results to be format-shaped. Native-text extraction typically preserves high-90s recall; OCR branches run lower, with the deficit concentrated in numeric identifiers where single-character errors matter most. That spread is actionable: it tells you whether to invest in image preprocessing, a lower OCR-branch threshold, or re-scanning the worst source (that one department's 200-DPI fax archive) at higher quality. The complete evaluation methodology — labeling protocol, precision/recall/F1 per type, threshold tuning — is in measuring PII detection accuracy.
Tune the error trade-off to the workflow. Discovery and DSAR scans should over-trigger: low thresholds, broad entity lists, humans triaging a slightly noisy report. Automated redaction for outbound disclosure should pair low thresholds with mandatory human review of the rendered output — no serious FOIA or eDiscovery shop releases machine-redacted documents sight unseen, and the detector's job is to make that review fast and its misses rare. Precision problems, meanwhile, are usually vocabulary problems: invoice numbers flagged as accounts, part codes as IDs — fixable with exclude_entities and a custom_instruction naming your formats, without sacrificing recall.
Above all, validate on your own documents before trusting any number — vendor benchmarks were not scored on your fax machine. Ten minutes pasting representative samples (including your ugliest OCR output) into the live demo tells you more than a datasheet, and a labeled pilot on a few hundred files turns "we think it works" into a defensible accuracy statement.
anonymized_text) and precise character offsets for every finding. For a visually redacted PDF, map those offsets to page coordinates via your extractor's word bounding boxes and draw opaque boxes while removing the underlying text layer — the step that, when skipped, causes the notorious copy-pasteable "redactions." For text-destined workflows (archives, LLM context, data lakes) the masked text alone is the deliverable and no PDF rendering is needed.lang parameter), because OCR-ing German with an English model degrades characters before detection ever runs.Paste extracted text from a contract, invoice, or your worst OCR output into the live demo and watch identifiers surface with offsets and confidence — then point the batch scanner at a real folder.
Try the Live Demo View Pricing