Regular expressions, dictionaries, checksum validators, and transformer-based NER each catch a different slice of personal data — and each fails in a different way. Here is how they actually compare, entity type by entity type, and how production systems combine them.
Compare TechniquesEvery PII detector ever built answers the same question — "which spans of this text refer to a person or their identifiers?" — but the four major families of techniques answer it with completely different machinery. Regular expressions match character patterns. Dictionaries (gazetteers) match known strings against curated lists. Validators apply mathematical or structural rules, such as check digits, to confirm that a candidate string could be a real identifier. Named entity recognition — today almost always transformer-based — reads the text the way a human does and predicts, token by token, which spans denote which entity types.
The families are not interchangeable, and the marketing framing of "AI vs regex" obscures the real engineering question. Regex is unbeatable for some entity types and useless for others; NER is essential for unformatted PII but benefits enormously from validators cleaning up its numeric detections. Choosing a technique is really choosing a point on three axes at once: recall (what fraction of true PII you find), precision (what fraction of your detections are real), and operational cost (latency, compute, and the human effort of maintaining rules).
The decisive variable is the shape of the entity. Identifiers with a rigid machine-defined syntax — email addresses, IPv4 addresses, IBANs — sit at one extreme, where patterns describe the entity almost perfectly. Human-language entities — names, addresses, health conditions, dates written in prose — sit at the other extreme, where no finite pattern can enumerate the valid surface forms. Most real corpora contain both extremes, which is why every serious production system ends up hybrid.
This guide walks through each family's mechanics and failure modes, then puts them side by side in a per-entity accuracy table, and finishes with the hybrid architecture used by the PII Detection API itself: pattern and validator layers fused with a multilingual transformer NER model, exposed through one endpoint with a tunable confidence threshold. If you want to jump straight to observed behavior, paste your own worst-case text into the live demo and watch which techniques would have caught what.
Regular expressions earn their permanent place in PII detection through four properties. They are fast — a compiled pattern scans megabytes per second on one core, with no model loading and no GPU. They are deterministic — the same input always produces the same output, which simplifies testing and incident reproduction. They are auditable — a compliance reviewer can read \b\d{3}-\d{2}-\d{4}\b and verify exactly what it claims to catch. And they are cheap to deploy — every language runtime ships an engine, so a regex layer can live inside a logging library or a database trigger where a model cannot.
Those strengths apply only where the entity has a fixed surface syntax. The limits appear immediately outside that zone. Regex cannot detect unformatted PII: no pattern matches "my neighbor Priya moved back to Lagos after the diagnosis," yet that sentence contains a name, a location, and an implied medical fact. Regex also produces lookalike false positives: a nine-digit order number matches an SSN pattern, a sixteen-digit tracking code matches a card pattern, and a build timestamp matches a date-of-birth pattern. Pattern engines have no notion of meaning, only of shape, so every collision between an identifier format and ordinary business data becomes noise your reviewers must triage.
Internationalization multiplies the maintenance burden. Phone numbers alone span hundreds of national formats with optional country codes, extensions, and separator conventions; postal addresses, national IDs, and dates each add their own matrix of country-specific patterns. Teams that start with five patterns routinely end up curating hundreds, each one a small liability that can silently rot when an upstream format changes. And carelessly written patterns carry a performance trap: nested quantifiers can trigger catastrophic backtracking, where a single adversarial input pins a CPU for minutes — a real denial-of-service vector when the regex runs inside a request path.
The honest summary: regex is a precision instrument for a minority of entity types and a liability when stretched beyond them. The per-entity table below makes the boundary concrete.
Dictionary matching — checking tokens against curated lists of known names, cities, drugs, or employers — predates machine learning NER and still appears inside many detection stacks. It is best understood as a recall booster with a built-in ambiguity problem.
Census-derived first- and surname lists cover the most common names in a population and can flag capitalized tokens that match. Coverage is the weakness: name distributions have an enormous long tail, immigration constantly imports names absent from any national census file, and transliteration produces dozens of spellings per name. A list that covers 90% of a population by frequency still misses a large share of distinct individuals — often exactly the minority names whose holders are most exposed by a leak.
Geographic gazetteers (GeoNames-style databases) list cities, regions, and street-name vocabularies, supporting address and location detection. They work well for unambiguous place names but collide constantly with ordinary English: Reading, Nice, Mobile, and Of (a village in Turkey) are all real places. Without context, a gazetteer either over-fires on prose or is filtered so aggressively that genuine locations slip through.
Dictionary ambiguity has a canonical illustration: is "April Bloom" a person, or a phrase about spring flowers? Amber, Rose, Sky, Mark, Bill, Grace, Junior — thousands of legitimate names double as common words. A pure lookup approach must either flag every occurrence (destroying precision) or maintain fragile stop-lists (destroying recall for people actually named Bill Mark). Only context resolves the ambiguity, which is precisely what lookup methods lack.
Inside hybrid systems, dictionaries survive as features and specialized vocabularies rather than standalone detectors: drug-name and diagnosis lexicons sharpen medical-entity detection, employer and institution lists support employment data, and domain-specific terms (internal project codenames, VIP client names) can be layered on top of a model's output as custom rules. As the sole detection mechanism, they are obsolete.
Between dumb patterns and full NER sits an underrated family: validators, which apply the mathematical and structural rules that real identifiers must satisfy. They rarely find anything on their own; their job is to reject lookalikes that pattern or model layers propose, converting mediocre precision into excellent precision at almost zero cost.
The classic example is the Luhn algorithm for payment cards. Every valid card number's digits satisfy a mod-10 checksum, so a random sixteen-digit string passes only 10% of the time. A regex for card shapes might flag every order number and tracking code in your logs; adding a Luhn check silently discards nine out of ten of those false positives before a human ever sees them. Combined with issuer-prefix rules (BIN ranges), the false-positive rate drops further still.
Nearly every serious identifier has an equivalent rule. IBANs embed a mod-97 checksum over the rearranged account string, plus per-country length tables — a candidate IBAN either satisfies ISO 13616 arithmetic or it is not an IBAN. US SSNs have structural constraints rather than a checksum: area group "000", "666", and 900–999 are never issued, and group/serial zeros are invalid, which rules out many nine-digit lookalikes. National IDs elsewhere are stricter: Spain's DNI ends in a computable check letter, China's resident ID carries an ISO 7064 check digit, and so on through dozens of national schemes. Even routing numbers, IMEIs, and credit-card expiration plausibility can be validated arithmetically.
Validators have one blind spot worth naming: they evaluate validity, not sensitivity. A test card number from payment-gateway documentation passes Luhn perfectly, and a real SSN typed with one digit wrong fails structural checks even though the surrounding sentence obviously discusses a real person's SSN. That is why validators belong inside a pipeline — tightening precision on candidates that context has already made plausible — rather than acting as the final arbiter of what is sensitive. The detection engine behind this site runs exactly such validator layers on its numeric entity types; the full identifier catalogue is on the entities page.
Modern NER models are transformers fine-tuned for token classification: the input text is split into subword tokens, each token is encoded into a contextual embedding — a vector that represents the token as used in this exact sentence — and a classification head predicts, for every token, whether it begins or continues an entity span of a given type. The output is a set of typed spans with character offsets and a probability, which is exactly the detected_entities structure the API returns: type, matched text, start/end, and a confidence score.
Contextual embeddings are the whole trick. In a static word list, "Bloom" is one entry with one meaning. In a transformer, the vector for "Bloom" in "Ms. Bloom will call you" is computed from the surrounding tokens and lands in a completely different region than "Bloom" in "the cherry trees bloom in April." The model never needs a rule about honorifics; it has absorbed from training data that the pattern honorific + capitalized token + verb of communication overwhelmingly indicates a person. That is why NER handles the unformatted majority of PII — names, addresses in prose, ages, employers, health conditions — that no pattern or list can enumerate.
Two further properties matter operationally. First, multilinguality: transformers pre-trained on multilingual corpora transfer entity knowledge across languages, so one model detects names and addresses in Spanish, German, Japanese, or Arabic text without per-language rule sets — the API covers 60+ languages this way (see supported languages). Second, calibrated confidence: every span carries a score, giving you a single dial to trade precision against recall. A compliance sweep can run at threshold: 0.3 to maximize recall; a real-time chat filter can run at 0.7 to avoid interrupting users with false alarms.
The costs are real but bounded: NER needs meaningful compute (typically served behind an API rather than embedded in every process), adds tens to a few hundred milliseconds of latency, and is probabilistic — identical-looking inputs can score differently near the decision boundary, which is why measurement discipline matters. How to quantify that behavior with span-level precision and recall is the subject of our accuracy measurement guide.
The practical difference between pattern matching and NER is easiest to see on a single ambiguous string. Take the digits 4556 7375 8689 9855. In "please charge card 4556 7375 8689 9855," the nearby token card pushes the model decisively toward CREDIT_CARD_NUMBER. In "your confirmation number is 4556 7375 8689 9855," the token confirmation pushes it away. A regex sees the identical sixteen digits in both sentences and must fire — or not fire — identically in both.
This works because self-attention lets every token weigh every other token in the window when computing its embedding. The evidence the model exploits is exactly what a human reviewer would cite: trigger words ("SSN:", "born on", "lives at"), syntactic role (object of "call" vs subject of "expired"), document register (a medication name in a clinical note vs in a pharmacy's stock report), and even cross-sentence cues within the window. Context also sets entity boundaries: in "transferred to Sacred Heart Medical Center, Portland," the model must decide where the facility name ends and the city begins — a span decision no character pattern can express.
Context windows have finite size, which creates two practical rules for API users. Send coherent units — a whole ticket, log line group, or paragraph — rather than isolated fragments, because a bare "Rodriguez, 4/12/85" carries far less signal than the sentence around it. And when you must chunk long documents under the 50,000-character request limit, overlap chunk boundaries by a sentence or two so entities straddling a boundary are seen with context at least once.
custom_instruction parameter manipulates. An instruction like "employee ID numbers in the format EMP-XXXXX are not PII" gives the engine document-level context that suppresses a systematic false positive without touching any regex or retraining anything.The table summarizes how each technique family typically performs per entity type. "High/Medium/Low" describe the achievable operating point for a well-built implementation of that technique alone; the final column notes the combination production systems actually use.
| Entity Type | Regex Alone | Dictionary Alone | + Checksum/Validator | Transformer NER | Best Production Combo |
|---|---|---|---|---|---|
EMAIL_ADDRESS |
Precision high, recall high — near-perfect syntax | Not applicable | Domain plausibility adds little | High both; adds obfuscated forms ("john dot doe at gmail") | Regex + NER for obfuscations |
CREDIT_CARD_NUMBER |
Recall high, precision low (any 13–19 digits) | Not applicable | Luhn + BIN lifts precision to high | High; context separates cards from order codes | Pattern + Luhn + NER context |
SSN / national IDs |
Medium precision; 9-digit lookalikes fire constantly | Not applicable | Structural rules (invalid areas, check letters/digits) lift precision | High; trigger words ("SSN", "NI number") disambiguate unformatted forms | Per-country validator + NER |
PHONE_NUMBER |
Medium/medium — international formats explode pattern count | Not applicable | Country-code & length plausibility helps | High both, all locales, incl. "call me on five five five…" | NER with light format validation |
PERSON_NAME |
Not viable | Medium recall, low precision (April Bloom problem) | Not applicable | High both — the flagship NER entity | NER; lists only as extra features |
ADDRESS |
Low — format diversity defeats patterns | Low precision from place-name ambiguity | Postal-code format checks only | High; resolves span boundaries across street/city/region | NER + gazetteer features |
DATE_OF_BIRTH |
High recall on date shapes, but cannot tell DOB from any other date | Not applicable | Range plausibility (age 0–120) helps slightly | High — "born 4/12/85" vs "invoiced 4/12/85" is a context call | NER with date-format normalization |
MEDICAL_DATA / DIAGNOSIS |
Not viable | Medium via clinical lexicons; misses lay phrasing ("her heart thing") | Not applicable | High; understands colloquial and clinical phrasing | NER + medical lexicon features |
API_KEY / PASSWORD / secrets |
High for known prefixes (AKIA…, sk-…); low for generic tokens | Provider prefix lists help | Entropy scoring is the effective "validator" | High; flags "the admin password is hunter2" with no pattern at all | Prefix patterns + entropy + NER |
A second lens — operational rather than accuracy — completes the picture:
| Property | Regex | Dictionary | Validators | Transformer NER |
|---|---|---|---|---|
| Throughput / latency | Fastest; microseconds | Fast with proper indexing | Negligible overhead | Tens–hundreds of ms; batch to amortize |
| Maintenance burden | High — pattern sprawl per format & country | High — lists decay continuously | Low — identifier math rarely changes | Low for users of a managed API; model updates arrive centrally |
| Multilingual coverage | Per-language rework | Per-language lists | Mostly language-independent | Native — one model, 60+ languages |
| Explainability | Fully transparent | Fully transparent | Fully transparent | Score-based; explain via confidence + matched span |
No credible production detector is single-technique. The question is not "NER or regex?" but "in what order, and who overrides whom?" Three composition patterns cover nearly every deployment.
When scanning enormous corpora (logs, archives, data lakes), a cheap local regex pass first triages which documents plausibly contain formatted identifiers, and the context-aware API pass then scans the flagged subset plus a random sample of the "clean" remainder. The sample is essential — it measures what the prefilter misses (chiefly names and prose PII) so the triage rule is a measured cost decision, not a blind spot.
The inverse composition runs inside the detection engine itself: the model proposes candidate spans with confidence scores, and validators then confirm or veto numeric candidates — Luhn for cards, mod-97 for IBANs, structural rules for SSNs. A card-shaped number in card-shaped context that fails Luhn is either a typo'd real card or a fake; the engine reflects that uncertainty in a lower confidence score rather than a hard yes/no, and your threshold decides the outcome.
The final layer is organization-specific policy: entity scoping and plain-language exceptions. With the API this is configuration, not code — entities / exclude_entities scope a scan to the types a use case cares about, threshold sets the precision-recall operating point, and custom_instruction expresses exceptions like "ticket references formatted TKT-nnnnn are not identifiers." Keep these overrides in version-controlled config and re-run your evaluation set when they change.
The fastest way to internalize the comparison is to run both approaches on the same text. Start with the API baseline — one call, scoped to a few entity types, with an explicit threshold:
curl -X POST https://piidetectionapi.com/api/moderate.php \
-H "Content-Type: application/json" \
-d '{
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": "Card 4556737586899855 declined for A. Okafor; call her on 0044 7911 123456. Order 4556737586899855001 shipped.",
"entities": ["CREDIT_CARD_NUMBER", "PERSON_NAME", "PHONE_NUMBER"],
"threshold": 0.6
}'Next, a Python script that pits a naive regex-plus-Luhn scanner against the API on identical input. Run it and compare the two result sets: the regex layer finds the well-formed card number (and correctly rejects the Luhn-invalid order code), but is structurally blind to the abbreviated name and the internationally formatted phone number:
import re, requests
TEXT = ("Card 4556737586899855 declined for A. Okafor; "
"call her on 0044 7911 123456. Order 4556737586899855001 shipped.")
def luhn_ok(number: str) -> bool:
digits = [int(d) for d in number]
checksum = 0
for i, d in enumerate(reversed(digits)):
if i % 2 == 1:
d *= 2
if d > 9:
d -= 9
checksum += d
return checksum % 10 == 0
# --- Technique 1: regex + Luhn validator ---
regex_hits = []
for m in re.finditer(r"\b\d{13,19}\b", TEXT):
verdict = "CREDIT_CARD_NUMBER" if luhn_ok(m.group()) else "rejected (Luhn)"
regex_hits.append((m.group(), m.start(), verdict))
print("regex+Luhn:")
for hit in regex_hits:
print(" ", hit)
# --- Technique 2: context-aware NER via the API ---
resp = requests.post(
"https://piidetectionapi.com/api/moderate.php",
json={
"api_key": "YOUR_API_KEY",
"api_type": "pii_detection",
"text": TEXT,
"threshold": 0.5,
},
timeout=30,
)
print("API NER:")
for e in resp.json()["detected_entities"]:
print(f" {e['type']:20s} {e['text']!r} "
f"[{e['start']}:{e['end']}] conf={e['confidence']:.2f}")Finally, the hybrid triage pattern in Node.js — a cheap local prefilter decides whether a document is obviously hot, and the API provides the authoritative scan whose results are merged with (and take precedence over) the prefilter's guesses:
const PREFILTER = [
{ type: "EMAIL_ADDRESS", re: /[\w.+-]+@[\w-]+\.[\w.]+/g },
{ type: "CANDIDATE_NUMBER", re: /\b\d{9,19}\b/g },
];
function prefilter(text) {
const hits = [];
for (const { type, re } of PREFILTER) {
for (const m of text.matchAll(re)) {
hits.push({ type, text: m[0], start: m.index, source: "regex" });
}
}
return hits;
}
async function scan(text) {
const cheap = prefilter(text);
// Escalate to full NER when the prefilter fires — plus a 5%
// random audit of "clean" docs so prose-only PII is measured.
if (cheap.length === 0 && Math.random() > 0.05) {
return { entities: [], mode: "prefilter-clean" };
}
const resp = await fetch("https://piidetectionapi.com/api/moderate.php", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
api_key: process.env.PII_API_KEY,
api_type: "pii_detection",
text,
threshold: 0.5,
}),
});
const { detected_entities } = await resp.json();
// API spans are authoritative; keep prefilter hits only where
// no API entity overlaps the same offsets.
const merged = [...detected_entities];
for (const h of cheap) {
const covered = detected_entities.some(
e => h.start >= e.start && h.start < e.end
);
if (!covered) merged.push({ ...h, confidence: 0.4, flagged_for_review: true });
}
return { entities: merged, mode: "full" };
}
scan("Refund for A. Okafor, card 4556737586899855.").then(console.log);threshold per use case, and validate the choice against a labeled sample.entities or exclude_entities to detect only the types a workflow cares about. Express organization-specific exceptions in plain language with custom_instruction — for example, excluding internal reference numbers that superficially resemble phone numbers. And nothing stops you from running your own regex layer beside the API and merging results, as in the hybrid example above; the returned character offsets make overlap resolution straightforward. Whatever you layer on, keep it in versioned configuration and re-test after changes.Pattern layers, checksum validators, and multilingual transformer NER, fused behind one API call with a tunable threshold. Benchmark it against your regex stack on your own data today.
Try the Live Demo View Pricing