Skip to content
anonymizer

Language packs

A language pack bundles the NER model and the tokenization rules for one language. The current release ships with two built-in packs — Russian (via Natasha) and English (via spaCy en_core_web_lg) — plus optional Spanish, German and Italian packs (see below).

What a pack provides

  • An NER model that emits Entity records for Persons, Companies, and Locations
  • A tokenizer that respects the language’s sentence boundaries and case
  • Optional surname/given-name postprocessing (Russian patronymics, English suffixes like Jr.)

Packs are loaded on demand. If you process only .docx files in English, the Russian pack stays on disk but never loads into memory.

Active packs

The product loads the language pack needed for the document’s predominant language. If a .docx mixes RU and EN paragraphs, both packs can run. Russian and English are always active. Spanish, German and Italian activate automatically once their models are installed (see below) — there is no per-run language flag; activation is driven by which models are present.

Tesseract OCR language data is separate from these NER packs: scanned PDF OCR requires Tesseract’s eng and rus traineddata files even though the NER models are bundled with anonymizer.

Enable Spanish (optional)

Russian and English work out of the box. Spanish is opt-in — it needs its model (es_core_news_lg, ~0.5 GB) downloaded once. Two ways to add it:

At install — pick Spanish in the installer prompt, or set it non-interactively:

Terminal window
curl -fsSL anonymizer.site/install | ANONYMIZER_LANGS=es sh

From the app — open Settings → Languages, install Spanish, then restart anonymize.

Once the model is present, Spanish documents get names (including double surnames like Juan García López), companies (S.A., S.L.), and addresses (calle, avenida) detected — alongside the always-on email, phone, IBAN and date detection. Without the model, Spanish documents still get those regex-based detectors; nothing breaks, the Spanish names and companies just aren’t recognized.

The model download needs network once. After that, processing is fully offline like the rest of the tool. Russian and English are unaffected either way. Spanish detection is pilot-grade in this release.

Enable German (optional)

German works the same opt-in way. It needs its model (de_core_news_lg, ~0.5 GB) downloaded once:

At install — pick German in the installer prompt, or set it non-interactively:

Terminal window
curl -fsSL anonymizer.site/install | ANONYMIZER_LANGS=de sh

(Languages can be combined: ANONYMIZER_LANGS="es de it".)

From the app — open Settings → Languages, install German, then restart anonymize.

Once the model is present, German documents get names (including particles like von/zu and academic titles), companies with German legal forms (GmbH, AG, GmbH & Co. KG, e.K. …), street addresses with PLZ (including suffixless streets like Prinzipalmarkt 12, 48143 Münster), German phone formats, and verbose German dates (15. Januar 2024) detected — alongside the always-on email, IBAN and date detection. Without the model, German documents still get the regex-based detectors; nothing breaks.

Enable Italian (optional)

Italian (added in 0.6.1) works the same opt-in way. It needs its model (it_core_news_lg, ~0.5 GB) downloaded once:

At install — pick Italian in the installer prompt, or set it non-interactively:

Terminal window
curl -fsSL anonymizer.site/install | ANONYMIZER_LANGS=it sh

From the app — open Settings → Languages, install Italian, then restart anonymize.

Once the model is present, Italian documents get names, companies with Italian legal forms (S.r.l., S.p.A. …), structured street addresses (including CAP postal codes), Italian phone formats, verbose notarial dates (il quindici gennaio duemilaventiquattro), and label-anchored identifiers — codice fiscale (including the C.F./cod. fisc. abbreviations), Partita IVA, cadastral references (Foglio/Particella/Subalterno), deed registers (Rep./Racc.), and C.C.I.A.A. numbers — detected alongside the always-on email, IBAN and date detection. Without the model, Italian documents still get the regex-based detectors; nothing breaks. Note: the upstream it_core_news_lg model is licensed CC BY-NC-SA 3.0.

Roadmap for additional languages

MVP-2 adds the remaining languages from the customer brief (French, Portuguese, Dutch, Polish, Czech, Turkish, and more — 25 in total). Until then, language-agnostic regex detectors (emails, phones, IBANs, etc.) still work on documents in any language.

Adding a custom pack

If you need a language not yet supported, see the engineer-facing guide in the product repo: docs/agents/extending-language.md. It walks through the LanguagePackRegistry registration.