Hebrew NLP: the tools and the problems
Why language-processing tools that work excellently in English fail in Hebrew, what to do about it, and which tools and repositories exist – for Modern, Rabbinic and Biblical Hebrew. Part A – research design.
On this page
natural language processing, NLP is the layer beneath every quantitative analysis of text – word counts, content analysis, topic modeling, sentiment detection. In English this layer is transparent: split on spaces, strip a suffix, count. In Hebrew it is not transparent, and a researcher who treats it as transparent will get numbers – silently wrong. The page explains what in Hebrew breaks general tools (section 1), the technical problem that precedes any analysis – encoding and directionality (2), what a proper Hebrew processing pipeline does and which tools implement it (3), which language models are available (4), which corpora and annotated treebanks exist (5), what is different in Biblical and Rabbinic Hebrew (6), and a practical workflow recipe for analyzing Hebrew text in social research (7) – up to the point where the text becomes numbers, and the site's calculators take over.
Token
A sequence of characters between spaces: ukhshehalakhnu ("and when we went") is one token. This is what a general tool counts – and in Hebrew it is almost never the unit of meaning.
Morpheme / segment
The minimal grammatical unit: u + kshe + halakhnu ("and" + "when" + "we went"). segmentation is the first step of every Hebrew pipeline, and it has no counterpart in English.
Lemma
The dictionary form of the word: halakh ("go"). Quantitative content analysis is almost always done on lemmas – otherwise "halakhnu," "halakhti" and "yelkhu" ("we went," "I went," "they will go") are counted as three different words.
Root
h-l-k. A deeper layer than the lemma, which also unites "halikha," "mahalakh" and "tahalikh" ("walking," "move," "process"). Most tools do not return it; a research question about roots requires a dictionary.
1. Why Hebrew is hard for a computer – four problems
A. The script lacks vowel marks, and is therefore ambiguous. The token SPR can be read sefer, sapar, safar, siper or sfar (book, barber, counted, told, frontier) – five different words in the same three letters, and the reader chooses by context without noticing. The computer cannot. In a measurement on six million words of newspaper text, the number of possible morphological analyses for a Hebrew token averages 2.4, compared with 1.4 in English (Adler & Elhadad, 2006) – which is why every Hebrew pipeline has a step that English does not: disambiguation, in which the model guesses which of the readings is correct. How good is the guess today? The best Hebrew model reported, DictaBERT-Parse-large, reaches on the treebank test 97.9% in prefix segmentation, 97.3% in part-of-speech tagging and 95.6% in morphological features – but only 85.4% in labeled syntactic parsing (LAS), and before the training corpus was expanded to 975 thousand words the numbers were slightly lower (Shmidman, Shmidman, Koppel & Tsarfaty, 2024, Table 1). In other words: at the word level above 95%, at the sentence level less; and on a text of 10,000 words even 2-5% are hundreds of errors that enter the count.
B. The morphology is rich, and the word "fuses." The prepositions, the conjunction and the definite article (u-/ve-, ha-, be-, ke-, le-, mi-, she-) attach before the word, and possessive and object pronouns – after it: beveitam = be + bayit + hem ("in their house" = in + house + their); re'itiv = ra'iti + oto ("I saw him" = I saw + him). In the Hebrew treebank (Table 3), 114,648 tokens contain 160,195 syntactic words – that is, one in three tokens in ordinary newspaper text has to be split before it can be counted. A tool that skips segmentation will count "babayit," "labayit," "habayit" and "bayit" ("in the house," "to the house," "the house," "house") as four words – and will not see the frequent word "she-" ("that") at all, because it never stands alone.
C. Root and pattern: non-concatenative morphology. In English a word is built by concatenation – walk + ed: discrete morphemes that join at the edge of the base, and therefore, as McCarthy put it, they can be recovered "by a left-to-right parse of words in search of recurrent partial strings" (McCarthy, 1981, p. 373). In Hebrew and Arabic the root is a "melody" of consonants on a separate tier, which is slotted into a prosodic template of consonant and vowel slots (pp. 386-387): k-t-b is "interwoven" into the pattern – katav, kotev, mikhtav, hikhtiv, ketuba, hitkatev (wrote, writes, letter, dictated, marriage contract, corresponded) – and McCarthy opens his article with exactly such a paradigm in Arabic (kataba, kattaba, kitaabun, maktabun…), forms that are "morphologically related to one another even though they do not share isolated strings of phonemes" (pp. 374-375). Hence there is no shared letter sequence that can be "cut" (stemming), and tools that strip suffixes – the standard in English – are meaningless in Hebrew. The solution is lemmatization with a dictionary, and even that returns a lemma (ktav, mikhtav, ketuba – as three lemmas), not a root. And for speakers the root is a real layer, not just a grammatical description: Ravid and Schiff summarize that the root "occupies a separate level of representation in the mental lexicon" of Hebrew and Arabic speakers, carrying the core meaning that links all members of the morphological family (g-d-l: gadal, gidel, higdil, gidul, migdal, megadelet, gadol… – grew, raised, enlarged, growth, tower, breeder, big…), and that children extract roots from words as early as age two (Ravid & Schiff, 2006, pp. 790-791); in their experiments with 300 schoolchildren, root identification was "the central and stable" one of the two, whereas perception of the pattern is more fragile and appears later – among other reasons because there are about 4,000 roots compared with about 200 patterns at most, and because the meaning of the pattern is not consistent (pp. 808-809). A research question about "the semantic field of a root" therefore rests on a real psychological unit – but requires an additional layer above the lemma, for which there is no ready-made tool.
D. Four registers, four languages. The accepted division in Hebrew linguistics is into four periods – Biblical, Rabbinic, Medieval and Israeli – with subdivisions and transitional stages: Biblical into Archaic, Classical and Late (post-exilic), and Rabbinic into the language of the Tannaim and the language of the Amoraim (Pérez Fernández, 1997, p. 1). These are not styles of one language but stages that succeeded one another also as a spoken language: Biblical Hebrew, "which was basically a literary dialect, ceased to be a living language in the period after the Babylonian exile," and Rabbinic Hebrew was everyday speech in Judea until the suppression of the Bar Kokhba revolt (135 CE) and afterwards only a written language alongside Aramaic (Sáenz-Badillos, 1993, pp. 166, 171). Pérez Fernández lists the differences at the opening of his book as what surprises every reader of a Rabbinic text: she- instead of asher ("that/which"), "shel" ("of") instead of the construct state, "haya" ("was") with the participle, the disappearance of the vav-consecutive, the loss of the infinitive absolute and the special imperative forms, and nun instead of mem in the plural suffix (p. 1); and only about half the vocabulary of Tannaitic Hebrew is shared with the Bible – the rest is Aramaic, Greek, and to a lesser extent Latin, Akkadian and Persian (p. 4). Medieval Hebrew was a written, not a spoken, language, revived from the tenth century under the inspiration of Arabic grammar and extended to secular poetry, philosophy and science (Sáenz-Badillos, pp. 202-203); and Israeli Hebrew is, according to Rosén (1958, as cited in Sáenz-Badillos, pp. 275-276), a koine in which Biblical and Rabbinic elements are used side by side as two stylistic levels, on a non-Hebrew substrate and with syntax and vocabulary influenced by Slavic, German and above all English (p. 280) – "what traditional grammar would have regarded as errors are, in fact, innovations of a new stage of the language" (p. 276). The implication for the computer: a model trained on Haaretz and Wikipedia (AlephBERT, DictaBERT) will fail on a page of Talmud; BEREL was built for that (Table 2); and the Bible has its own manually annotated database (BHSA, Table 4) that needs no model at all. The researcher's first choice is not "which tool" but "which Hebrew."
The basic surveys of the field: Wintner (2004) – the first decade of Hebrew computational linguistics; Itai & Wintner (2008) – the resources built at the Technion on which everything that came after stands; and what changed in the neural decade – Tsarfaty, Bareket, Klein & Seker (2020).
2. The problem nobody sees: encoding, vowel marks and directionality
Before talking about morphology, there is a layer in which Hebrew breaks without anyone noticing – and most of the "gibberish" researchers encounter comes from it.
Encoding. A Hebrew file is UTF-8 or windows-1255 (the old Windows Hebrew encoding, which still comes out of SPSS and older software). A 1255 file opened as UTF-8 looks like a string of symbols; a UTF-8 file without a BOM opened by a library that assumes a Latin encoding – the same. The site's calculator pages fix both cases automatically (see the tools map, section 7); an external tool – not necessarily. The rule: save everything as UTF-8, and check the first row by eye before any analysis. A rarer pitfall: "visual Hebrew" (the old ISO-8859-8), in which the letters are stored in display order, right to left – the words look reversed after conversion.
Vowel and cantillation marks. In Unicode, a vowel mark is a separate character that combines with the letter (combining mark): in the Hebrew block (U+0590 to U+05FF) there are 51 such marks – cantillation marks, vowel points, dagesh and the shin and sin dots – from U+0591 (etnahta) to U+05C7 (qamats qatan), alongside 27 letters (U+05D0 to U+05EA) and punctuation marks (Unicode Consortium, 2026, Hebrew block). Pointed and unpointed "shalom" are two different strings, and the pointed string has more than one storage form: the order of the marks on the same letter ("shin + qamats + shin dot" versus "shin + shin dot + qamats") and the presentation forms of the block U+FB1D to U+FB4F (46 characters, for example shin-with-shin-dot as a single character). Normalization (NFC or NFD) solves both – it decomposes the presentation forms and sorts the marks by their combining class (Whistler, 2026, the canonical ordering algorithm). Before counting – normalize, and then remove vowel marks by category (Mn, nonspacing mark) and not by range: the range U+0591 to U+05C7 also contains maqaf (U+05BE), paseq, sof pasuq and inverted nun – and whoever deletes the range deletes them too. A pointed and cantillated biblical text that has not been normalized will give every word a frequency of 1.
Full and defective spelling, final letters, Hebrew punctuation. tokhnit and tkhnit, amiti and amti, liberali and libera'li (each pair written with and without an extra vowel letter) ("program," "genuine," "liberal") – the same word, two strings, two frequencies. The Academy of the Hebrew Language's full-spelling rules (2017) did not end the split – they only changed its direction. Hebrew geresh and gershayim (U+05F3, U+05F4) versus ASCII apostrophe and quotation marks (' ") – the abbreviation for "Dr." written with gershayim and with an ASCII quotation mark ("Dr.") are two different tokens, and so are all acronyms. Hebrew maqaf (U+05BE) versus ASCII hyphen (-). There is no complete automatic solution; there is explicit normalization that is documented.
Directionality. Hebrew text is RTL, but the numbers, the English variable names and the punctuation at its end are not: in the Unicode bidirectional algorithm Hebrew letters are of type R, digits of type EN, and punctuation marks and parentheses are "neutral" – they take the direction of the strong text on both sides of them, and if the sides differ, the paragraph direction (Goregaokar & Leroy, 2026, Sections 3.2, 3.3.5) – and the result is familiar: reversed parentheses, a minus sign to the right of the number, variable names that break in a chart. The solution the standard itself offers is isolation (isolate): "characters within an isolate cannot affect the ordering of characters outside it, and vice versa," and the isolate as a whole behaves toward its surroundings as a neutral character (ibid., section 2.4). On this site that is span.en (unicode-bidi: isolate) for every Latin-script value; in R – the AGG graphics engine (see the scatter plot page, section "And how do you do it in R"). In tools such as AntConc or Voyant Hebrew works – but the display may be reversed or broken, not the analysis. To distinguish: a broken display is not evidence of a wrong count, and a wrong count (because of segmentation) is not visible in the display.
3. The pipeline: from segmentation to syntactic parsing – and the tools
A proper Hebrew processing pipeline does, in this order: segmentation (separating prefixes and suffixes), morphological analysis (all possible readings for each segment), disambiguation (choosing the correct reading in context), part-of-speech and lemma tagging, syntactic parsing (who is the subject of whom) and entity recognition (names of people, places, organizations). The central insight of the last decade is that these stages are not independent: the correct segmentation of "habayit" ("the house") depends on syntax, and syntax depends on segmentation – so the good models do them jointly, not one after the other (More, Seker, Basmova & Tsarfaty, 2019; Tsarfaty, Sadde, Klein & Seker, 2019). The current generation – DictaBERT-joint – does all six in one model.
For the social researcher the meaning is practical: there is no "tokenizer for Hebrew" that can be run before continuing in an English tool. Segmentation and lemmatization must come from a Hebrew pipeline, and its output – lemmatized and segmented text – is the input to all the other tools (topic modeling, concordance, dictionaries).
Table 1. Processing pipelines and morphological tools for Hebrew
| Tool / resource | What it does | License | Register | Notes |
|---|---|---|---|---|
| DictaBERT-joint | One model for prefix segmentation, morphology, lemma, syntactic parsing (UD) and entities – in a single call. | Open source | Modern | Shmidman, Shmidman, Koppel & Tsarfaty (2024): an "inverted pipeline" – an expert classifier for each task on the whole token, without a lexicon, 100 times faster than the previous joint model. Dicta, 2024; updated 2026. The current choice for Modern Hebrew. License CC BY 4.0. Requires Python and transformers. |
| HebPipe | A complete Python pipeline: segmentation, part-of-speech and morphology tagging, lemmas, UD parsing, entities and coreference (experimental). | Open source | Modern | Version 4.0 (2025), pip install hebpipe. Preserves alignment to the source string – important for re-annotation. Citation: Zeldes et al. (2022). |
| Stanza – Hebrew | Stanford's neural pipeline; for Hebrew: segmentation, tagging, lemma, syntactic parsing, entities. | Open source | Modern | The default "combined" package (HTB + IAHLT); entities on IAHLT data. The easiest for those who already work in Python. |
| Trankit | A multilingual pipeline based on XLM-R; a Hebrew pipeline trained on UD HTB. | Free non-commercial | Modern | The code is Apache; the Hebrew model is CC BY-NC-SA. The download servers broke (2025) – install from source. HTB only. |
| UDPipe 2 | Segmentation, tagging, lemma and syntactic parsing – in a free web service and via REST, for models of all UD treebanks. | Free non-commercial | Modern | The models are for non-commercial use. Convenient for a quick check without installation; the text is sent to a server in Prague. |
| YAP | A joint morphological analyzer, disambiguator and syntactic parser for Hebrew (Go). | Open source | Modern | ONLP Lab, Bar-Ilan. The theoretical basis of joint analysis (More et al., 2019); no new version since 2017. The lexicon – MILA's, under a separate license. |
| HebSpacy | A spaCy pipeline for Hebrew with AlephBERT-based entity recognition (16 entity types). | Open source | Modern | MIT. Low activity (0.1.7, 2025). For those who must work inside spaCy; for medicine – the accompanying HebSafeHarbor. |
| spaCy | The popular NLP library – for Hebrew it has tokenization only, with no trained pipeline. | Open source | – | Listed here as a warning: in spaCy's model table – Hebrew: "none yet." Use HebSpacy or Stanza through spacy-stanza. |
| Nakdan (Dicta) | Automatic vocalization – modern, liturgical-poetic and classical – in the browser; part of Dicta's tool suite. | Free (closed) | everyone | Shmidman et al. (2020). For vocalization for reading or reading aloud, not for analysis (vocalization removes ambiguity – but the tool itself guesses). |
| Hebrew-Tokenizer | A lightweight Python tokenizer with no dependencies: words, punctuation, dates, numbers. | Open source | Modern | MIT; no update since 2021. Tokenization only – not morphological segmentation and not tagging. |
Note. "Register" – the layer of Hebrew on which the tool was trained. All the tools in the table segment and tag; only the first ones also do syntactic parsing and entities. Development status checked in September 2026.
4. Language models for Hebrew – and what tokenization costs us
pretrained language models are what makes the joint pipeline, entity recognition and sentiment classification possible. Hebrew today has a small but serious family, all of it on Hugging Face; Table 2 arranges it by register and role. The important distinction is between encoder models (BERT and its derivatives – they analyze existing text) and generative models (DictaLM, and the large commercial models – they produce text). Quantitative research analysis needs the former; the latter are useful for summarizing and auxiliary tagging, and on this site – outside the principle of "no artificial intelligence at run time." What such a model is technically – a deep neural network that was pretrained and is fine-tuned – and when such a network suits social research and when it does not: the machine learning page, section 6.
The cost of tokenization. A language model does not see words but "pieces" (subword tokens) determined by frequency in the training corpus. In a multilingual model trained mostly on English, a Hebrew word splits into more pieces than an English word of the same length – so Hebrew text "costs" more tokens, is analyzed with lower accuracy, and in commercial models also costs more money for the same content (Rust, Pfeiffer, Vulić, Ruder & Gurevych, 2021; Ahia et al., 2023). That is why the vocabulary grew from 52K in AlephBERT to 128K in AlephBERTGimmel, and why generative models for Hebrew deliberately expand the Hebrew vocabulary. For the researcher: a model built for Hebrew is better than a multilingual model ten times larger – not because of size but because of tokenization.
Table 2. Pretrained language models for Hebrew
| Tool / resource | What it does | License | Register | Notes |
|---|---|---|---|---|
| AlephBERT | A BERT-base model for Modern Hebrew: OSCAR, Wikipedia and 70 million tweets; vocabulary 52K. | Open source | Modern | Seker et al. (2022). The first academic model at scale; 2021, not updated – but still widely used. |
| AlephBERTGimmel | The same architecture with a vocabulary of 128K – less splitting of Hebrew words into pieces. | Open source | Modern | Dicta and ONLP, 2022; CC0. The basis of Stanza's entity models. Effectively replaced by DictaBERT. |
| HeBERT / HebEMO | A BERT for Hebrew with sentiment (3 levels) and emotion (8 emotions) classifiers trained on user comments. | Open source | Modern | Chriqui & Yahav (2022): F1 .96 for polarity, .78-.97 for emotions (surprise .41) – on the coronavirus comments it was trained on. The only available tool for sentiment and emotion in Hebrew – see the warning in section 7, step 6. Not updated since 2022. |
| DictaBERT | A current BERT suite (base and large) with fine-tuned heads: segmentation, morphology, lemma, syntax, entities, question answering. | Open source | Modern | Shmidman, Shmidman & Koppel (2023); CC BY 4.0; updated. The state of the art for Modern Hebrew as of today. |
| BEREL | A BERT for Rabbinic Hebrew and Talmudic Aramaic – Talmud, Midrash, Rishonim. | Open source | Rabbinic | Shmidman et al. (2022); version 3.0. The only model trained on a non-modern register; a modern model will fail on Talmudic text. |
| DictaLM 2.0 | A generative language model (7B, based on Mistral) in Hebrew and English, with an expanded Hebrew vocabulary; an instruct version. | Open source | Modern | Shmidman et al. (2024). According to the Dicta organization on Hugging Face there is also a version 3.0 (2026). A generative model – not an analysis tool; see section 4 on the cost of tokenization. |
| fastText – word vectors for Hebrew | Static word vectors (300 dimensions) for Hebrew from Common Crawl and Wikipedia. | Open source | Modern | CC BY-SA. 2018, not updated. The approach that preceded BERT; still useful for word similarity and clustering – without morphological segmentation. |
Note. "Base" and "large" – 0.2 and 0.4 billion parameters respectively. All the models are on Hugging Face; licenses according to the model card.
Open or closed, local or remote. The digital humanities textbook proposes two axes for choosing a language model, and it is worth applying them to Table 2: a closed model run on a company's server means that "your data are captured and processed on proprietary servers," and under the terms of use of many tools the user agrees that the prompts and outputs will be used for further training; the alternative is an open model that is downloaded and run locally (for example in Ollama), and for questions about a defined collection of documents – a RAG-type retrieval system, which answers only from the repository the researcher has defined, and is therefore more accurate and suits sensitive or protected data (Drucker & Albrezzi, 2026, pp. 118-119, 121). All the models in Table 2 are of the first kind – open and running on your computer; for interview transcripts this is not a convenience but a condition (see "Where is your text?" in section 7). And another warning from the same chapter: hallucinations still happen, and whatever is not in a general model's training set looks as if it does not exist – the output of a generative model is checked against a source, not cited as a source (pp. 119, 121).
5. Corpora, treebanks and repositories
A treebank is a corpus in which humans have marked the correct segmentation, tagging and syntax – it is the "teacher" of every model, and also the yardstick by which it is measured. Hebrew has two public ones in Universal Dependencies: the veteran HTB (newspaper text from the 90s) and the new IAHLTwiki (contemporary Wikipedia). Their small size – about 5,000-6,000 sentences each, compared with tens of thousands in English – is a structural limitation of the whole field (Tsarfaty et al., 2020). Alongside them: the reference corpus of political speech – the Knesset protocols – and the large web corpora.
Table 3. Treebanks, corpora and datasets – Modern Hebrew
| Tool / resource | What it does | License | Register | Notes |
|---|---|---|---|---|
| UD Hebrew – HTB | The Hebrew Treebank: 6,143 sentences from Haaretz (the 90s), 114,648 tokens = 160,195 syntactic words. | Free non-commercial | Modern | CC BY-NC-SA. A conversion of the MILA treebank (Sima'an et al., 2001). One in three tokens splits – the number that explains everything else. |
| UD Hebrew – IAHLTwiki | 5,039 contemporary Wikipedia sentences, UD syntax and nested entities; biography, law, economics, health. | Open source | Modern | CC BY-SA 4.0. Zeldes et al. (2022). A public subset of the large IAHLT treebank; the basis of the Stanza and HebPipe models. |
| IAHLT | The Israeli Association for Human Language Technologies: treebanks for Hebrew and Arabic, entities, Knesset, open tools. | Free + paid | Modern | A gateway page and resource guide; the data themselves are on GitHub, UD and Hugging Face. Access to the full corpora – with membership. |
| NEMO | Entity annotation (9 types, nested) on the HTB treebank – at the token level and at the morpheme level. | Not verified | Modern | Bareket & Tsarfaty (2021). About 5,000 training sentences. The license is not stated in the README. |
| The Knesset Corpus | Plenary and committee protocols 1992-2024 – 35+ million sentences – morphologically and syntactically annotated, with speaker metadata. | Open source | Modern (spoken) | Goldin et al. (2025), CC BY-SA. Tens of gigabytes; for the social researcher – a unique repository of Hebrew political discourse. |
| HeDC4 | A cleaned and deduplicated Hebrew Common Crawl – about 47.6GB, about 910 thousand documents – for training models. | Not verified | Modern (web) | 2023. The license is not stated. Not for direct analysis – raw material for training. |
| heTenTen | A Hebrew web corpus of 2.7 billion words (2019-2021), tagged and lemmatized with YAP, in the Sketch Engine interface. | Paid | Modern (web) | Access by subscription; 30-day trial. The largest Hebrew corpus with a ready-made corpus-linguistics interface. |
| MILA – Knowledge Center for Processing Hebrew | The Hebrew lexicon, a morphological analyzer, corpora and the original treebank (Technion). | Free non-commercial | Modern | Itai & Wintner (2008). The site was not accessible when checked (September 2026) – a historical resource; the lexicon lives on inside YAP. |
| ivrit.ai | Hebrew speech datasets (22,000+ hours) and Whisper models fine-tuned for Hebrew – free transcription. | Free (closed) | Modern (spoken) | A nonprofit; a dedicated license. For the researcher who interviews: automatic Hebrew transcription that finally works – check on a sample. |
Note. Treebank sizes – according to the UD pages (version 2.18, 2026). "Token" – a unit between spaces; "syntactic word" – after segmentation.
6. Hebrew before Modern Hebrew: Bible, Rabbinic literature, the Middle Ages
For the Bible scholar and for the researcher of Rabbinic literature the situation is fundamentally different – and in one sense better. The entire biblical text is annotated, word by word, at a level no modern corpus has: the database of the ETCBC (the Eep Talstra Centre for Bible and Computer at the Vrije Universiteit Amsterdam; until 2013 – the WIVU informatics group, founded in 1977; Kingham & van Peursen, 2018, pp. 1-2) records for every word its spelling, morphology, part of speech and lexeme, for every phrase and every clause their boundaries and function, and for every clause also its type, its domain of discourse and its place in the clause hierarchy (ibid., p. 3). On the 426,555 words of the Leningrad Codex (according to BHS) sit about 946 thousand linguistic objects and 22.6 million feature values (Roorda, 2017, Table 18.2, p. 222), open to syntactic queries on the entire Hebrew Bible – in the browser through SHEBANQ (since 2014) and in Python through Text-Fabric (since 2017; Roorda, 2017, pp. 223-224; Roorda, 2018). The price: the linguistic model behind the annotation is a specific theory, and research results that rely on it inherit its decisions. What exactly is inherited – below, in the words of the database's builders themselves.
"Observations, not theory" – and the limits of the claim. Roorda describes the founding choice this way: the team "refrained from shaping the annotations in a hierarchical linguistic model, because it wanted to represent observations, not theory," and what emerged is "a database in the sense of observational data on which theories can be based" (Roorda, 2017, pp. 218-219). Two working principles embody this (Kingham & van Peursen, 2018, pp. 3-4; Talstra & Sikkel, 2000, pp. 54-56, Figure 7): from form to function – surface features are recorded first and only then function labels, without commitment to a linguistic theory until after the data are collected; and bottom up – from morphemes to lexemes, to phrases, to clauses, to sentences, to text, by identifying patterns that the user confirms and the software learns ("the grammar will not be presupposed, but will take shape in the course of analyzing the data" – Talstra & Sikkel, p. 46). But the builders themselves qualify this: "commitment to no theory is itself a kind of theory – one that works only if structure and function in the text are directly related" (Kingham & van Peursen, p. 4); the theoretical foundations go back to the structuralist text grammar of Wolfgang Schneider and Harald Weinrich (ibid., Note 10; Talstra & Sikkel, p. 62, Note 26); and "there can be no 'neutral,' that is, theoretically unreflected, input of linguistic data" – building the database is itself research, not merely infrastructure for it (Talstra & Sikkel, p. 53). The four decisions every result inherits:
1. Clause boundaries. Segmentation is done in two stages: first into distributional units – "atoms," sequences of words that cannot be divided further and are not interrupted – and only then into functional units, which can consist of several atoms and can therefore contain embedding and gaps (Talstra & Sikkel, 2000, pp. 47-48, 51; Kingham & van Peursen, 2018, pp. 5-6; their example: Genesis chapter three, verse 11, where an asher ("which") clause and a levilti ("not to") clause intervene between the two parts of the enclosing clause). The hard decision is the participle: when the morphology is ambiguous and adjacent phrases can be interpreted as dependent on it, the participle is treated as a predicate at the clause level – so "erets okhelet yoshveha hi" ("it is a land that devours its inhabitants," Numbers, chapter thirteen, verse 32) is coded as a separate attributive clause and not as one noun phrase (Talstra & Sikkel, pp. 49-51, 52). Whoever counts clauses, measures clause length or computes clause relations is counting this decision.
2. Clause type and domain of discourse. Every clause receives a type according to the form of the main verb and the order of constituents (wayyiqtol without an explicit subject, we-X-qatal, nominal clause, participle clause, etc.), a "kind" (verbal, nominal, without predicate) and a domain of discourse – narrative, discourse or embedded (Kingham & van Peursen, 2018, p. 3). The domain of discourse is determined by the verb and not by a literary reading: a wayyiqtol clause is marked narrative (N), a clause opening with amar or dibber ("said," "spoke") opens a quotation (Q), an imperative or yiqtol without a speech introduction – discourse (D), and in the absence of a grammatical cue the clause inherits the marking of the clause above it; thus "QN" marks wayyiqtol within direct speech, and "ND" – yiqtol within narrative (Talstra & Sikkel, 2000, p. 62). This is the text-syntax of Weinrich and Schneider, so "narrative" and "discourse" are here categories of the verbal system, not of genre. The functions of constituents (subject, predicate, object, complement) are the functional, second stage, and they are what make possible, for example, a systematic comparison of verb valency (ibid., p. 63).
3. The conjunction vav and the clause hierarchy. A clause boundary is set, among other things, when phrases of different types – a prepositional phrase, vav, a noun phrase – stand on either side of a vav, and the authors' example is Genesis chapter twenty-two, verse 1 (Kingham & van Peursen, 2018, p. 5). But the database does not stop at the boundary: it builds a hierarchy of "mother" and "daughter" clauses. Its less experimental part relies on explicit conjunctions and relative particles; beyond that the software looks for similarity – proximity of verb forms, continuity of person, number and gender, and the frequency of connections already found between the two types of clauses (ibid., p. 6). Talstra and Sikkel present this as a correction of two rejected alternatives: Richter's linear segmentation in BHt, which does not address at all the connection between main clauses by vav (Talstra & Sikkel, 2000, pp. 39-40), and Seilheimer's "top-down" analysis, which separates text levels by literary rather than linguistic criteria and gives up on participle and infinitive clauses as clauses (ibid., pp. 40-42). And importantly: the clause hierarchy and the paragraph are recorded in the database as experimental features (Kingham & van Peursen, p. 3) – whoever relies on them should say so.
4. The text, the transcription and the encoding. The text is the Leningrad Codex according to the BHS edition (Roorda, 2017, p. 218); the morphology is "encoded, not tagged" – prefixes, lexeme, distinctive vowels and suffixes in the ETCBC's internal transcription, from which the software derives the analysis (Verheij, 1994, as cited in Kingham & van Peursen, 2018, p. 4); and Roorda himself writes that the center is "set in its ways" – its own transcription and its own encoding practices, which were documented for the public only with the opening in 2014 (Roorda, 2017, p. 226). The database is updated in versions (ETCBC4, 4b, 4c; ibid., p. 221), and Text-Fabric is not always identical to the server version (Kingham & van Peursen, Note 26).
What to write in the methodology chapter. That the segmentation into clauses, the clause types, the domains of discourse and the clause hierarchy are products of the ETCBC model – bottom up, from form to function, on the foundations of Weinrich and Schneider's text grammar – and not raw facts of the text; which version of the data was used (and the BHSA number on GitHub or the DOI at DANS); and which experimental features entered the analysis. The database's builders themselves regard it as "an instrument for text-grammatical research" and not a final result (Talstra & Sikkel, 2000, p. 35) – and the researcher who cites it inherits that status as well. The final wording of this paragraph, like all the decisions on this page, is not the ETCBC's but the site's; a researcher working in computational biblical criticism will prefer to phrase it out of her own work.
Rabbinic literature has no complete manual annotation, so that is where a language model comes in: BEREL was trained on Talmud, Midrash and Rishonim – interwoven Hebrew and Aramaic – and it is the basis of Dicta's tools for vocalization, punctuation and identifying parallels in these texts. For those looking for "when did the word first appear" – the Academy's Historical Dictionary (Ma'agarim); and for those who need the texts themselves, with programmatic access – Sefaria's API.
Table 4. Hebrew before Modern Hebrew – Bible, Rabbinic literature and the Middle Ages
| Tool / resource | What it does | License | Register | Notes |
|---|---|---|---|---|
| BHSA + Text-Fabric | The Bible (Leningrad) with the ETCBC's complete morpho-syntactic database, queried in Python. | Free non-commercial | Biblical | Roorda (2017, 2018); Kingham & van Peursen (2018). CC BY-NC. The central research tool for computational biblical criticism – see section 6 on the linguistic model behind it and on what the researcher inherits from it. |
| OSHB (morphhb) | The Leningrad Codex in OSIS XML with Strong's lemmas and morphology. | Open source | Biblical | CC BY 4.0 for the annotation; the text is in the public domain. Low activity since 2019; coverage of the morphology – check book by book. |
| STEP Bible | A free Bible study tool (Tyndale House) with a Hebrew interlinear, lexicons and morphology. | Free (closed) | Biblical | Browser, no account. The tagged data are also on GitHub. For reading and spot checks, not for mass processing. |
| Sefaria API | A free REST API, with no key, for the library of Jewish texts: Hebrew Bible, Talmud, Midrash, halakha, commentators. | Free (closed) | everyone | A license for each text version separately (public domain, CC0, CC BY, CC BY-NC). Bulk export in CSV/JSON. |
| Ma'agarim (Academy of the Hebrew Language) | The Historical Dictionary corpus: search by word, root, pattern and period – from the Second Temple to Modern Hebrew. | Free non-commercial | Historical | Free for research; no API. The only place to ask "when does this word first appear." |
| Dicta Library | Talmudic and rabbinic literature with automatic vocalization, punctuation and sources. | Free (closed) | Rabbinic | Automatically processed texts (OCR + models) – verify against an edition before citing. |
Note. Text licenses vary within the same repository (on Sefaria – each text version has its own license). BEREL (Table 2) is the language model of this register.
7. Workflow recipe: quantitative analysis of Hebrew text in social research
The typical case: 400 open-ended survey responses, 30 interview transcripts, or a thousand posts. The goal: to get from the text to numbers that can be passed to the site's calculators – frequencies of categories, sentiment scores per participant, association between content and background variables. Seven steps, and each one is documented.
| # | Step | What to do – and what to document |
|---|---|---|
| 1 | Normalization | UTF-8; normalization (NFC) and removal of vowel marks by category Mn; unification of geresh/gershayim/maqaf; a decision on spelling (section 2). Document: the normalization rules as a list. |
| 2 | Segmentation and lemmatization | DictaBERT-joint or HebPipe (Table 1). Save the segmented output as a separate file – from now on this is "the text." Document: tool name, version, date. |
| 3 | Unit of counting | Lemma (the default), surface form (for research on style), or root (requires a dictionary). stop words – there is no standard list for Hebrew; build one from the segmented output (the 100 most frequent lemmas) and document it. |
| 4 | Frequencies and collocations | Concordance on the lemmas – AntConc or quanteda (see the tools map). Read the contexts, not just the numbers. |
| 5 | Topic modeling | On the lemmas only. The number of topics (k) is a decision, not a result – run several values, read examples, choose and document why. Topics are named by humans – like factors and clusters, and by the same rules (Unsupervised learning, sections 5-6). |
| 6 | Sentiment and emotion | HebEMO (Table 2) – trained and tested only on user comments about the coronavirus (Chriqui & Yahav, 2022), not on survey responses or interview transcripts; we found no study that reported its agreement on another genre. What happens without checking: Neguse et al. (2026) ran HeBERT on 1,846 citizen inquiries to a government chatbot – a bureaucratic genre, not comments – and validated against manual coding only the topic classification (50 inquiries: κ = .43, 64% agreement), not the sentiment; and even on the simpler task agreement came out moderate. Required: a manual sample of 100-200 items, coded by two people, and a comparison with the model's output; measure the agreement on the Intercoder reliability page (Cohen's κ, Krippendorff's α) and report it. Without this – there is no finding. And what the model is not sure about – do not force a decision: the "reject option" and classification metrics by the cost of error, in Performance metrics and validation, section 1. |
| 7 | From text to numbers | A category for each participant → a frequency table and chi-square against background variables; a sentiment score for each participant → correlations and a scatter plot against the survey scales. The file – in the site's standard data template: row = participant, column = variable. And those who want to predict the category or the score from the background variables – not just explain – continue to the machine learning page (the workflow recipe there, section 8). |
Common mistakes
Counting tokens and thinking you counted words. "babayit," "labayit," "habayit," "beitenu" ("in the house," "to the house," "the house," "our house") – four rows in the frequency table, one word in reality. Any analysis that skips segmentation reports a distribution of prepositions, not of content.
Running a stemmer. General tools offer "suffix stripping for Hebrew." There is no such thing in a root-and-pattern morphology; the result is letter sequences that are not words.
Passing the text to LIWC or to a translated sentiment dictionary. There is no official Hebrew LIWC dictionary: the LIWC-22 dictionary repository lists 18 languages (September 2026), and Hebrew is not among them. The only group that translated it in full – Shapira et al. (2021) – did so with 150 thousand translation candidates and about 1,000 hours of judging, and is not permitted to publish the result because LIWC is commercial (pp. 56, 62). They also explain why word-for-word translation fails: "direct translation leads to partial or wrong results" (p. 55) – inflections and verb forms with no English counterpart, different words that merge into the same lemma, and LIWC's "I" dictionary (12 words in English) which Hebrew morphology "prevents from counting" (p. 56). What exists instead: Hebrew Psychological Lexicons – lexicons of emotion, depression and well-being built for Hebrew from 872 psychotherapy transcripts and validated by three judges (Fleiss's κ .54 in the first round, .95 after reconciliation; p. 57), free on GitHub (CC BY-SA; the hepsylex package). HebEMO at least has Hebrew training – and still requires validation (section 7, step 6).
Trusting "Hebrew support" in a general tool. In spaCy Hebrew is "none yet"; in AntConc and Voyant Hebrew loads but is not segmented; in multilingual models it is analyzed through an English vocabulary. Support = tokenization, unless stated otherwise.
Running a modern model on non-modern text. AlephBERT on a page of Talmud, HebPipe on a chapter of Isaiah – the output will look fine and be mostly wrong. Register comes before the tool (section 1D).
Not normalizing – and getting two words for every word. Vowel marks not removed, gershayim not unified, full and defective spelling side by side. The symptom: the most frequent word appears with a surprisingly low frequency.
Letting the model make the decision. The number of topics, the sentiment threshold, the stop-word list – decisions of the researcher, documented and justified. A model that chose "by itself" is a model that chose someone else's default.
Checklist – before analyzing Hebrew text
| Question | Why it matters |
|---|---|
| Which Hebrew? Modern, Rabbinic, Biblical – or mixed? | Determines the tool, the model and the repository (sections 1D, 6). |
| Is the file in UTF-8? Were the vowel marks removed? Were the gershayim unified? | Without this – every word is counted twice (section 2). |
| Was the text segmented and lemmatized in a Hebrew pipeline? With which tool and which version? | Segmentation is the difference between counting content and counting particles (section 3). |
| What is the unit of counting – lemma, form or root – and why? | Three different research questions, three different numbers. |
| Where is the text sent – to my computer or to a server? | Interviews and open-ended responses – sensitive material (section 7). |
| Did I verify a manual sample against the automatic output? How many? Who coded? What is the agreement? | Digit-by-digit cross-checking – the same principle as the site's calculators. |
| Are the decisions (k, stop words, thresholds) documented and justified? | Reproduction. An undocumented computational result is not a finding. |
| Has the text become a table in the standard data template? | From here – correlations, chi-square and the other calculators, in the browser and with no data sent. |
References
Academy of the Hebrew Language. (2017). Rules of full spelling. https://hebrew-academy.org.il/topic/hahlatot/missingvocalizationspelling/
Adler, M., & Elhadad, M. (2006). An unsupervised morpheme-based HMM for Hebrew morphological disambiguation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (pp. 665-672). Association for Computational Linguistics. https://doi.org/10.3115/1220175.1220259
Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., & Tsvetkov, Y. (2023). Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 9904-9923). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.614
Bareket, D., & Tsarfaty, R. (2021). Neural modeling for named entities and morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9, 909-928. https://doi.org/10.1162/tacl_a_00404
Chriqui, A., & Yahav, I. (2022). HeBERT and HebEMO: A Hebrew BERT model and a tool for polarity analysis and emotion recognition. INFORMS Journal on Data Science, 1(1), 81-95. https://doi.org/10.1287/ijds.2022.0016
Drucker, J., & Albrezzi, F. (2026). The digital humanities coursebook: Applied concepts and critical approaches (2nd ed.). Routledge. https://doi.org/10.4324/9781003589792
Goldin, G., Howell, N., Ordan, N., Rabinovich, E., & Wintner, S. (2025). The Knesset corpus: An annotated corpus of Hebrew parliamentary proceedings. Language Resources and Evaluation, 59(3), 2973-3004. https://doi.org/10.1007/s10579-025-09833-4
Goregaokar, M., & Leroy, R. (Eds.). (2026). Unicode bidirectional algorithm (Unicode Standard Annex No. 9, Revision 52). Unicode Consortium. https://www.unicode.org/reports/tr9/
Itai, A., & Wintner, S. (2008). Language resources for Hebrew. Language Resources and Evaluation, 42(1), 75-98. https://doi.org/10.1007/s10579-007-9050-8
Kingham, C., & van Peursen, W. (2018). The ETCBC database of the Hebrew Bible. Journal for Semitics, 27(1), Article 2974. https://doi.org/10.25159/1013-8471/2974
McCarthy, J. J. (1981). A prosodic theory of nonconcatenative morphology. Linguistic Inquiry, 12(3), 373-418.
More, A., Seker, A., Basmova, V., & Tsarfaty, R. (2019). Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew. Transactions of the Association for Computational Linguistics, 7, 33-48. https://doi.org/10.1162/tacl_a_00253
Neguse, T. T., Gabay, H., Reychav, I., McHaney, R., & Jonathan, G. M. (2026). AI-based chatbots and public value creation in digital government services. Transforming Government: People, Process and Policy. Advance online publication. https://doi.org/10.1108/TG-02-2026-0074
Pérez Fernández, M. (1997). An introductory grammar of Rabbinic Hebrew (J. Elwolde, Trans.). Brill. (Original work published 1992)
Ravid, D., & Schiff, R. (2006). Roots and patterns in Hebrew language development: Evidence from written morphological analogies. Reading and Writing, 19(8), 789-818. https://doi.org/10.1007/s11145-006-9004-3
Roorda, D. (2017). The Hebrew Bible as data: Laboratory – sharing – experiences. In J. Odijk & A. van Hessen (Eds.), CLARIN in the Low Countries (pp. 217-229). Ubiquity Press. https://doi.org/10.5334/bbi.18
Roorda, D. (2018). Coding the Hebrew Bible. Research Data Journal for the Humanities and Social Sciences, 3(1), 27-41. https://doi.org/10.1163/24523666-01000011
Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., & Gurevych, I. (2021). How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 3118-3135). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.243
Sáenz-Badillos, A. (1993). A history of the Hebrew language (J. Elwolde, Trans.). Cambridge University Press. (Original work published 1988)
Seker, A., Bandel, E., Bareket, D., Brusilovsky, I., Greenfeld, R., & Tsarfaty, R. (2022). AlephBERT: Language model pre-training and evaluation from sub-word to sentence level. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 46-56). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.4
Shapira, N., Atzil-Slonim, D., Juravski, D., Baruch, M., Stolowicz-Melman, D., Paz, A., Alfi-Yogev, T., Azoulay, R., Singer, A., Revivo, M., Dahbash, C., Dayan, L., Naim, T., Gez, L., Yanai, B., Maman, A., Nadaf, A., Sarfati, E., Baloum, A., . . . Goldberg, Y. (2021). Hebrew psychological lexicons. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access (pp. 55-69). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.clpsych-1.6
Shmidman, A., Guedalia, J., Shmidman, S., Shmidman, C. S., Handel, E., & Koppel, M. (2022). Introducing BEREL: BERT embeddings for rabbinic-encoded language. arXiv. https://arxiv.org/abs/2208.01875
Shmidman, A., Shmidman, S., Koppel, M., & Goldberg, Y. (2020). Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 197-203). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-demos.23
Shmidman, S., Shmidman, A., Cohen, A. D. N., & Koppel, M. (2024). Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with enhanced vocabulary and instruction capabilities. arXiv. https://arxiv.org/abs/2407.07080
Shmidman, S., Shmidman, A., & Koppel, M. (2023). DictaBERT: A state-of-the-art BERT suite for Modern Hebrew. arXiv. https://arxiv.org/abs/2308.16687
Shmidman, S., Shmidman, A., Koppel, M., & Tsarfaty, R. (2024). MRL parsing without tears: The case of Hebrew. In Findings of the Association for Computational Linguistics: ACL 2024 (pp. 4537-4550). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.269
Sima'an, K., Itai, A., Winter, Y., Altman, A., & Nativ, N. (2001). Building a tree-bank of Modern Hebrew text. Traitement Automatique des Langues, 42(2), 347-380.
Talstra, E., & Sikkel, C. J. (2000). Genese und Kategorienentwicklung der WIVU-Datenbank, oder: Ein Versuch, dem Computer Hebräisch beizubringen. In C. Hardmeier, W.-D. Syring, J. D. Range, & E. Talstra (Eds.), Ad fontes! Quellen erfassen – lesen – deuten. Was ist Computerphilologie? Ansatzpunkte und Methodologie – Instrumente und Praxis (Applicatio 15, pp. 33-68). VU University Press.
Tsarfaty, R., Bareket, D., Klein, S., & Seker, A. (2020). From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 7396-7408). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.660
Tsarfaty, R., Sadde, S., Klein, S., & Seker, A. (2019). What's wrong with Hebrew NLP? And how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations (pp. 259-264). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-3044
Unicode Consortium. (2026). The Unicode Standard, Version 18.0.0. https://www.unicode.org/versions/Unicode18.0.0/
Whistler, K. (Ed.). (2026). Unicode normalization forms (Unicode Standard Annex No. 15, Revision 58). Unicode Consortium. https://www.unicode.org/reports/tr15/
Wintner, S. (2004). Hebrew computational linguistics: Past and future. Artificial Intelligence Review, 21(2), 113-138. https://doi.org/10.1023/B:AIRE.0000020865.73561.bc
Zeldes, A., Howell, N., Ordan, N., & Ben Moshe, Y. (2022). A second wave of UD Hebrew treebanking and cross-domain parsing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 4331-4344). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.292