Token counts for the Universal Declaration of Human Rights in 23 languages.
Counted 2026-10-10 by howitticks.com. License: CC BY 4.0.

Text: Universal Declaration of Human Rights, full text, data/corpora/udhr (NFC, trimmed lines). Source: nltk_data udhr2 corpus (UDHR in Unicode).
gpt4o_o200k_tokens: o200k_base via gpt-tokenizer 4.0.0 (MIT), encoding/o200k_base.js.
gpt4_cl100k_tokens: cl100k_base via gpt-tokenizer 4.0.0.
gemini_tokens: gemini-3.1-flash-lite countTokens API, one user message, 2026-10-10; seems to include 1 extra token per count ("hello" counts 2, "hello hello" 3). Cross-check: gemini-3.8-flash gave the same count on 23 of 23 texts (2026-10-10).
characters: grapheme clusters (Intl.Segmenter). words: word-like segments (Intl.Segmenter, language of the text).
Special-token strings, if any, were counted as plain text.

Open models (tokens-open-models-2026-10.csv): tokenizer.json from Hugging Face at the pinned revision in each row, tokenizers 0.22.1, Tokenizer.from_str(tokenizer.json), encode(text, add_special_tokens=False); count = len(ids). No chat template, no BOS.

Books (tokens-books-2026-10.csv): Project Gutenberg plain text, between the START and END markers; line wraps inside a paragraph joined with a space, blank lines kept as paragraph breaks. gpt4o_tokens_with_original_line_breaks counts the text with the original line wraps. Downloaded from Project Gutenberg 2026-10-10; sha256 of each file in the project's run record.