sk-tokenizer-bench

How many tokens does Slovak cost?

Language models bill per token, and tokenizers were trained mostly on English. Paste Slovak text to see how many tokens and how much money it costs in each model, and how much more that is than the same text in English.

Everything runs in your browser. Your text never leaves it.

I also have the English version (exact comparison)

Without an English version the comparison is estimated from the ratio measured on 1012 FLORES-200 sentences.

Cost per 1,000 texts like this

Tokens by tokenizer

Numbers with ≈ are estimates from the measured characters per token. Count exactly downloads the tokenizer from Hugging Face (once; your browser caches it) and counts your text exactly, the same way as the benchmark.

How the tokenizer splits your text

Each coloured span is one token. A dashed outline means a single character (such as ľ or ô) was split across several tokens.

Why does Slovak cost more?

A tokenizer cuts text into pieces (tokens) using a vocabulary learned from its training data. That data is mostly English, so common English words are one token while Slovak words, with their diacritics and many forms (mesto, mesta, mestu, mestom, mestách), break into several pieces. Every token is billed and takes up room in the context window.

  • Mainstream models: Slovak needs 1.6× to 2.3× the tokens of English (GPT-4o / GPT-5 1.68×, Llama 4 1.56×, GPT-4 2.13×).
  • Diacritics are not the main problem: dropping them saves only 1–5 % of tokens on modern models.
  • It is about training data: SlovakBERT, trained on Slovak, needs fewer tokens for Slovak than for English.

Full benchmark, methodology and results for 10 languages: github.com/ltchurik/sk-tokenizer-bench.

Chart: Slovak/English token ratio by tokenizer release date. OpenAI fell from 2.13× (GPT-4) to 1.68× (GPT-4o), Meta from 1.84× (Llama 3) to 1.56× (Llama 4).