Language models bill per token, and tokenizers were trained mostly on English. Paste Slovak text to see how many tokens and how much money it costs in each model, and how much more that is than the same text in English.
Everything runs in your browser. Your text never leaves it.
Without an English version the comparison is estimated from the ratio measured on 1012 FLORES-200 sentences.
Numbers with ≈ are estimates from the measured characters per token. Count exactly downloads the tokenizer from Hugging Face (once; your browser caches it) and counts your text exactly, the same way as the benchmark.
Each coloured span is one token. A dashed outline means a single character (such as ľ or ô) was split across several tokens.
A tokenizer cuts text into pieces (tokens) using a vocabulary learned from its training data. That data is mostly English, so common English words are one token while Slovak words, with their diacritics and many forms (mesto, mesta, mestu, mestom, mestách), break into several pieces. Every token is billed and takes up room in the context window.
Full benchmark, methodology and results for 10 languages: github.com/ltchurik/sk-tokenizer-bench.