Token Tax Index

LLM APIs bill per token, and tokenizers are trained mostly on English. The same sentence in another language is split into more tokens, so it costs more and fills the context window faster. This index measures that token tax for every language and every major tokenizer.

112 of 203languages need at least twice as many tokens as English on major models (median).
12.3×Santali is the most expensive language: the same text costs 12.3× the English price.
1.75× vs 2.17×The fairest major tokenizer is GPT-4o / 4.1 / 5 (o200k_base); the least fair is Claude Sonnet 4.6 (API) (median language).
1.56×Claude Opus / Sonnet / Haiku 5.5 (API) needs 1.56× as many tokens as GPT-4o / 4.1 / 5 (o200k_base) for the same English text, so its lower ratios do not mean lower absolute cost.

Your language

Bars show tokens relative to the same English text (1.00× = English, marked by the grey line). Lower is cheaper.

All 203 languages × 35 tokenizers

★ = included in the headline median (current tokenizers of major vendors; models that share a vocabulary are counted once). Click a column to sort.

cheaper than English≈ English+5–40 %+40–80 %+80–120 %+120–160 %2.6× and more
LanguageMedian ★major modelsGPT-4 / 3.5 (cl100k_base)OpenAIGPT-4o / 4.1 / 5 (o200k_base) ★OpenAIgpt-oss (o200k_harmony)OpenAIClaude Opus / Sonnet / Haiku 5.5 (API) ★AnthropicClaude Sonnet 4.6 (API) ★AnthropicGemma 2 / 3 ★GooglemT5GoogleLlama 2MetaLlama 3.x ★MetaLlama 4 ★MetaMistral / Mixtral (v3) ★MistralMistral Nemo / Small 3 / Large 2 (tekken) ★MistralDeepSeek V3 / V3.1 / R1 ★ChinaGLM-4.5 / 4.6 ★ChinaKimi K2 ★ChinaMiniMax-M2 ★ChinaQwen 2.5 / 3 ★ChinaApertus (Swiss AI)EuropeanEuroLLMEuropeanTeuken (OpenGPT-X)EuropeanBLOOMOtherClaude 2 (legacy, Xenova mirror)OtherCommand ROtherFalcon 3OtherGranite 3.3OtherNemotron Nano 2OtherOLMo 2OtherPhi-4OtherPhi-4-miniOtherSmolLM3OtherBielik (Polish, SpeakLeash)RegionalCSMPT (Czech, BUT-FIT)RegionalCSTinyLlama (Czech, BUT-FIT)RegionalSlovakBERT (Slovak, Gerulata)RegionalUTF-8 bytes (reference)Reference
Santali sat_Olck12.3312.7413.7013.709.4113.665.252.6911.8812.749.4012.3312.446.4212.747.3913.038.9212.4412.8112.5612.6213.4112.7112.5710.5212.4412.7412.7413.7012.7412.338.438.439.762.80
Tamasheq (Tifinagh) taq_Tfng10.1010.1010.2110.218.0911.735.183.588.3910.107.676.969.7910.1510.1010.1110.335.299.7910.4310.237.669.968.646.818.369.7910.1010.1010.2110.106.966.436.437.962.29
Central Atlas Tamazight tzm_Tfng10.0210.0310.1510.157.7711.274.463.488.8710.047.627.039.8110.0810.0210.0510.265.029.8110.5110.307.639.908.497.018.309.8110.0310.0310.1510.047.036.426.427.992.29
Shan shn_Mymr9.7814.987.937.937.1210.255.143.2811.8314.798.7412.289.787.9714.986.0615.3210.439.2515.8217.4511.9910.1612.7014.9712.429.2514.9814.987.9314.7912.2811.7411.7413.593.96
Odia ory_Orya8.4212.404.994.995.107.393.513.0911.5412.168.4211.9712.875.5512.406.3513.509.7012.8712.4412.301.3712.9510.059.9610.7812.8712.4012.404.9912.1611.978.238.239.522.74
Lao lao_Laoo7.979.548.238.234.045.812.621.2611.448.793.2510.6912.667.979.546.169.835.6912.6612.3212.228.6313.039.639.557.3012.669.549.548.238.7910.698.108.109.392.73
Dzongkha dzo_Tibt7.7912.148.998.995.337.704.214.197.3312.146.377.7913.386.8112.125.5912.428.0313.3813.3416.277.2417.309.6217.3414.4613.3812.1412.148.9912.147.7910.9310.9312.643.62
Tibetan bod_Tibt7.1311.188.108.104.716.833.673.666.6411.195.707.1312.296.1311.184.7911.447.2312.2912.2514.936.6015.838.9116.0813.2512.2911.1811.188.1011.197.1310.0210.0211.593.32
Punjabi (Eastern) pan_Guru6.477.852.622.626.479.392.692.109.447.813.1310.403.454.477.853.709.847.282.9711.8411.801.4412.517.978.7110.362.977.857.852.627.8110.407.837.839.072.60
Khmer khm_Khmr6.478.713.343.344.045.822.731.4110.158.723.196.1314.706.798.716.4810.576.4714.7012.2914.116.2515.069.1811.418.5714.708.718.713.348.726.139.489.4811.183.29
Tigrinya tir_Ethi5.977.785.975.973.114.512.482.027.467.783.006.757.606.127.764.617.924.157.608.047.885.128.326.486.116.447.607.787.785.977.786.755.275.276.101.75
Kannada kan_Knda5.828.861.971.975.828.441.921.4310.818.862.476.273.193.698.853.629.616.922.2112.9712.721.3010.348.5013.218.652.218.868.861.978.866.278.548.547.532.84
Amharic amh_Ethi5.787.595.785.782.693.891.941.717.257.592.396.447.435.977.594.247.744.047.437.827.675.008.076.325.966.297.437.597.595.787.596.445.125.125.931.71
Burmese mya_Mymr5.3011.623.163.164.105.932.411.568.3511.623.158.174.845.3011.623.4811.888.933.2114.1715.789.995.778.8611.609.593.2111.6211.623.1611.628.1710.6110.6112.283.52
Sinhala sin_Sinh4.578.682.692.693.034.392.291.637.778.632.717.4012.324.578.634.098.896.8812.3211.9011.678.077.397.9111.897.1912.328.688.682.698.637.407.857.859.092.62
Malayalam mal_Mlym4.428.941.961.964.045.861.851.355.538.942.7710.823.664.428.933.249.767.232.5612.8713.961.3811.599.1410.286.572.568.948.941.968.9410.829.389.3810.853.12
Tamil tam_Taml4.027.641.981.983.074.451.471.265.867.653.755.843.413.077.644.164.026.112.188.2914.321.2810.416.408.514.662.187.647.641.987.655.849.619.6111.123.19
Kabiyè kbp_Latn4.024.703.903.903.785.163.202.813.464.693.953.584.514.024.694.114.223.324.513.814.533.314.384.064.363.984.514.704.703.904.693.583.013.013.481.37
Manipuri mni_Beng3.966.713.073.073.024.382.512.215.866.673.375.353.553.286.713.964.605.613.396.8012.732.358.806.247.884.123.396.716.713.076.675.358.388.389.702.79
Telugu tel_Telu3.928.291.931.932.834.091.781.4110.668.292.837.183.123.748.293.924.176.992.2610.0411.961.339.678.008.304.932.268.298.291.938.297.188.038.039.292.68
Georgian kat_Geor3.919.751.781.782.433.512.291.534.849.762.204.842.393.919.754.266.695.192.395.7413.224.936.295.619.765.392.399.759.751.789.764.848.868.8610.262.95
Gujarati guj_Gujr3.867.661.791.792.774.021.861.729.967.672.518.633.113.867.663.756.706.742.7210.3011.251.3511.987.749.2610.042.727.667.661.797.678.637.547.548.732.51
Uyghur uig_Arab3.835.172.522.523.134.542.892.564.884.393.135.004.103.764.553.834.473.714.104.266.463.645.884.205.524.494.105.175.172.524.395.005.735.736.141.98
Assamese asm_Beng3.556.151.991.993.234.692.001.936.116.113.295.683.552.696.153.364.525.393.286.9411.551.418.115.977.104.023.286.156.151.996.115.687.637.638.842.54
Kurdish (Central) ckb_Arab3.454.802.772.772.523.652.271.744.433.942.674.343.813.074.052.684.493.453.814.295.653.195.473.315.033.993.814.804.802.773.944.345.195.195.661.79
Nuer nus_Latn3.423.993.243.243.274.362.932.422.823.983.283.263.773.423.953.473.573.383.772.863.692.783.693.473.693.423.773.993.993.243.983.262.582.582.961.32
Yiddish (Eastern) ydd_Hebr3.305.532.022.022.283.302.641.664.565.532.714.732.953.395.533.434.572.742.954.936.454.385.162.784.944.382.955.535.532.025.534.735.355.356.761.94
Armenian hye_Armn3.299.881.791.792.273.292.841.575.098.654.765.301.913.019.883.078.855.271.916.289.234.289.825.749.205.231.919.889.881.798.655.306.146.147.112.04
Kashmiri (Devanagari) kas_Deva3.154.662.232.232.443.501.941.784.432.652.394.462.733.204.663.153.314.372.452.438.161.855.193.294.654.322.454.664.662.232.654.467.157.155.732.40
Kashmiri (Arabic script) kas_Arab3.074.602.612.612.764.002.362.004.433.292.814.422.623.053.633.074.003.472.613.875.342.345.463.474.993.982.614.604.602.613.294.424.964.965.501.73
Fon fon_Latn3.053.662.882.882.963.942.652.362.833.632.882.963.583.183.633.053.122.873.582.973.422.213.482.883.393.123.583.663.662.883.632.962.352.352.841.26
Maithili mai_Deva3.024.881.761.762.383.451.641.744.622.592.054.602.783.124.883.023.234.392.261.978.711.565.263.294.854.622.264.884.881.762.594.607.647.646.002.54
Bengali ben_Beng2.955.801.701.702.713.931.211.575.355.762.054.902.862.025.802.953.755.032.056.0611.851.188.305.586.823.742.055.805.801.705.764.907.867.869.102.61
Bashkir bak_Cyrl2.954.251.751.752.042.952.311.592.903.682.393.092.572.953.702.533.303.122.572.993.503.554.323.065.053.572.574.254.251.753.683.094.864.863.721.86
Mongolian (Halh) khk_Cyrl2.943.751.891.892.083.022.331.472.753.231.912.982.372.943.162.803.083.082.372.692.993.373.842.695.143.152.373.753.751.893.232.984.914.913.841.91
Marathi mar_Deva2.905.051.811.812.423.501.401.524.582.801.904.662.903.005.052.883.284.622.142.128.991.225.623.745.304.802.145.055.051.812.804.668.108.106.632.69
Chhattisgarhi hne_Deva2.884.671.731.732.263.281.491.594.422.481.884.372.672.914.672.883.054.212.031.778.331.445.073.184.684.422.034.674.671.732.484.377.427.425.862.47
Bhojpuri bho_Deva2.874.681.771.772.223.211.571.634.522.591.954.432.752.914.682.873.084.302.081.858.571.535.023.234.824.442.084.684.681.772.594.437.487.485.922.49
Awadhi awa_Deva2.864.751.711.712.243.251.471.614.512.531.824.442.742.904.752.863.054.302.031.768.291.435.103.234.874.492.034.754.751.712.534.447.537.535.912.50
Sindhi snd_Arab2.853.981.711.712.513.642.021.744.202.862.094.312.502.893.032.803.022.852.503.004.692.494.772.844.113.422.503.983.981.712.864.314.564.565.001.60
Magahi mag_Deva2.834.681.721.722.213.211.481.614.472.561.864.392.742.884.682.833.044.192.041.768.551.455.063.254.704.442.044.684.681.722.564.397.447.445.922.47
Minangkabau (Arabic script) min_Arab2.833.972.662.662.483.592.361.844.342.922.794.432.572.833.083.082.952.782.572.794.652.584.832.854.043.402.573.973.972.662.924.435.035.035.311.75
Sanskrit san_Deva2.814.942.082.082.463.561.771.634.552.632.184.602.812.864.942.802.954.512.282.018.771.625.573.524.854.672.284.944.942.082.634.607.897.896.482.62
Nepali npi_Deva2.794.761.611.612.403.471.531.464.482.581.834.442.722.974.762.793.154.422.131.978.341.185.293.485.074.562.134.764.761.612.584.447.727.726.322.57
Central Kanuri (Arabic script) knc_Arab2.763.622.762.762.503.612.432.443.812.642.563.922.612.713.213.232.903.152.492.794.132.105.193.153.793.642.493.623.622.762.643.924.574.575.091.60
Hindi hin_Deva2.764.761.571.572.193.171.311.584.582.521.654.552.762.794.762.742.924.421.801.578.641.285.103.234.834.621.804.764.761.572.524.557.677.676.142.55
Tajik tgk_Cyrl2.743.621.891.892.153.112.291.612.702.932.262.742.682.782.902.622.972.902.682.672.723.283.732.705.143.022.683.623.621.892.932.744.864.863.542.01
Yoruba yor_Latn2.722.952.172.172.403.202.412.062.702.722.502.852.812.832.732.602.872.692.812.703.111.623.302.593.162.592.812.952.952.172.722.852.232.232.471.28
Acehnese (Arabic script) ace_Arab2.713.772.622.622.423.522.241.794.002.872.704.102.482.712.992.922.942.652.482.724.262.644.412.843.873.242.483.773.772.622.874.104.354.354.731.51
Banjar (Arabic script) bjn_Arab2.683.772.542.542.403.482.251.754.212.722.654.302.422.682.872.952.792.602.422.614.532.454.532.673.913.232.423.773.772.542.724.304.914.915.111.69
Thai tha_Thai2.684.341.961.962.683.871.600.994.352.191.554.222.871.774.343.534.752.542.375.1110.064.577.884.686.583.652.374.344.341.962.194.228.298.297.032.76
Kikuyu kik_Latn2.633.262.602.602.533.152.522.162.492.672.572.602.592.712.693.263.272.632.592.543.142.453.282.593.242.822.593.263.262.602.672.602.152.152.471.29
Pashto (Southern) pbt_Arab2.623.761.711.712.042.941.841.604.192.682.034.301.982.522.822.623.082.801.983.204.702.514.722.634.033.181.983.763.761.712.684.304.594.595.011.63
Belarusian bel_Cyrl2.583.501.991.991.892.742.161.572.362.582.232.582.152.622.642.692.712.972.152.302.893.203.572.384.882.972.153.503.501.992.582.585.165.163.782.05
Kyrgyz kir_Cyrl2.563.501.671.671.772.561.941.322.452.902.002.632.182.582.822.382.752.752.182.372.673.023.622.404.672.952.183.503.501.672.902.634.794.793.381.89
Kazakh kaz_Cyrl2.523.781.581.581.742.522.041.202.513.081.832.692.052.653.102.192.873.012.052.493.073.223.862.634.773.182.053.783.781.583.082.694.884.883.581.89
Tumbuka tum_Latn2.522.582.112.112.312.502.301.612.472.562.302.602.442.562.542.552.522.522.442.302.312.202.632.402.552.312.442.582.582.112.562.601.701.701.981.32
Tatar tat_Cyrl2.493.731.751.751.762.542.001.402.533.112.142.672.112.493.082.402.842.872.112.492.893.133.802.514.643.142.113.733.731.753.112.674.824.823.461.85
Ewe ewe_Latn2.482.732.342.342.212.782.191.812.272.632.292.352.622.482.702.522.522.342.622.292.592.102.752.402.642.382.622.732.732.342.632.351.791.792.041.06
Greek ell_Grek2.485.122.132.132.483.592.181.644.982.232.035.242.102.682.393.354.084.902.101.761.383.786.351.595.244.272.105.125.122.132.235.245.985.986.662.17
Standard Arabic (romanized) arb_Latn2.402.452.252.252.122.472.311.732.312.422.282.402.322.412.422.412.402.412.322.282.242.142.452.342.462.182.322.452.452.252.422.401.611.611.871.16
Urdu urd_Arab2.404.341.651.652.403.471.521.514.353.071.894.301.762.373.292.173.763.151.753.775.361.355.333.114.653.701.754.344.341.653.074.305.005.005.461.75
Igbo ibo_Latn2.392.461.671.672.152.652.171.812.342.402.052.392.392.422.412.432.492.382.392.343.271.722.952.412.812.162.392.462.461.672.402.392.252.252.261.22
Scottish Gaelic gla_Latn2.392.411.991.992.062.522.201.842.322.402.152.422.312.422.392.412.412.382.312.072.162.242.472.352.412.192.312.412.411.992.402.421.701.701.991.28
Azerbaijani (North) azj_Latn2.372.641.711.711.942.372.021.352.602.421.812.731.812.482.512.172.672.491.812.252.582.303.222.262.882.281.812.642.641.712.422.732.312.312.491.26
Akan aka_Latn2.362.692.092.092.322.852.201.832.212.692.272.292.602.362.672.372.402.292.602.262.572.052.532.332.522.332.602.692.692.092.692.291.781.782.091.10
Kachin kac_Latn2.332.362.122.122.182.352.121.812.272.332.212.362.252.362.322.332.362.332.252.212.322.162.422.272.302.152.252.362.362.122.332.361.661.661.941.28
Maltese mlt_Latn2.292.421.871.871.992.272.171.702.172.402.292.292.312.322.382.332.412.242.311.671.412.252.522.282.532.182.312.422.421.872.402.291.701.701.951.16
Maori mri_Latn2.292.361.831.831.902.232.041.702.192.342.102.292.182.292.312.312.302.342.182.052.122.112.382.192.292.082.182.362.361.832.342.291.721.721.861.16
West Central Oromo gaz_Latn2.282.311.881.882.102.292.131.682.222.302.162.282.222.322.272.302.292.292.222.082.092.182.362.182.292.062.222.312.311.882.302.281.611.611.881.20
Irish gle_Latn2.272.321.851.851.932.382.071.662.162.272.042.292.172.312.262.312.292.282.171.611.812.152.392.202.322.082.172.322.321.852.272.291.561.561.781.23
Samoan smo_Latn2.222.281.921.921.952.252.141.912.122.242.092.222.162.272.242.282.192.252.162.072.112.122.332.182.272.022.162.282.281.922.242.221.571.571.911.21
Tsonga tso_Latn2.212.261.741.742.072.182.091.602.162.242.112.252.162.212.242.232.252.242.162.082.032.012.292.162.232.062.162.262.261.742.242.251.501.501.761.21
Central Kanuri (Latin script) knc_Latn2.202.382.122.121.972.282.001.582.152.342.162.122.202.312.362.292.302.162.202.112.192.012.502.212.362.132.202.382.382.122.342.121.621.621.861.11
Tswana tsn_Latn2.202.271.911.912.052.202.091.652.102.262.092.192.172.252.252.252.222.242.172.052.082.002.272.172.232.002.172.272.271.912.262.191.521.521.761.24
Malagasy (Plateau) plt_Latn2.192.251.791.792.002.222.061.592.122.232.042.212.122.172.232.192.192.232.122.052.042.062.272.152.202.032.122.252.251.792.232.211.501.501.741.25
Somali som_Latn2.192.211.721.722.022.211.931.492.152.191.952.192.152.232.192.192.212.192.152.032.082.042.252.082.172.012.152.212.211.722.192.191.561.561.791.15
Bambara bam_Latn2.192.562.042.042.122.612.081.651.982.522.102.082.442.192.512.202.222.132.442.012.391.882.432.132.392.252.442.562.562.042.522.081.671.671.911.04
Bemba bem_Latn2.172.211.931.932.062.172.021.552.132.192.052.222.102.212.192.212.172.192.102.042.031.902.312.082.222.022.102.212.211.932.192.221.511.511.761.22
Turkmen tuk_Latn2.172.411.671.671.932.302.061.682.222.152.052.332.082.332.172.322.312.202.082.002.062.192.522.072.582.062.082.412.411.672.152.331.641.641.761.17
Twi twi_Latn2.172.501.861.862.132.611.951.702.012.492.062.082.392.172.472.172.182.142.392.042.381.802.322.132.292.182.392.502.501.862.492.081.651.651.931.05
Southern Sotho sot_Latn2.172.211.831.832.012.162.011.582.072.192.012.172.132.192.182.222.172.192.131.992.041.962.252.122.181.972.132.212.211.832.192.171.461.461.681.20
Kabyle kab_Latn2.162.462.142.141.982.412.081.822.002.382.162.092.302.152.422.282.252.132.302.002.402.022.382.162.362.082.302.462.462.142.382.091.651.651.901.06
Zulu zul_Latn2.152.211.721.721.962.141.981.402.102.191.962.212.102.192.182.202.152.192.102.011.961.762.302.062.191.992.102.212.211.722.192.211.511.511.741.12
Kurdish (Northern) kmr_Latn2.142.202.022.021.792.141.971.412.052.162.002.162.032.172.142.142.152.162.031.982.002.022.282.032.202.002.032.202.202.022.162.161.751.751.761.10
Latvian lvs_Latn2.132.341.811.811.742.061.891.292.092.311.732.201.912.222.302.132.272.281.911.381.072.072.442.122.342.051.912.342.341.812.312.201.681.681.811.11
Central Aymara ayr_Latn2.132.181.941.941.892.172.001.572.032.152.002.132.032.142.152.132.102.152.031.981.871.942.272.022.171.932.032.182.181.942.152.131.441.441.701.07
Uzbek (Northern) uzn_Latn2.122.171.771.771.932.141.911.382.032.121.732.131.822.132.142.042.132.151.821.981.911.982.222.052.171.941.822.172.171.772.122.131.461.461.701.13
Rundi run_Latn2.122.141.781.781.952.141.981.512.052.121.992.142.032.132.122.122.102.122.031.981.961.642.202.032.141.942.032.142.141.782.122.141.471.471.691.12
Fijian fij_Latn2.112.161.851.851.882.061.931.592.032.151.982.112.062.122.142.162.142.142.061.941.871.992.232.062.171.952.062.162.161.852.152.111.431.431.671.18
Kinyarwanda kin_Latn2.102.131.711.711.952.121.971.502.042.111.992.132.022.122.102.112.102.112.021.971.941.572.182.022.131.922.022.132.131.712.112.131.461.461.701.13
Albanian (Tosk) als_Latn2.102.251.831.831.772.051.951.482.252.221.742.371.872.302.212.102.402.231.872.132.312.162.492.292.442.171.872.252.251.832.222.371.591.591.831.20
Welsh cym_Latn2.102.131.741.741.952.101.981.702.102.121.962.182.002.102.112.092.172.112.002.062.082.112.192.172.222.002.002.132.131.742.122.181.531.531.761.07
Northern Sotho nso_Latn2.102.181.861.861.882.061.991.572.012.152.012.122.062.122.142.152.102.172.061.941.941.942.242.052.171.962.062.182.181.862.152.121.411.411.621.17
Swati ssw_Latn2.102.171.811.811.922.081.941.412.032.141.952.132.042.122.142.142.102.152.041.941.871.832.212.002.141.942.042.172.171.812.142.131.431.431.671.13
Icelandic isl_Latn2.082.141.611.611.611.931.861.311.972.091.822.091.882.102.082.102.112.101.881.891.881.992.172.022.241.861.882.142.141.612.092.091.541.541.791.09
Mossi mos_Latn2.082.282.042.041.872.312.001.782.092.152.062.072.082.162.152.252.282.042.082.052.281.962.322.102.291.992.082.282.282.042.152.071.581.581.791.02
Latgalian ltg_Latn2.082.191.881.881.742.071.901.461.982.151.862.081.932.112.152.132.182.141.931.681.551.982.242.012.211.931.932.192.191.882.152.081.501.501.671.04
Shona sna_Latn2.082.131.701.701.912.051.871.342.002.081.872.121.992.092.082.132.082.111.991.881.781.802.191.962.081.931.992.132.131.702.082.121.431.431.601.12
Tamasheq (Latin script) taq_Latn2.082.222.082.081.832.231.901.631.922.182.021.962.092.072.172.192.161.982.091.882.111.892.232.022.201.952.092.222.222.082.181.961.531.531.701.01
Kimbundu kmb_Latn2.062.131.831.831.942.021.881.482.022.111.932.121.992.092.102.112.062.111.991.881.921.812.191.952.091.941.992.132.131.832.112.121.401.401.631.11
Chichewa nya_Latn2.052.081.611.611.882.011.861.342.012.051.832.061.982.072.052.082.052.061.981.881.821.772.131.952.071.901.982.082.081.612.052.061.381.381.601.12
South Azerbaijani azb_Arab2.053.321.861.861.982.881.771.423.761.921.903.861.772.052.152.282.522.601.772.664.131.804.211.823.652.901.773.323.321.861.923.864.684.684.951.63
Guarani grn_Latn2.042.161.771.771.862.181.931.621.982.081.892.061.952.062.042.182.032.081.951.881.881.872.241.952.151.981.952.162.161.772.082.061.461.461.691.08
Silesian szl_Latn2.032.191.961.961.792.051.891.571.952.131.872.111.952.032.032.242.142.091.951.751.672.152.391.832.361.981.952.192.191.962.132.111.541.541.711.10
Faroese fao_Latn2.032.081.651.651.641.941.861.401.932.041.862.031.882.052.032.082.082.051.881.791.771.962.141.982.161.871.882.082.081.652.042.031.451.451.711.09
Iloko ilo_Latn2.032.051.841.841.922.051.861.602.012.041.862.101.962.031.972.062.042.031.961.871.911.902.171.982.071.941.962.052.051.842.042.101.451.451.651.21
Quechua (Ayacucho) quy_Latn2.022.081.831.831.832.021.851.421.972.071.872.061.952.052.042.051.982.061.951.851.731.842.141.892.041.851.952.082.081.832.072.061.441.441.681.08
Tok Pisin tpi_Latn2.022.041.791.791.802.061.921.651.952.031.942.021.982.052.022.042.012.021.981.891.881.932.091.962.081.851.982.042.041.792.032.021.401.401.611.28
Xhosa xho_Latn2.022.061.641.641.862.021.891.341.962.041.852.061.972.042.032.042.022.031.971.901.881.662.151.942.041.851.972.062.061.642.042.061.431.431.651.05
Sango sag_Latn2.012.081.821.821.841.991.871.641.952.061.922.011.952.052.022.062.012.041.951.881.991.802.121.952.041.871.952.082.081.822.062.011.391.391.601.12
Southwestern Dinka dik_Latn2.002.231.851.851.852.311.881.571.872.171.881.962.051.952.212.082.012.002.051.822.131.802.251.992.201.992.052.232.231.852.171.961.531.531.780.96
Tagalog tgl_Latn2.002.051.681.681.982.041.821.461.992.031.772.091.922.002.022.021.992.031.921.931.971.852.171.972.031.921.922.052.051.682.032.091.531.531.741.26
Dyula dyu_Latn1.992.061.861.861.732.011.851.551.892.031.892.001.932.042.022.051.991.961.931.811.871.812.131.902.031.851.932.062.061.862.032.001.351.351.531.07
Lombard lmo_Latn1.992.041.841.841.672.021.901.701.852.021.881.971.882.021.992.072.002.001.881.821.891.832.101.922.111.851.882.042.041.842.021.971.421.421.691.16
Kamba kam_Latn1.992.171.821.821.852.121.861.521.902.001.861.981.932.021.992.162.171.991.931.852.081.762.191.912.161.921.932.172.171.822.001.981.441.441.671.01
Macedonian mkd_Cyrl1.992.761.781.781.341.941.781.291.832.131.681.991.642.002.062.282.082.271.641.751.702.502.811.944.452.381.642.762.761.782.131.994.624.623.021.90
Umbundu umb_Latn1.982.031.791.791.811.981.881.481.901.991.881.961.922.011.992.042.011.991.921.801.821.762.061.912.041.811.922.032.031.791.991.961.361.361.591.05
Acehnese (Latin script) ace_Latn1.951.991.811.811.792.021.811.441.901.971.812.001.861.951.972.011.881.961.861.841.841.742.081.812.011.831.861.991.991.811.972.001.401.401.611.09
Mizo lus_Latn1.951.981.821.821.771.951.801.581.871.941.851.961.891.971.952.012.001.961.891.801.801.852.051.871.991.811.891.981.981.821.941.961.361.361.551.11
Ganda lug_Latn1.941.961.661.661.791.951.821.401.901.941.831.981.871.951.941.951.941.941.871.801.811.672.041.881.971.811.871.961.961.661.941.981.341.341.561.03
Luxembourgish ltz_Latn1.941.991.671.671.822.091.811.461.801.971.831.941.841.951.932.001.951.961.841.761.751.882.061.842.021.831.841.991.991.671.971.941.391.391.571.15
Hausa hau_Latn1.931.991.551.551.761.921.731.351.881.971.731.961.901.941.971.961.931.941.901.811.791.772.051.851.951.811.901.991.991.551.971.961.301.301.521.07
Ligurian lij_Latn1.931.981.811.811.702.001.881.691.841.971.811.961.821.971.922.021.931.951.821.771.851.812.081.892.021.841.821.981.981.811.971.961.461.461.701.17
Kongo kon_Latn1.921.991.761.761.921.921.841.451.931.971.842.021.902.001.951.991.911.971.901.791.801.742.041.831.981.811.901.991.991.761.972.021.321.321.541.14
Crimean Tatar crh_Latn1.922.101.711.711.792.121.751.312.111.821.712.231.782.141.922.062.121.941.781.682.032.062.271.702.311.851.782.102.101.711.822.231.741.741.801.12
Sardinian srd_Latn1.921.991.781.781.681.921.801.571.821.961.801.931.811.951.931.981.881.971.811.711.751.732.031.811.921.801.811.991.991.781.961.931.391.391.581.19
Sicilian scn_Latn1.922.001.811.811.721.911.801.521.811.981.801.921.841.941.932.001.931.951.841.721.781.792.051.811.981.801.842.002.001.811.981.921.351.351.591.10
Chokwe cjk_Latn1.911.961.731.731.781.911.771.391.891.941.791.981.871.921.921.981.911.921.871.771.761.702.041.811.941.801.871.961.961.731.941.981.321.321.521.06
Buginese bug_Latn1.911.961.741.741.711.941.761.421.851.941.751.951.801.911.941.951.831.951.801.791.731.692.031.771.951.791.801.961.961.741.941.951.311.311.541.09
Swahili swh_Latn1.911.951.491.491.781.901.641.241.851.931.671.941.821.941.931.911.911.941.821.731.711.242.031.811.931.771.821.951.951.491.931.941.301.301.481.05
Waray war_Latn1.911.951.721.721.911.951.791.452.021.931.802.131.851.851.921.882.021.941.851.952.011.802.202.022.051.941.851.951.951.721.932.131.481.481.681.25
Cebuano ceb_Latn1.911.931.681.681.931.951.771.421.961.911.752.061.841.831.911.851.951.921.841.891.931.782.121.951.991.871.841.931.931.681.912.061.461.461.661.20
Lithuanian lit_Latn1.902.191.681.681.641.901.671.221.872.141.611.991.731.962.141.901.992.041.731.321.031.922.241.902.151.891.732.192.191.682.141.991.391.391.601.05
Serbian srp_Cyrl1.902.901.781.781.261.821.731.301.722.181.671.901.641.902.222.271.982.331.641.901.942.562.922.024.362.481.642.902.901.782.181.904.494.493.041.80
Hungarian hun_Latn1.892.151.791.791.561.781.681.261.792.121.562.001.611.991.892.151.972.121.611.441.112.072.382.092.351.911.612.152.151.792.122.001.491.491.631.16
Wolof wol_Latn1.881.931.731.731.731.901.771.451.811.901.771.881.861.881.911.941.931.911.861.791.871.691.981.821.911.761.861.931.931.731.901.881.331.331.541.00
Luba-Lulua lua_Latn1.881.951.701.701.811.881.741.371.901.931.791.981.841.921.921.951.871.941.841.751.811.692.021.811.911.811.841.951.951.701.931.981.311.311.521.08
Finnish fin_Latn1.871.991.561.561.761.951.671.161.911.991.602.011.651.871.832.001.971.981.651.351.031.892.101.942.071.861.651.991.991.561.992.011.411.411.591.10
Lingala lin_Latn1.851.871.621.621.721.821.731.381.791.861.781.871.791.871.861.871.851.861.791.691.661.651.941.791.891.731.791.871.871.621.861.871.241.241.461.08
Slovak slk_Latn1.842.131.681.681.581.801.621.291.811.841.561.951.671.911.952.012.012.071.671.351.412.002.251.692.271.841.672.132.131.681.841.950.920.920.771.09
Bulgarian bul_Cyrl1.842.631.711.711.281.841.621.281.772.041.501.931.641.981.772.271.982.211.641.411.242.482.781.814.272.261.642.632.631.712.041.934.674.673.001.90
Ukrainian ukr_Cyrl1.832.961.751.751.351.961.611.321.711.641.541.941.732.161.832.161.992.501.731.442.202.733.021.394.582.481.732.962.961.751.641.944.724.723.311.86
Persian (Western) pes_Arab1.813.261.531.531.922.771.461.343.961.521.494.031.441.791.812.152.462.581.442.614.361.764.361.483.772.781.443.263.261.531.524.034.724.725.121.69
Basque eus_Latn1.811.881.601.601.661.811.681.211.791.861.721.891.531.741.841.891.831.871.531.711.671.151.961.761.871.731.531.881.881.601.861.891.271.271.471.06
Estonian est_Latn1.801.871.501.501.571.801.571.111.761.861.561.841.601.851.831.801.861.851.601.291.031.771.941.791.941.691.601.871.871.501.861.841.311.311.461.01
Luo luo_Latn1.801.821.681.681.721.801.721.411.751.811.731.831.761.821.801.841.811.811.761.671.691.671.911.751.841.671.761.821.821.681.811.831.251.251.421.04
Nigerian Fulfulde fuv_Latn1.801.851.601.601.571.801.651.321.721.831.651.801.741.841.821.821.831.751.741.691.691.651.911.751.801.661.741.851.851.601.831.801.221.221.420.95
Bosnian bos_Latn1.801.871.501.501.481.551.491.321.681.801.571.821.521.821.821.831.831.861.521.321.101.831.991.731.981.711.521.871.871.501.801.821.121.121.281.03
Friulian fur_Latn1.801.851.731.731.541.841.711.521.701.831.701.791.721.831.811.891.801.831.721.641.691.701.921.711.811.741.721.851.851.731.831.791.331.331.521.13
Slovenian slv_Latn1.781.891.511.511.491.551.571.201.661.801.571.781.561.811.831.831.831.871.561.311.071.801.951.701.961.681.561.891.891.511.801.781.071.071.221.01
Hebrew heb_Hebr1.773.641.481.481.682.441.731.213.293.651.743.411.631.773.642.253.051.471.633.554.512.893.201.383.242.681.633.643.641.483.653.413.793.794.821.39
Limburgish lim_Latn1.771.811.601.601.611.801.661.381.651.801.641.761.691.771.771.841.791.791.691.541.661.761.901.681.911.681.691.811.811.601.801.761.301.301.501.08
Esperanto epo_Latn1.761.871.531.531.631.811.561.191.671.851.591.771.511.761.821.761.801.791.511.571.591.641.901.711.861.671.511.871.871.531.851.771.211.211.361.01
Croatian hrv_Latn1.761.841.481.481.451.511.471.291.651.761.541.781.491.781.781.801.791.821.491.291.071.791.951.701.931.681.491.841.841.481.761.781.091.091.261.00
Occitan oci_Latn1.761.831.601.601.551.761.681.491.661.821.681.771.651.791.791.841.711.811.651.591.671.491.881.711.791.691.651.831.831.601.821.771.341.341.551.18
Sundanese sun_Latn1.731.821.391.391.661.821.501.221.761.801.571.841.681.731.781.821.621.801.681.681.631.471.921.571.831.691.681.821.821.391.801.841.231.231.431.05
Romanian ron_Latn1.731.881.631.631.511.661.571.361.701.871.471.811.611.861.731.901.811.861.611.441.191.912.081.382.031.981.611.881.881.631.871.811.601.601.761.19
Dari prs_Arab1.723.151.481.481.802.611.431.313.871.491.473.931.431.621.722.092.342.481.422.514.271.644.201.423.582.681.423.153.151.481.493.934.574.574.951.63
Papiamento pap_Latn1.701.751.581.581.551.701.601.351.641.731.611.731.611.721.701.771.671.731.611.521.481.541.811.611.731.621.611.751.751.581.731.731.171.171.341.08
Arabic (Moroccan) ary_Arab1.692.961.481.481.802.621.521.283.341.711.673.411.371.691.812.071.491.691.371.543.661.263.681.553.202.591.372.962.961.481.713.414.454.454.601.56
Balinese ban_Latn1.691.791.571.571.641.801.561.281.761.771.541.841.651.691.761.781.581.771.651.661.591.451.881.551.811.691.651.791.791.571.771.841.251.251.461.10
Polish pol_Latn1.691.911.641.641.601.721.511.301.701.891.401.901.571.681.692.111.901.761.571.221.102.132.321.342.371.841.571.911.911.641.891.901.401.401.581.12
Haitian Creole hat_Latn1.681.741.411.411.531.711.531.211.571.701.551.671.601.681.681.711.691.721.601.521.621.561.751.601.721.541.601.741.741.411.701.671.111.111.300.94
Minangkabau (Latin script) min_Latn1.681.781.561.561.671.791.551.251.781.761.551.861.621.681.761.771.581.771.621.691.651.451.911.551.841.691.621.781.781.561.761.861.241.241.421.07
Javanese jav_Latn1.671.731.371.371.621.721.481.201.711.711.461.781.631.671.711.731.551.721.631.611.621.401.841.521.771.631.631.731.731.371.711.781.211.211.401.04
Kabuverdianu kea_Latn1.651.731.551.551.531.661.561.281.581.691.551.681.561.661.651.731.611.711.561.471.431.511.771.551.681.581.561.731.731.551.691.681.121.121.281.01
Tunisian Arabic aeb_Arab1.652.911.421.421.762.551.481.283.301.651.623.351.301.641.762.011.421.651.301.483.651.193.661.503.212.601.302.912.911.421.653.354.394.394.581.54
Turkish tur_Latn1.651.921.431.431.721.981.371.122.091.401.322.221.502.041.651.951.961.621.491.291.951.962.211.222.271.731.491.921.921.431.402.221.681.681.741.12
Venetian vec_Latn1.641.711.551.551.481.671.551.371.571.701.531.691.571.671.641.741.641.681.571.441.521.581.841.521.771.611.571.711.711.551.701.691.241.241.461.06
Afrikaans afr_Latn1.641.691.431.431.531.671.521.191.551.681.501.641.521.631.641.711.681.681.521.401.561.691.791.501.791.581.521.691.691.431.681.641.221.221.411.07
Czech ces_Latn1.642.101.601.601.421.641.521.271.691.501.471.861.551.701.751.991.902.061.551.341.062.022.231.282.281.761.552.102.101.601.501.860.620.620.941.08
Arabic (Egyptian) arz_Arab1.632.951.411.411.782.591.441.273.351.651.603.411.281.631.752.021.401.631.281.473.671.173.661.493.242.591.282.952.951.411.653.414.444.444.651.56
Najdi Arabic ars_Arab1.633.031.391.391.842.661.481.343.421.661.593.481.271.621.772.061.361.631.251.453.831.153.861.493.352.701.253.033.031.391.663.484.554.554.761.60
Ta'izzi-Adeni Arabic acq_Arab1.633.001.391.391.822.641.471.313.381.651.593.431.271.621.772.051.371.631.251.453.761.153.771.483.292.661.253.003.001.391.653.434.504.504.701.58
Arabic (Modern Standard) arb_Arab1.633.031.381.381.832.661.471.343.411.651.593.471.261.621.772.051.351.631.241.443.831.153.851.483.342.701.243.033.031.381.653.474.544.544.751.60
Korean kor_Hang1.632.371.471.471.632.381.371.273.161.491.272.451.281.641.921.981.331.631.281.304.052.752.541.263.711.711.282.372.371.471.492.453.543.544.121.20
Banjar (Latin script) bjn_Latn1.621.711.451.451.621.751.411.161.751.701.411.831.551.591.701.721.461.701.551.641.571.301.921.441.781.661.551.711.711.451.701.831.251.251.411.05
Mesopotamian Arabic acm_Arab1.622.981.381.381.802.611.461.283.331.641.593.391.261.611.752.021.361.621.251.463.701.153.691.473.252.601.252.982.981.381.643.394.434.434.621.56
Norwegian Nynorsk nno_Latn1.591.641.411.411.421.551.481.171.541.631.451.631.471.621.591.701.671.631.471.331.311.641.741.641.731.541.471.641.641.411.631.631.201.201.411.03
Malay (Standard) zsm_Latn1.591.621.301.301.671.731.221.111.811.611.201.901.351.411.601.591.191.611.351.691.651.061.921.201.801.611.351.621.621.301.611.901.271.271.461.11
Russian rus_Cyrl1.582.461.421.421.321.921.361.251.621.621.231.851.501.581.422.051.731.751.501.271.902.462.681.274.502.101.502.462.461.421.621.854.964.963.101.97
North Levantine Arabic apc_Arab1.572.821.351.351.712.481.411.223.171.591.553.231.241.571.691.931.331.571.231.413.501.133.471.443.082.471.232.822.821.351.593.234.204.204.391.48
South Levantine Arabic ajp_Arab1.562.831.341.341.722.491.391.233.191.571.533.251.231.561.681.921.331.551.231.403.511.123.481.423.102.481.232.832.831.341.573.254.214.214.401.49
Catalan cat_Latn1.561.711.481.481.411.561.541.361.501.701.521.621.441.621.671.711.551.691.441.391.541.181.771.581.661.581.441.711.711.481.701.621.281.281.451.13
Danish dan_Latn1.551.621.381.381.391.471.431.141.531.611.421.621.471.621.551.721.651.601.471.261.061.671.741.651.751.561.471.621.621.381.611.621.231.231.411.04
Italian ita_Latn1.541.641.471.471.531.571.381.341.471.641.351.611.391.541.531.751.561.631.391.251.541.621.861.321.651.601.391.641.641.471.641.611.301.301.491.19
Indonesian ind_Latn1.541.551.251.251.621.691.151.071.751.541.131.841.321.351.541.571.091.541.321.641.590.961.871.091.751.571.321.551.551.251.541.841.221.221.421.08
Dutch nld_Latn1.531.601.251.251.551.531.381.161.461.601.331.601.411.501.521.691.601.591.411.191.591.711.811.261.791.571.411.601.601.251.601.601.271.271.421.12
Pangasinan pag_Latn1.521.551.431.431.451.541.451.211.491.541.461.541.481.521.531.551.501.541.481.441.451.431.581.461.541.451.481.551.551.431.541.541.111.111.260.98
German deu_Latn1.511.591.311.311.741.761.331.191.411.581.301.581.301.511.451.731.491.561.301.221.141.681.721.321.841.611.301.591.591.311.581.581.231.231.321.18
Asturian ast_Latn1.511.581.411.411.401.561.421.281.471.561.401.571.401.541.511.641.401.561.401.351.351.311.681.401.521.511.401.581.581.411.561.571.201.201.381.07
French fra_Latn1.511.611.371.371.491.631.411.401.471.601.361.601.321.541.511.731.301.581.321.351.711.201.721.341.411.621.321.611.611.371.601.601.401.401.581.24
Norwegian Bokmål nob_Latn1.511.561.351.351.381.491.401.111.491.551.381.591.411.551.511.661.621.551.411.241.211.611.681.601.701.511.411.561.561.351.551.591.181.181.381.02
Japanese jpn_Jpan1.502.271.661.661.271.861.210.902.231.511.312.141.571.421.501.641.031.401.571.323.331.772.171.222.861.581.572.272.271.661.512.143.803.803.401.26
Swedish swe_Latn1.501.581.351.351.321.431.381.111.441.571.321.541.411.551.501.681.561.561.411.241.031.641.651.591.761.491.411.581.581.351.571.541.211.211.371.04
Vietnamese vie_Latn1.492.441.491.491.842.431.371.952.911.391.332.911.432.121.462.792.781.441.433.123.901.263.511.333.302.201.432.442.441.491.392.912.802.802.961.39
Galician glg_Latn1.491.561.331.331.421.531.351.311.481.561.341.591.321.521.491.631.331.541.321.211.251.271.671.321.471.521.321.561.561.331.561.591.241.241.401.13
Spanish spa_Latn1.461.541.321.321.461.551.291.311.451.541.301.571.321.491.441.691.251.521.321.251.181.211.691.251.401.551.321.541.541.321.541.571.301.301.481.21
Portuguese por_Latn1.401.481.221.221.381.461.251.291.411.471.221.551.271.421.401.611.191.451.271.151.091.121.571.161.321.461.271.481.481.221.471.551.271.271.361.12
Chinese (Traditional) zho_Hant1.342.141.421.421.141.681.120.982.161.351.141.761.471.061.341.381.121.151.471.262.890.941.941.062.301.461.472.142.141.421.351.762.622.622.960.88
Cantonese yue_Hant1.302.091.381.381.111.641.080.952.111.311.101.711.441.021.301.331.081.111.441.222.820.911.901.022.241.421.442.092.091.381.311.712.582.582.910.87
Chinese (Simplified) zho_Hans1.081.871.251.251.141.671.080.911.981.281.061.611.400.940.950.870.911.001.401.262.600.911.691.042.021.251.401.871.871.251.281.612.692.693.040.92

Ratios vs absolute tokens

Each ratio compares a language with English on the same tokenizer. Tokenizers differ in how many tokens they spend on English itself, so a lower ratio does not always mean fewer tokens. Token counts below are for the whole FLORES-200 devtest set; prices per token also differ by model (see data.json).

TokenizerEnglish tokensvs GPT-4o / 4.1 / 5 (o200k_base)Slovaktokens · ratioGermantokens · ratioHinditokens · ratioJapanesetokens · ratio
MiniMax-M226,5670.99×53,355 · 2.01×39,650 · 1.49×77,638 · 2.92×27,510 · 1.03×
GPT-4o / 4.1 / 5 (o200k_base)26,8731.00×45,067 · 1.68×35,224 · 1.31×42,276 · 1.57×44,559 · 1.66×
DeepSeek V3 / V3.1 / R127,0411.01×51,511 · 1.91×40,957 · 1.51×75,573 · 2.79×38,441 · 1.42×
Llama 427,0481.01×42,295 · 1.56×35,060 · 1.30×44,644 · 1.65×35,318 · 1.31×
Gemma 2 / 327,1141.01×43,833 · 1.62×36,020 · 1.33×35,521 · 1.31×32,897 · 1.21×
Kimi K227,1371.01×54,544 · 2.01×46,831 · 1.73×74,336 · 2.74×44,599 · 1.64×
Llama 3.x27,1711.01×50,069 · 1.84×42,974 · 1.58×68,512 · 2.52×41,051 · 1.51×
GLM-4.5 / 4.627,1851.01×52,909 · 1.95×39,410 · 1.45×129,371 · 4.76×40,692 · 1.50×
Qwen 2.5 / 327,6211.03×57,272 · 2.07×43,058 · 1.56×121,957 · 4.42×38,743 · 1.40×
Mistral Nemo / Small 3 / Large 2 (tekken)27,8271.04×46,466 · 1.67×36,297 · 1.30×76,799 · 2.76×43,666 · 1.57×
Claude Sonnet 4.6 (API)28,8851.07×52,119 · 1.80×50,976 · 1.76×91,679 · 3.17×53,713 · 1.86×
Mistral / Mixtral (v3)29,9971.12×58,427 · 1.95×47,442 · 1.58×136,600 · 4.55×64,264 · 2.14×
Claude Opus / Sonnet / Haiku 5.5 (API)41,9161.56×66,054 · 1.58×72,753 · 1.74×91,714 · 2.19×53,223 · 1.27×

Method

Every tokenizer encodes the same 1012 professionally translated sentences of FLORES-200 (flores200_devtest) in each language. The token tax of a language is its total token count divided by the token count of the English text. Confidence intervals (bootstrap over sentences) and API prices per model are in data.json and the repository, where anyone can rerun the benchmark.

Why does this matter?

You pay the token tax on input and output, it shrinks the usable context window, and it slows generation. A model that is cheap in English can be the expensive choice for your language.

Why are some languages cheaper than English?

Tokenizers trained for a specific language (e.g. SlovakBERT for Slovak or CSMPT for Czech) encode that language more efficiently than English.

A tokenizer or model is missing

Open an issue or a pull request on GitHub; the tokenizer list is a single JSON file.

Measured 2026-11-14 with token-tax-index v0.1.0.dev0.