Shulou(Shulou.com)11/24 Report--

CTOnews.com, July 31 (Xinhua): the language used by users has a great impact on the cost of large language models (LLM), which may create an artificial intelligence gap between English speakers and other language speakers. A recent study shows that because of the server cost measurement and billing methods used for services such as OpenAI, the cost of English input and output is much lower than that of other languages, with simplified Chinese about twice as expensive as English, Spanish 1.5 times as much as English, and Burmese Shan 15 times as much as English.

CTOnews.com noted that Twitter user Dylan Patel (@ dlan522p) shared a photo showing a study conducted by the University of Oxford, which found that it takes 198 tokens for a LLM to process a Burmese sentence, while it takes only 17 words to write the same sentence in English. Morphemes represent the computational cost of accessing LLM through API (such as OpenAI's ChatGPT or Anthropic's Claude 2), which means that Burmese sentences are 11 times more expensive to use this service than English sentences.

The lexical model, the way artificial intelligence companies convert user input into calculating costs, means that it is much more expensive to use and train models in languages other than English. This is because languages like Chinese have different and more complex structures (whether in terms of grammar or the number of characters), which leads to higher lexicalization. For example, according to OpenAI's GPT3 word splitter, the word "your affection" requires only two words in English, but eight in simplified Chinese. Although simplified Chinese text is only 4 characters (your love), while English text has 14 characters.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.