Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is Unicode text standardization in python". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

I didn't know one of its applications until I encountered the unicodedata module recently. Some characters can be represented by multiple legal encodings, which can cause some problems.

For example, a character ñ can be represented by either\ u00f1 or n\ u0303, as shown below:

In [2]:'\ u00f1'

Out [2]:'ñ'

In [3]:'n\ u0303'# notice that there is a character n before it.

Out [3]:'ñ'

The reason is that the first representation\ u00f1 is the overall representation, and the second n\ u0303 is a combinatorial representation, which is a combination of n and character ~.

Obviously, using multiple representations of characters like the above can cause problems in programs that need to compare strings, as shown below:

In [4]: s1century'\ u00f1'

In [5]: s2roomn\ u0303'

In [6]: s1==s2

Out [6]: False

We expect the characters ñ above to be equal in both representations, which requires the use of the unicodedata module to standardize these characters:

S1N1'\ u00f1'

S2roomroomn\ u0303'

T1 = unicodedata.normalize ('NFC', S1)

T2 = unicodedata.normalize ('NFC', S2)

In [25]: t1==t2

Out [25]: True

The first argument to normalize () specifies how the string is normalized. NFC indicates that characters should be made up as a whole, and there are other standardized methods such as NFD, where the combination of the characters n and\ u0303 is the NFD representation.

In NFC, the Eagstrom symbol is always replaced by the visually identical U+00C5 (A with rings at the top). In NFD, it is replaced by a sequence of two characters, Ubun0041 (A) and Ubun030A (°).

Normalization is important for any program that needs to process Unicode text in a consistent manner because it affects the meaning of comparison, search, and sorting.

This is the end of the content of "what is Unicode text Standardization in python". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.