Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the method of python dealing with Unicode files". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's way of thinking to study and learn "what is the method of python dealing with Unicode files".

For practitioners of natural language processing, processing Unicode files is a nightmare, especially using the Windows operating system. Imagine the frustration when you encounter an error during encoding or decoding, such as:

UnicodeEncodeError: 'mbcs' codec can't decode characters in positionUnicodeDecodeError:' charmap' codec can't decode byte 0x90 in position

Most of the time, such mistakes do not provide enough information unless they are experienced in the field. You may ask why characters need to be encoded and decoded. We can answer this question from a simple interpretation of Unicode.

Based on the official python documentation, Unicode Unicode (Universal coded character set) is a specification designed to list each character used in human language and to provide a unique code for each character. Unicode specifications are constantly revised and updated to add new languages and symbols.

Therefore, encoding and decoding is a way to map characters from text to bytes, and vice versa. This allows them to be transferred between computers and used in daily life. The situation becomes more complicated when you have different sets of operating systems.

In addition, different languages have their own character sets that can only be displayed in specific fonts. To put it simply, it can be regarded as translating a foreign character into a character that the machine can understand. This article will explore some methods that can be used to deal with Unicode files in Python, starting with available patterns and standard coding.

Read and write files through the context manager

The safest way to open a file is to use the with statement through the context manager. It automatically closes the file to prevent any problems that may arise.

With open ('name.txt') as f: f.readlines ()

The default mode is' rt', which reads and sends files. You can write using the following code:

With open ('name.txt',') as f: f.write ('Hello worldview')

The above code will rewrite and truncate the file. In some cases, you may prefer to use the pattern'a 'rather than' w'. The following list shows the full modes available:

R: open for reading (default)

W: turn on writing and truncate the file first

X: open exclusive creation and fail if the file already exists

A: turn on writing and, if the file exists, add to the end of the file

B: binary mode

T: text mode (default)

+: open a disk file for update (read and write)

You can combine some patterns. As described in the original document (https://docs.python.org/3.5/library/functions.html#open), for binary read-write access, the mode 'wroomb' opens and truncates the file to 0 bytes.' Open files will not be truncated.

Standard coding in Python

To specify the encoding in Python, you only need to pass in another parameter during context manager initialization. Whenever you read and write a Unicode character, you need to specify it. The following example shows the correct way to add Unicode text to an existing file:

With open ('name.txt',' asides, encoding='utf8') as f: f.write ('Hello!')

If you're not sure which encoding to use, just type utf8 and check for errors. In most cases, UTF-8 is good enough for encoding and decoding characters. In some cases, however, different encodings are required.

Check the encoding type through Notepad++

It is generally more likely to use Notepad++ to view the contents of the file. If you open a file using Notepad++, you can see the type of encoding used in the lower-right corner of the user interface.

A sample file using UTF-8 encoding

The encoding can be modified through the coding menu, which accepts a large number of the most commonly used encodings.

Display the image of the drop-down menu when you click the encoding menu

If you have ever encountered the problem of not being able to convert a file to another encoding, or if you cannot read it even if some encoding is specified correctly, you can try the following methods. It's a little stupid, but the personal test works.

Create an empty text file with the encoding you want.

Copy everything from the original file.

Paste it into a new file and save it.

In most cases, this automatically converts all characters to the new encoding. Note that data loss may occur if characters cannot be converted according to the new encoding.

Handling characters in unknown encodings

If you encounter a situation where the encoding is not recognized and the characters are unknown, you can try to modify the error parameters to solve this problem:

With open ('name.txt',' ritual, encoding='utf8', errors='ignore') as f: f.readlines ()

Error parameters refer to how to handle encoding and decoding errors. Note that this parameter cannot be used in binary mode. Available error handlers are:

Strict: if there is a coding error, it will cause a ValueError exception. The default value of None has the same effect.

Ignore: ignore errors. Note that ignoring coding errors can result in data loss.

Replace: will replace the tag (e. G.'?') Insert a location with malformed data.

Surrogateescape: represents any incorrect bytes in the Unicode private use area as code points, ranging from U+DC80 to U+DCFF. When this error handler is used to write data, these private code points are converted to the same bytes. This is useful for handling files with unknown encodings.

Xmlcharrefreplace: supported only when writing to a file. Characters that are not supported by encoding will be replaced with a character reference of the appropriate extensible markup language & # nnn;.

Backslashreplace: replace malformed data with Python's backslash escape characters.

Namereplace: (supported only when writing) replaces unsupported\ N {...} escape characters.

Display Unicode characters in a command prompt

If you run the command prompt in the Windows operating system, there will be problems displaying Unicode characters in most cases, with garbled characters as shown in the following figure:

A command prompt that displays garbled characters

To solve this problem, you need to change the settings to the correct font.

Right-click the top menu and click Properties.

Click the font menu.

Change the font to the desired font that displays characters. For example, you can use italics to render Chinese characters.

Font properties of the command prompt

Open a file path with Unicode characters-applicable to read_csv through the pandas module

This part is a bit tricky, especially when using certain Python modules, such as pandas. Suppose you have the following non-English file paths:

File_path ='C:\ path\ to\ data Analysis\ data.csv'

Trying to read a file through read_csv will throw an error because the file path contains Unicode characters. Using the built-in open () function in Python is not the problem. To solve this problem, you need to open the file and pass it to the read_csv function:

With open (file_path, 'ringing, encoding='utf-8') as f: df = pd.read_csv (f, encoding='utf-8') Thank you for reading, this is the content of "what is the method of python dealing with Unicode files". After the study of this article, I believe you have a deeper understanding of what the method of python is to deal with Unicode files, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.