Shulou(Shulou.com)06/02 Report--

This article focuses on "what are Python threads and locks". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what Python threads and locks are.

Overview

Threads and locks are the underlying software definitions of the hardware and therefore contain the simplest possible concurrency model. It forms the basis of other concurrency abstractions built on top of it, so it's important to understand this. However, it is difficult or impossible to build reliable, scalable systems directly on these foundations.

Although most languages support threads and locks, CPython still uses global interpreter locks to prevent threads from accessing shared memory at the same time because CPython's memory management is non-thread safe. Although blocking operations occur outside the GIL and may improve performance, the system call overhead required for thread switching may degrade performance. This means that threads in Python are mainly used in scenarios where I am limited, rather than in scenarios where CPU is restricted.

As an aside, I mentioned CPython because other implementations of the Python specification, such as Jython, do not have global interpreter locks. However, these implementations are not widely used in practice because *: no one wants to support multi-Python implementations unless they have to; second: they are not rich enough; and third: due to the need to natively support C Python + extended API,Python language definition is tightly coupled with C API,Python +, it is not so much a technical specification as a reference implementation.

Python supports threads directly through the high-level module threading module and the low-level module _ thread. For more information about how these modules work, you can get the source code online.

Introduction

The typical single-threaded "Hello World" execution in Python is very simple:

Multithreaded simulation is not much different:

Based on my limited number of tests, the above script runs as follows:

I used get_ident () to print the "thread identifier" (a magic value that makes no sense except to disambiguate different threads at run time). You can see how thread identifiers are different in some cases and the same in others. The same thread identifier does not mean that it is still working on the same thread, but Python reuses it if the work does not overlap and different thread identifiers are not required.

Pitfalls: timing and consistency

If you swap thread identifiers with thread names with threading.current_thread (). GetName (), you may get ordered results, probably because each thread uses the same function and source path, so the delay difference between each thread is negligible, second only to the delay of the interpreter. However, this does not mean that orderly execution is guaranteed; this is an example of "Python Programming" on WikiBooks, where the creation of each thread and the execution of each thread have a significantly different timing:

The following results are the output of the same sample run:

This log indicates that thread creation / execution is interlaced. As the variability of added functionality increases, these results will become more and more unpredictable as the timing between thread creation and execution becomes more and more inconsistent. But the principle is still the same; consistent behavior cannot be guaranteed when using multiple threads.

Trap: accessing shared memory

This can lead to incorrect behavior when different threads access shared memory. You can extend this example to view race conditions when counting with multiple threads:

This produces the following output when the sample is run:

This result varies depending on the number of threads created, but you can see how much the result 28 is different from the expected value of 100. Counter (). Count is not thread-safe, which is demonstrated here (if you have a different machine from mine, you may get different results from 28). If you encounter competitive conditions and do not have enough logging, it may be difficult to find the relevant parts of the code.

Trap: deadlock

When two agents try to acquire the same area of shared memory, a deadlock eventually occurs. When dealing with low-level abstractions of threads and locks, the only solution is to ensure that each agent has a way to manage its locks correctly, or has an overall specification for lock coordination. For example, the issue of dining philosophy emphasizes the importance of process synchronization. Rosetta Code's dining philosophy python solution solves this synchronization problem: if you (the agent) can't get the two forks in time, you can release any forks you already have so that another agent can get both forks at the same time.

This approach does not rule out other locking methods, such as locking order, or system design that involves process synchronization, such as a producer-consumer model that uses semaphores, but it may not be as common in Python as in other languages.

Pitfalls: alien methods and dependencies

If you want to apply multithreading in Python applications, then you want to ensure the correctness of the entire stack, you must manually verify the thread model that verifies thread safety and dependencies. Some dependencies, such as redis, designed for use in an enterprise-class multiservice environment, can first consider their concurrency model at design time (see Hacker News antirez's comments on the multithreaded version of redis). Some dependencies may not; when using boto2, I may encounter a deadlock when downloading files from S3 in parallel using multiprocessing.pool.Pool in parallel, which requires rewriting a function. As a result, another dependency difficulty arises; they cannot be assimilated, which means that if you do not verify all the dependencies that will be used before your application uses the dependency model, then when you try to add dependencies for a particular purpose, you may fall into a dead end of the project.

Multithreaded logging

If you choose to use the native threading model in Python, you may be pleasantly surprised to find that the logging module is not only thread-safe, but also supports logging from any particular thread or process (example in the logging manual). Then, the difficulty is where exceptions are more likely to be triggered in your application, how this affects your threading model and ensures reliable logging around these code snippets. Adding logs to your application can cause significant delays, as pylint informs you through the warning module logging-lazy-interpolation, which can also cause difficulties for your threading model.

Concurrent.futures

When writing this article, I found that Python

The multiprocessing.pool.ThreadPool implementation was never recorded or tested because it was never completed, which made me feel very unhappy. It is still the same in Python3.7 because it appears in the source code of the GitHub image. Given the ubiquity of global interpreter locks and the fact that future concurrent programs are mainly parallel Imax O-related work, it may make more sense to use new asynchronous patterns like concurrent.futures.Executor or similar provided in Python3.x because they are more comprehensive. I haven't used this module but I don't think it will cause significant performance loss compared to multiprocessing.

At this point, I believe you have a deeper understanding of "what Python threads and locks are". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.