Shulou(Shulou.com)06/02 Report--

How to deeply understand the ThreadLocal variable in Python, I believe that many inexperienced people are at a loss about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

In the previous article, we saw the simple use of the ThreadLocal variable. The middle part analyzed the implementation of ThreadLocal in python, but the story is not over yet. In this article, let's look at the design of ThreadLocal in Werkzeug.

Werkzeug as a WSGI tool library, due to some considerations, does not directly use python built-in ThreadLocal classes, but implements a series of Local classes. It includes simple Local, as well as LocalStack,LocalManager and LocalProxy implemented on this basis. Next, let's take a look at how these classes are used, the original intention of the design, and the specific implementation techniques.

The Design of Local Class

The designers of Werkzeug believe that python's native ThreadLocal does not meet the requirements, mainly for the following two reasons:

Werkzeug mainly uses "ThreadLocal" to meet the concurrency requirements, and the ThreadLocal that comes with python can only achieve thread-based concurrency. There are many other ways of concurrency in python, such as the common greenlet, so you need to implement a Local object that can support it.

WSGI does not guarantee that a new thread will be generated each time to process the request, that is, the thread can be reused (a thread pool can be maintained to handle the request). In this way, if werkzeug uses the ThreadLocal that comes with python, a thread that is "dirty (with data related to previously processed requests) will be used to process new requests."

To solve these two problems, the Local class is implemented in werkzeug. The Local object enables data isolation between threads and cooperators. In addition, it also supports cleaning up data under a thread or co-program (so that you can clean up the corresponding data after one request is processed, and then wait for the next request to arrive).

How to achieve it? the idea is actually very simple. We have mentioned in the article "ThreadLocal variable (above) in Python" that is to create a global dictionary, and then take the thread (or co-program) identifier as key, and the local data of the corresponding thread (or co-program) as value. Here, werkzeug is implemented according to the above idea, but using some dark magic of python, * * provides users with a clear and simple interface.

Concrete realization

The implementation of the Local class is analyzed in werkzeug.local with the 8a84b62 version of the code. Through the understanding of ThreadLocal in the first two articles, we already know the characteristics and usage of Local objects. So instead of giving examples of the use of Local objects, let's just look at the code.

Class Local (object): _ _ slots__ = ('_ _ storage__','_ _ ident_func__') def _ _ init__ (self): object.__setattr__ (self,'_ storage__', {}) object.__setattr__ (self,'_ ident_func__', get_ident).

Because there may be a large number of Local objects, in order to save the space occupied by Local objects, we use _ _ slots__ to write down the properties that Local can have:

The _ _ storage__: value is a dictionary, which is used to hold the actual data and is initialized to empty.

The _ _ ident_func__: value is a function used to find the identifier of the current thread or co-program.

Because the actual data of the Local object is stored in _ _ storage__, the operation on the Local property is actually an operation on _ _ storage__. For obtaining attributes, the magic method _ _ getattr__ is used to intercept the acquisition of attributes other than _ _ storage__ and _ _ ident_func__, and direct them to the data of the current thread or co-program stored by _ _ storage__. For set or del of attribute values, they are implemented with _ _ setattr__ and _ _ setattr__, respectively (for an introduction to these magic methods, see attribute control). The key code is as follows:

Def _ _ getattr__ (self, name): try: return self.__storage__ [self. _ _ ident_func__ ()] [name] except KeyError: raise AttributeError (name) def _ setattr__ (self, name Value): ident = self.__ident_func__ () storage = self.__storage__ try: storage [ident] [name] = value except KeyError: storage [ident] = {name: value} def _ _ delattr__ (self, name): try: del self.__storage__ [self. _ ident_func__ ()] [name] except KeyError: raise AttributeError (name)

Suppose we have N threads or cooperators whose ID is 1BI 2,..., N, each of which has its own local data stored in the Local object, then the content of the Local object is shown below:

In addition, the Local class provides a _ _ release_local__ method to release data saved by the current thread or co-program.

Local extension interface

Werkzeug implements LocalStack and LocalManager on the basis of Local to provide more friendly interface support.

LocalStack

LocalStack implements a thread (or co-program) independent stack structure by encapsulating Local. There are specific usage methods in the comments. A simple example is as follows:

Ls = LocalStack () ls.push (12) print ls.top # 12 print ls._local.__storage__ # {140735190843392: {'stack': [12]}

The implementation of LocalStack is interesting, it takes a Local object as its own property _ local, and then defines interface push, pop, and top methods for corresponding stack operations. Here, the list of _ local.__storage__._local.__ident_func__ () is used to simulate the stack structure. In the interfaces push, pop and top, you can simulate the operation of the stack by manipulating the list. It is important to note that when getting the list inside the interface function, it is not as complicated as the boldface above. You can directly use the getattr () method of _ local. Taking the push function as an example, the implementation is as follows:

Def push (self, obj): "Pushes a new item to the stack" rv = getattr (self._local, 'stack', None) if rv is None: self._local.stack = rv = [] rv.append (obj) return rv

The implementations of pop and top are similar to normal stacks in that they operate on the list stack = getattr (self._local, 'stack', None). In addition, LocalStack also allows us to customize _ _ ident_func__,. Here, a descriptor is generated with the built-in function property, which encapsulates the get and set operations of _ _ ident_func__, and provides an attribute value of _ _ ident_func__ as the interface. The specific code is as follows:

Def _ get__ident_func__ (self): return self._local.__ident_func__ def _ set__ident_func__ (self, value): object.__setattr__ (self._local,'_ _ ident_func__', value) _ _ ident_func__ = property (_ get__ident_func__, _ set__ident_func__) del _ get__ident_func__, _ set__ident_func__

LocalManager

Local and LocalStack are both thread-independent or co-program independent single objects, so we often need a thread or co-program-independent container to organize multiple Local or LocalStack objects (just as we use a list to organize multiple int or string types).

Werkzeug implements LocalManager, which stores managed Local or LocalStack objects through a property of type list locals, and provides cleanup methods to release all Local objects. The main interface of LocalManager in Werkzeug is the decorator method make_middleware. The code is as follows:

Def make_middleware (self, app): "Wrap a WSGI application so that cleaning up happens after request end." Def application (environ, start_response): return ClosingIterator (app (environ, start_response), self.cleanup) return application

The decorator registers the callback function cleanup, and when a thread (or co-program) finishes processing the request, it calls cleanup to clean up the Local or LocalStack objects it manages (the implementation of ClosingIterator is in werkzeug.wsgi). Here is a simple example of using LocalManager:

From werkzeug.local import Local, LocalManager local = Local () local_2 = Local () local_manager = LocalManager ([local, local2]) def application (environ, start_response): local.request = request = Request (environ)... # application automatically cleans up the contents of local_manager application = local_manager.make_middleware (application)

With LocalManager's make_middleware, we can clear all Local or LocalStack objects after a thread (co-program) has finished processing one request, so that the thread can process another request. At this point, the second problem mentioned at the beginning of the article can be solved. Werkzeug.local also implements a LocalProxy as a proxy for Local objects, which is also worth learning.

Both the Python standard library and Werkzeug use a lot of python's dark magic in their implementation, but in the end they provide users with very friendly interfaces. As a WSGI toolset, Werkzeug provides an improved version and a series of packages to solve the specific usage problems in Web development. I have to say that werkzeug's code is very readable and the comments are very well written, so it is recommended to read the source code.

After reading the above, have you mastered how to deeply understand the ThreadLocal variable in Python? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.