Shulou(Shulou.com)06/01 Report--

How to make itertools.tee thread safe, I believe that many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

We mentioned that itertools.tee is not thread-safe and give an example, as shown in the following figure:

Running a split generator object in two threads at the same time will result in an error.

Now, you want to look at the source code of itertools.tee, but you will find that in PyCharm, its source code is shown in the following figure:

This is because, in CPython, the underlying itertools.tee is implemented in C, so you can't see its source code in PyCharm. But you can find its implementation algorithm by reading the Modules/itertoolsmodule.c file [1] in Python's source code.

The core of the problem is in the two pieces of code shown in the following figure:

It doesn't matter if you don't understand it, according to the simplified version of the implementation using Python in my last article, it's enough to help you understand.

We use a simplified version to explain where the thread is unsafe:

Def generator ():

For i in range (3):

Yield f'I am your {I} grandpa'

Def split (g):

Value_list_1 = []

Value_list_2 = []

Def wrap (queue):

While True:

If not queue:

Try:

Value = next (g)

Except StopIteration:

Return

Value_list_1.append (value)

Value_list_2.append (value)

Yield queue.pop (0)

Grou1 = wrap (value_list_1)

Gambi2 = wrap (value_list_2)

Return g_1, g_2

G = generator ()

Group1, Garg2 = split (g)

For value in g_1:

Print (value)

For value in g_2:

Print (value)

When two threads run to if not queue at the same time, they find that their current queues are empty, so they go into value = next (g) to get the next value. Where thread An enters the first few milliseconds. Then thread B enters value = next (g). However, because next (g) in thread An is running and not finished, thread B runs again, which leads to the occurrence of an error. In Python, generators are not thread-safe.

So how do you make multiple generators split out of itertools.tee run in multithreading? The key factor is to make the line value = next (g) run only one thread at a time. So we can do this by adding locks.

Import itertools

From threading import Lock

Class KingnameTee:

Def _ _ init__ (self, tee_obj, lock):

Self.tee_obj = tee_obj

Self.lock = lock

Def _ iter__ (self):

Return self

Def _ next__ (self):

With self.lock:

Return next (self.tee_obj)

Def _ copy__ (self):

Return KingnameTee (self.tee_obj.__copy__ (), self.lock)

Def safe_tee (iterable, nasty 2):

"" tuple of n independent thread-safe iterators "

Lock = Lock ()

Return tuple (KingnameTee (tee_obj, lock) for tee_obj in itertools.tee (iterable, n))

Let's see how it works:

Multithreading runs perfectly.

After reading the above, have you mastered how to make itertools.tee thread safe? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.