Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to understand Python iteration objects and iterators and generators. The content of the article is of high quality, so Xiaobian shares it with you as a reference. I hope you have a certain understanding of related knowledge after reading this article.

When understanding Python's data structure, container, iterable, iterator, generator, list/set/dictionary inference are so many concepts that beginners are inevitably confused. I will use an article to try to clarify these concepts and their relationships.

container

A container is a data structure that organizes multiple elements together. The elements in the container can be iteratively obtained one by one. The keyword in, not in can be used to determine whether the elements are contained in the container. Usually this type of data structure stores all elements in memory (there are some exceptions, not all elements are stored in memory, such as iterators and generator objects). In Python, common container objects are:

list, deque, ….

set, frozensets, ….

dict, defaultdict, OrderedDict, Counter, ….

tuple, namedtuple, …

str

A container is easier to understand because you can think of it as a box, a house, a cabinet, and you can stuff anything into it. Technically, when it can be used to ask whether an element is contained in it, then the object can be considered a container, such as list, set, tuples are container objects:

>>> assert 1 in [1, 2, 3] # lists >>> assert 4 not in [1, 2, 3] >>> assert 1 in {1, 2, 3} # sets >>> assert 4 not in {1, 2, 3} >>> assert 1 in (1, 2, 3) # tuples >>> assert 4 not in (1, 2, 3)

Ask if an element has a key in dict:

>>> d = {1: 'foo', 2: 'bar', 3: ' qux'}>> assert 1 in d >>> assert 'foo' not in d # 'foo' is not an element in dict

Ask if a substring is in string:

>>> s = 'foobar' >>> assert 'b' in s >>> assert 'x' not in s >>> assert 'foo' in s

Although most containers provide some way to retrieve each of their elements, this is not a capability provided by the container itself, but rather an iterable object that gives the container this capability. Of course, not all containers are iterable, such as: Bloom filter, although Bloom filter can be used to detect whether an element is contained in a container, it cannot obtain every value from the container, because Bloom filter does not store the element in the container at all, but maps it to a value stored in an array through a hash function.

iterable object

As mentioned earlier, many containers are iterable objects, and many more are also iterable objects, such as open files, sockets, and so on. Any object that returns an iterator can be called an iterable object. This may sound confusing, but it doesn't matter. Let's look at an example:

>>> x = [1, 2, 3] >>> y = iter(x) >>> z = iter(x) >>> next(y) 1 >>> next(y) 2 >>> next(z) 1 >>> type(x) >>> type(y)

Here x is an iterable object, iterable object and container is a common name, does not refer to a specific data type, list is iterable object, dict is iterable object, set is iterable object. y and z are two independent iterators that hold a state inside that records the position of the current iteration so that the correct element can be obtained in the next iteration. Iterators have a specific iterator type, such as list_iterator, set_iterator. Iterable objects implement the__iter__method, which returns an iterator object.

When running code:

x = [1, 2, 3] for elem in x: ...

Actual performance was as follows:

Decompile this code, you can see the interpreter explicitly call GET_ITER instruction, equivalent to calling iter(x), FOR_ITER instruction is to call the next() method, constantly get the next element in the iterator, but you can not see directly from the instruction, because it has been optimized by the interpreter.

>>> import dis >>> x = [1, 2, 3] >>> dis.dis('for _ in x: pass') 1 0 SETUP_LOOP 14 (to 17) 3 LOAD_NAME 0 (x) 6 GET_ITER >> 7 FOR_ITER 6 (to 16) 10 STORE_NAME 1 (_) 13 JUMP_ABSOLUTE 7 >> 16 POP_BLOCK >> 17 LOAD_CONST 0 (None) 20 RETURN_VALUE

iterator

So what iterator? It is an object with state that returns the next value in the container when you call the next() method, any object that implements__iter__and__next__()(Python2 implements the next() method) is an iterator,__iter__returns the iterator itself,__next__returns the next value in the container, and throws a StopIteration exception if there are no more elements in the container, it does not matter how they are implemented.

So, iterators are objects that implement the factory pattern and return to you each time you ask for the next value. There are many examples of iterators, such as itertools function returns iterator objects.

Generate *** sequence:

>>> from itertools import count >>> counter = count(start=13) >>> next(counter) 13 >>> next(counter) 14

Generating *** sequences from a finite sequence:

>>> from itertools import cycle >>> colors = cycle(['red', 'white', 'blue']) >>> next(colors) 'red' >>> next(colors) 'white' >>> next(colors) 'blue' >>> next(colors) 'red'

Generating finite sequences from sequences of ***:

>>> from itertools import islice >>> colors = cycle(['red', 'white', 'blue']) # infinite >>> limited = islice(colors, 0, 4) # finite >>> for x in limited: ... print(x) red white blue red

In order to more intuitively feel the execution process inside the iterator, we define a custom iterator, taking the Fibonacci sequence as an example:

class Fib: def __init__(self): self.prev = 0 self.curr = 1 def __iter__(self): return self def __next__(self): value = self.curr self.curr += self.prev self.prev = value return value >>> f = Fib() >>> list(islice(f, 0, 10)) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Fib is both an iterable object (because it implements the__iter__method) and an iterator (because it implements the__next__method). The instance variables prev and curr maintain the state inside the iterator. Do two things each time you call the next() method:

Modify state for next call to next() method

Generate a return result for the current call

An iterator is like a lazy loading factory, waiting until someone needs it to generate a value to return, and sleeping until the next call.

generator

Generators are one of the most attractive features of Python. Generators are a special kind of iterator, but this kind of iterator is more elegant. It doesn't need to write the__iter__() and__next__() methods like the above class, just a yiled keyword. Generators must be iterators (and vice versa is not true), so any generator also generates values in a lazy loading pattern. An example of using generators to implement Fibonacci sequences is:

def fib(): prev, curr = 0, 1 while True: yield curr prev, curr = curr, curr + prev >>> f = fib() >>> list(islice(f, 0, 10)) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

fib is an ordinary python function, which is special in that there is no return keyword in the function body, and the return value of the function is a generator object. When executing f=fib() returns a generator object, the code in the function body is not executed, only when next is explicitly or implicitly called.

Generators are a very powerful programming construct in Python that can write streaming code with fewer intermediate variables, save more memory and CPU than other container objects, and of course can implement similar functionality with less code. Now you can start refactoring your code, if you see something like:

def something(): result = [] for ... in ...: result.append(x) return result

can be replaced by generator functions:

def iter_something(): for ... in ...: yield x

generator expression

A generator expression is a generator version of list pushdown that looks like a list deductive, but returns a generator object instead of a list object.

>>> a = (x*x for x in range(10)) >>> a >>> sum(a) 285

A container is a collection of elements, str, list, set, dict, file, sockets objects can be considered containers, containers can be iterated (used in for, while, etc.), so they are called iterable objects.

Iterable objects implement the__iter__method, which returns an iterator object.

The iterator holds an internal state field that records the return value of the next iteration. It implements the__next__and__iter__methods. The iterator does not load all the elements into memory at once, but generates the return result only when needed.

A generator is a special iterator whose return value is not returned by return but by yield.

About how to understand Python iteration objects and iterators and generators to share here, I hope the above content can be of some help to everyone, you can learn more knowledge. If you think the article is good, you can share it so that more people can see it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.