Shulou(Shulou.com)06/02 Report--

Python iterates through the iterator's automatic chained code to deal with data. I believe many inexperienced people don't know what to do about it. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Python traversal iterator automatically chained data processing

Pytorch.utils.data is compatible with iterative data training processing, and can be used in dataloader to improve training efficiency: avoiding insufficient memory overflow with iterators, and making data reading and utilization more efficient with chained processing (comparable to resource control of operating systems)

Following the above, use the iterator chain to process the data, and perform the mount preprocessing method in the _ _ iter__ method of the Process class, which can nest the multi-layer processing method, similar to the KoaJs onion model, and automatically execute the preprocessing method to return the processed data during the for loop.

Analyze the order of input data in the following example: travel-> deep-> shuffle-> sort-> batch. Actually, due to the existence of nested loops or setting cache, the data flow will change, as shown in the figure below.

From torch.utils.data import IterableDataset#... import randomclass Process (IterableDataset): def _ _ init__ (self, data, f): self.data = data # binding handler self.f = f def _ iter__ (self): # for loop traversal Returns an iterator object return self.f (iter (self.data)) a = ['a0cycles,' a1levels, 'a2layers,' a3levels, 'a4bands,' a5cycles, 'a6cycles,' a7cycles, 'a8bands,' a9'] b = ['b0cycles,' b1bands, 'b2domains,' b3cycles, 'b4bands,' b5bands, 'b6bands,' b7cycles, 'b8') 'b9'] c = [' c0,'c1,'c2,'c3,'c4,'c5,'c6,'c7,'c8,'c9'] # data = [[j + str (I) for i in range (10)] for j in ['axie paper,' c']] data = [a, b, c] def travel (d): for i in d: # print ('travel') I) yield idef deep (d): for arr in d: for item in arr: yield itemdef shuffle (d, sf_size=5): buf = [] for i in d: buf.append (I) if len (buf) > = sf_size: random.shuffle (buf) for j in buf: # print ('shuffle' J) yield j buf = [] for k in buf: yield kdef sort (d): buf = [] for i in d: buf.append (I) if len (buf) > = 3: for i in buf: # print ('sort' I) yield i buf = [] for k in buf: yield kdef batch (d): buf = [] for i in d: buf.append (I) if len (buf) > = 16: for i in buf: # print ('batch' I) yield i buf = [] # multiple preprocessing steps for training data dataset = Process (data, travel) dataset = Process (dataset, deep) dataset = Process (dataset, shuffle) dataset = Process (dataset, sort) train_dataset = Process (p, batch) # you can test for i in p: print (I, 'train') # train_data_loader = DataLoader (train_dataset,num_workers=args.num_workers) here Prefetch_factor=args.prefetch) # train (model, train_data_loader)

The data stream direction can be constructed from the above: batch (iter (sort (iter) (iter (iter) (deep (travel (iter (d)

According to the partial process of data stream extraction, the sequence diagram is drawn as follows:

Attached: python manual traversal iterator

Want to iterate through all the elements in an iterable object, but don't want to use for loops

To traverse iterable objects manually, use the next () function and catch StopIteration exceptions in the code. For example, the following example reads all lines in a file manually

Def manual_iter (): with open ('/ etc/passwd') as f: try: while True: line = next (f) print (line, end='') except StopIteration: pass

Generally speaking, StopIteration is used to indicate the end of an iteration. However, if you manually use the next () function demonstrated above, you can also mark the end by returning a specified value, such as None. Here is an example:

With open ('/ etc/passwd') as f: while True: line = next (f) if line is None: break print (line, end='')

In most cases, we use for loop statements to iterate through an iterable object. However, occasionally there is a need for more precise control of iterations, so it is particularly important to understand the underlying iteration mechanism. The following interaction example shows us the basic details of what happened during the iteration:

> items = [1,2,3] > > # Get the iterator > it = iter (items) # Invokes items.__iter__ () > > # Run the iterator > next (it) # Invokes it.__next__ () 1 > next (it) 2 > > next (it) 3 > next (it) Traceback (most recent call last): File ", line 1, in StopIteration > after reading the above, have you mastered how to write the code for python traversing iterator automatic chain processing data? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.