Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what problems should Python pay attention to when calculating memory". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. What is the calculation?

Let's first take a look at the list objects:

As shown in the figure, the results of calculating the an and b lists separately are 36 and 48, and then when they are used as child elements of the c list, the calculation result of the list is only 36. (PS: I use a 32-bit interpreter)

If you do not use reference, but write the sublist directly, for example, "d = [[1mag2], [1mag2], [1pjin2]", then the result of calculating the d list is still 36, because the sublist is an independent object, and their id is stored in the d list.

In other words, when the getsizeof () method calculates the size of the list, the result is related to the number of elements, but has nothing to do with the size of the element itself.

Let's take a look at the dictionary example:

It is obvious that all the memory actually occupied by the three dictionaries cannot be equal, but the getsizeof () method gives the same result, which means that it only cares about the number of keys, not what the actual key-value pair is, similar to a list.

2. "shallow calculation" and other problems

There is a concept called "shallow copy", which means that the copy () method copies only the memory address of the reference object, not the actual reference object. By analogy to this concept, we can think of getsizeof () as a kind of "shallow computing".

Shallow computing does not care about real objects, so the result of its calculation is just an illusion. This is a problem worth paying attention to, but it is not enough to pay attention to it. We can also think about the following questions diversely:

What is the underlying implementation of the shallow computing method?

Why does getsizeof () adopt the "shallow computing" approach?

On the first question, the getsizeof (x) method actually calls the _ _ sizeof__ () magic method of the x object, which is implemented through the CPython interpreter for built-in objects.

I looked up this article, "memory usage of objects in Python (1)", which analyzed the CPython source code, and finally located the core code in this paragraph:

/ * longobject.c*/ static Py_ssize_t int___sizeof___impl (PyObject * self) {Py_ssize_t res; res = offsetof (PyLongObject, ob_digit) + Py_ABS (Py_SIZE (self)) * sizeof (digit); return res;}

I don't understand this code, but what I can know is that when calculating the size of a Python object, it is only related to the properties of the object's structure, and there is no further "depth calculation".

For this implementation of CPython, we can notice the difference in two levels:

Byte size: the int type takes up only 4 bytes in C, but in Python, int is actually encapsulated as an object, so the size of the object structure is included when calculating its size. In the 32-bit interpreter, the result of getsizeof (1) is 14 bytes, which is larger than the 4 bytes of the number itself.

Byte reduction: for relatively complex objects, such as lists and dictionaries, this calculation mechanism produces a result that is less than the actual memory footprint because it does not accumulate the amount of internal elements.

As a result, I have an immature guess: based on the "everything is an object" design principle, int and other basic C data types are "shelled" in Python, so a method is needed to calculate their size, that is, getsizeof ().

The official document says "All built-in objects will return correct results" [1], which should refer to simple objects such as numbers, strings, and Boolean values. However, it does not include types such as lists, tuples, and dictionaries that have internal reference relationships.

Why not extend it to all built-in types? I have not found any explanation in this respect. If there are any students who know about it, please let me know.

3. "Deep Computing" and other problems

Corresponding to "shallow computing", we can define a "deep computing". For the previous two examples, Deep Computing should traverse each internal element and possible child elements, accumulate their bytes, and finally calculate the total memory size.

So, the problems we should pay attention to are as follows:

Is there a method / implementation of "deep computing"?

What should be paid attention to when implementing "deep computing"?

There is an age-old question on the Stackoverflow website, "How do I determine the size of an object in Python?" [2], which actually asks how to implement "deep computing".

There are different developers contributed to two projects: pympler and pysize: the first project has been released on Pypi, you can "pip install pympler" installation; the second project is a bad end, the author did not release to Pypi (Note: Pypi already has a pysize library, is used for format conversion, do not confuse), but you can get its source code on Github.

For the previous two examples, we can test these two projects separately:

Looking at the numbers alone, pympler does seem to be much more reasonable than getsizeof ().

If you look at pysize, you can see the test results directly (the process of getting its source code is brief):

64 118 190 206 300281 30281

As you can see, it is slightly smaller than the result of pympler calculation. In terms of the integrity, usage and community contributor size of the two projects, pympler's results seem to be more credible.

So, how are they realized respectively? How did that slight difference come about? What can we learn from their implementation?

The pysize project is simple, with only one core approach:

Def get_size (obj, seen=None): "" Recursively finds sizeof objects in bytes "size = sys.getsizeof (obj) if seen is None: seen= set () obj_id = id (obj) if obj_id in seen: return 0 # Important mark as seen * before* entering recursion to gracefully handle # self-referential objects seen.add (obj_id) if hasattr (obj '_ dict__'): for cls in obj.__class__.__mro__: if' _ dict__' in cls.__dict__: d = cls.__dict__ ['_ dict__'] if inspect.isgetsetdescriptor (d) or inspect.ismemberdescriptor (d): size + = get_size (obj.__dict__) Seen) break if isinstance (obj, dict): size + = sum ((get_size (v, seen) for v in obj.values ()) size + = sum ((get_size (k, seen) for k in obj.keys ()) elif hasattr (obj,'_ iter__') and not isinstance (obj, (str, bytes, bytearray)): size + = sum (get_size (I) Seen) for i in obj)) if hasattr (obj,'_ slots__'): # can have _ slots__ with _ dict__ size + = sum (get_size (getattr (obj, s), seen) for s in obj.__slots__ if hasattr (obj, s) return size

Except for the part of judging _ _ dict__ and _ _ slots__ attributes (for class objects), it is mainly the recursive calculation of dictionary types and iterable objects (except string, bytes, bytearray), and the logic is not complicated.

Take the list [1Jing 2] as an example, which uses sys.getsizeof () to calculate 36 bytes, then calculates the two internal elements to get 14.2 million 28 bytes, and finally adds up to 64 bytes.

By contrast, pympler has a lot more to consider, and the entry is here:

Def asizeof (self, * objs, * * opts):''Return the combined sizeof the given objects (with modified options, see method * * set**). If opts: self.set (* * opts) self.exclude_refs (* objs) # skip refs to objs return sum (self._sizer (o, 0,0, None) for o in objs)

It can accept multiple parameters and merge them with the sum () method. So the core calculation method is actually _ sizer (). But the code is complex, circling around like a maze:

Def _ sizer (self, obj, pid, deep, sized): # MCCABE 19''Size an object, recursively.'' S, f, I = 0,0, id (obj) if i not in self._seen: self._ seen [I] = 1 elif deep or self._seen [I]: # skip obj if seen before # or if ref of a given obj self._seen.again (I) if sized: s = sized (s, f) Name=self._nameof (obj)) self.exclude_objs (s) return s # zero else: # deep = = seen [I] = 0 self._seen.again (I) try: K, rs = _ objkey (obj) [] if k in self._excl_d: self._excl_ d [k] + = 1 else: v = _ typedefs.get (k, None) if not v: # new typedef _ typedefs [k] = v = _ typedef (obj, derive=self._derive_ Frames=self._frames_ Infer=self._infer_) if (v.both or self._code_) and v.kind is not self._ign_d: # cat note: calculate flat size s = f = v.flat (obj) here Self._mask) # flat size if self._profile: # profile based on * flat* size self._prof (k) .update (obj, s) # recurse, but not for nested modules if v.refs and deep

< self._limit_ \ and not (deep and ismodule(obj)): # add sizes of referents z, d = self._sizer, deep + 1 if sized and deep < self._detail_: # use named referents self.exclude_objs(rs) for o in v.refs(obj, True): if isinstance(o, _NamedRef): r = z(o.ref, i, d, sized) r.name = o.name else: r = z(o, i, d, sized) r.name = self._nameof(o) rs.append(r) s += r.size else: # just size and accumulate for o in v.refs(obj, False): # 猫注：这里递归计算 item size s += z(o, i, d, None) # deepest recursion reached if self._depth < d: self._depth = d if self._stats_ and s >

Self._above_ > 0: # rank based on * total* size self._rank (k, obj, s, deep, pid) except RuntimeError: # XXX RecursionLimitExceeded: self._missed + = 1 if not deep: self._total + = s # accumulate if sized: s = sized (s, f, name=self._nameof (obj)) Refs=rs) self.exclude_objs (s) return s

Its core logic is to divide the size of each object into two parts: flat size and item size.

The logic for calculating flat size is as follows:

Def flat (self, obj, mask=0): 'Return the aligned flat size.'' S = self.base if self.leng and self.item > 0: # include items s + = self.leng (obj) * self.item # workaround sys.getsizeof (and numpy?) Bug... Some # types are incorrectly sized in some Python versions # (note, isinstance (obj, ()) = = False) # cat note: if not sys.getsizeof, use the above logic, if possible, use the following logic if not isinstance (obj, _ getsizeof_excls): s = _ getsizeof (obj, s) if mask: # align s = (s + mask) & ~ mask return s

The mask that appears here is for byte alignment, and the default value is 7, which means that the formula is aligned by 8 bytes. For the [1J2] list, it calculates (36-7) & ~ 7-40 bytes. Similarly, for a single item, for example, the number 1 in the list is equal to 14, while pympler is calculated as an aligned number 16, so it adds up to 40'16'16'72 bytes. This explains why the result of pympler is larger than that of pysize.

Byte alignment is generally implemented by specific compilers, and different compilers have different strategies. Theoretically, Python should not care about such low-level details, and the built-in getsizeof () method does not take into account byte alignment.

Without considering other edge cases, it can be considered that pympler is based on getsizeof (), considering not only the size of traversing reference objects, but also the problem of byte alignment in actual storage, so it will seem closer to reality.

This is the end of the content of "what problems should Python pay attention to when calculating memory"? thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.