Shulou(Shulou.com)06/03 Report--

Lack of memory is a common problem in project development, and my team and I also encountered this problem in a previous project, where our project needed to store and process a fairly large dynamic list. Testers often complain to me about lack of memory. But in the end, we solved the problem by adding a simple line of code.

The result is shown in the figure:

I will explain how it works below.

Take a simple "learning" example-create a DataItem class that defines some personal information properties, such as name, age, and address.

Quiz-how much memory does such an object take up?

First, let's try the following test scenario:

D1 = DataItem ("Alex", 42, "-")

Print ("sys.getsizeof (D1):", sys.getsizeof (D1))

The answer is 56 bytes. It looks small, and the result is satisfactory.

However, let's examine another object with more data:

D2 = DataItem ("Boris", 24, "In the middle of nowhere")

Print ("sys.getsizeof (D2):", sys.getsizeof (D2))

The answer is still 56. This makes us understand that the result is not entirely correct.

Our intuition is right, and the problem is not that simple. Python is a very flexible language with dynamic typing that stores a lot of extra data as it works. The extra data itself takes up a lot of memory.

For example, sys.getsizeof ("") returns 33, yes, as many as 33 bytes per blank line! And sys.getsizeof (1) will return 24-24 bytes for this number (I suggest that C programmers click to finish reading now so as not to lose faith in the beauty of Python).

For more complex elements, such as dictionaries, sys.getsizeof (dict ()) returns 272 bytes, which is just an empty dictionary. So much for example, but the facts are clear, not to mention that RAM manufacturers also need to sell their chips.

Now, let's get back to our DataItem class and "quiz" problem.

How much memory does this class take up?

First, we will output the entire contents of the class at a lower level:

Def dump (obj):

For attr in dir (obj):

Print ("obj.%s =% r"% (attr, getattr (obj, attr)

This function will show what is hidden under the invisibility cloak so that all Python functions (types, inheritance, and other packages) can be run.

The results are impressive:

Current location: technology sharing > Technical reference > text

There is no memory problem that cannot be solved by a single line of Python code.

2018-12-18 10:24:56 | Editor: hely | View: 96 | comment: 0

Lack of memory is a common problem in project development, and my team and I also encountered this problem in a previous project, where our project needed to store and process a fairly large dynamic list. Testers often complain to me about lack of memory. But in the end, we solved the problem by adding a simple line of code.

Lack of memory is a common problem in project development, and my team and I also encountered this problem in a previous project, where our project needed to store and process a fairly large dynamic list. Testers often complain to me about lack of memory. But in the end, we solved the problem by adding a simple line of code.

The result is shown in the figure:

I will explain how it works below.

Take a simple "learning" example-create a DataItem class that defines some personal information properties, such as name, age, and address.

Quiz-how much memory does such an object take up?

First, let's try the following test scenario:

D1 = DataItem ("Alex", 42, "-")

Print ("sys.getsizeof (D1):", sys.getsizeof (D1))

The answer is 56 bytes. It looks small, and the result is satisfactory.

However, let's examine another object with more data:

D2 = DataItem ("Boris", 24, "In the middle of nowhere")

Print ("sys.getsizeof (D2):", sys.getsizeof (D2))

The answer is still 56. This makes us understand that the result is not entirely correct.

Our intuition is right, and the problem is not that simple. Python is a very flexible language with dynamic typing that stores a lot of extra data as it works. The extra data itself takes up a lot of memory.

For example, sys.getsizeof ("") returns 33, yes, as many as 33 bytes per blank line! And sys.getsizeof (1) will return 24-24 bytes for this number (I suggest that C programmers click to finish reading now so as not to lose faith in the beauty of Python).

For more complex elements, such as dictionaries, sys.getsizeof (dict ()) returns 272 bytes, which is just an empty dictionary. So much for example, but the facts are clear, not to mention that RAM manufacturers also need to sell their chips.

Now, let's get back to our DataItem class and "quiz" problem.

How much memory does this class take up?

First, we will output the entire contents of the class at a lower level:

Def dump (obj):

For attr in dir (obj):

Print ("obj.%s =% r"% (attr, getattr (obj, attr)

This function will show what is hidden under the invisibility cloak so that all Python functions (types, inheritance, and other packages) can be run.

The results are impressive:

How much memory does it take up?

On GitHub, there is a function that calculates the actual size, which is implemented by recursively calling the getsizeof of all objects.

Let's try this:

We get 460 and 484 bytes respectively, which seems closer to the truth.

Using this function, we can do a series of experiments. For example, I want to know how much space the data would take up if DataItem were in the list.

The get_size ([D1]) function returns 532 bytes, which is obviously the "original" 460 + some extra overhead. But get_size ([D1 and D2]) returns 863 bytes-less than 460 bytes 484. The result of get_size ([d1MageD2recoverd1]) is even more interesting, producing 871 bytes, just a little more, indicating that Python is smart enough to no longer allocate memory for the same object.

Now let's look at the second part of the problem.

Is it possible to reduce memory consumption?

The answer is yes. Python is an interpreter, and we can extend our class at any time, for example, by adding a new field:

This is a great feature, but if we don't need it, we can force the interpreter to use the slots directive to specify a list of class attributes:

For more information, please refer to the "dict and weakref" section of the documentation. The space savings from using dict can be significant.

When we tried it, we found that get_size (D1) returned 64 bytes, which was about 7 times less than 460. As a reward, the creation of objects has been increased by about 20% (see the first screenshot of the article).

The actual use of such a large memory gain will not lead to other overhead costs. Simply add elements to create 100000 arrays and view memory consumption:

The result is 16.8MB when there is no slots and 6.9MB when slots is used. Not seven times, of course, but considering the small changes in the code, it still performs well.

Now let's discuss the disadvantages of this approach. Activating slots disables the creation of all other elements, including dict, which means that, for example, the following code that converts a structure to json will not work:

Def toJSON (self):

Return json.dumps (self.dict)

But this is also easy to do, and you can generate your dict programmatically, iterating through all the elements in the loop:

It is also impossible to dynamically add new variables to the class, but in our project, this is not necessary.

Here is the last quiz. Let's see how much memory the whole program needs. Add an infinite loop at the end of the program to keep it running, and view the memory consumption in the Windows task manager.

Without slots

69Mb becomes 27Mb. Well, after all, we saved memory. It's good for the result of adding only one line of code.

Note: the tracemalloc debug library uses a lot of extra memory. Obviously, it adds additional elements to each object created. If you turn it off, the total memory consumption will be much less, and the screenshot shows two options:

How to save more memory?

You can use the numpy library, which allows you to create structures in C style, but in this project, it needs to improve the code more deeply, so the first approach is sufficient for me.

Oddly enough, the use of slots has never been analyzed in detail on Habr é, and I hope this article will fill this gap.

Conclusion

This article may seem like an anti-Python advertisement, but it is not at all. Python is very reliable (you have to work very hard to "delete" programs in Python), a language that is easy to read and easy to write. In many cases, these advantages far outweigh the disadvantages, but if you need to maximize performance and efficiency, you can use the numpy library to write code like C++, which can process data very quickly and efficiently.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.