Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to set uninterrupted breakpoints in the process of Python debugging". In daily operation, I believe many people have doubts about how to set uninterrupted breakpoints in the process of Python debugging. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "how to set uninterrupted breakpoints in the process of Python debugging". Next, please follow the editor to study!

The heart of the Python debugger: sys.set_trace

Of the many optional Python debuggers, the three most widely used are:

Pdb, which is part of the Python standard library

PyDev, which is a debugger embedded in IDE such as Eclipse and Pycharm

Ipdb, which is the debugger for IPython

The Python debugger has many choices, but they are almost all based on the same function: sys.settrace. It is worth mentioning that sys.settrace is also probably the most complex function in the Python standard library.

To put it simply, the role of settrace is to register a trace function for the interpreter, which is called when the following four situations occur:

Function call

Statement execution

Function return

Exception thrown

A simple trace function looks something like this:

Def simple_tracer (frame, event, arg): co = frame.f_code func_name = co.co_name line_no = frame.f_lineno print ("{e} {f} {l}" .format (e=event, f=func_name, l=line_no)) return simple_tracer

When analyzing the function, we first focus on the parameters and the return value, and the parameters of the tracking function are:

Frame, the current stack frame, which is the object that contains the complete state in the interpreter when the current function is executed

Event, event, which is a string whose value may be call, line, return, or exception

Arg, parameter, whose value is based on the type of event, is optional

The return value of this trace function is itself, because the interpreter needs to continuously track two types of trace functions:

Global trace function (per thread): this trace function is set by the current thread calling sys.settrace and is called when the interpreter creates a new stack frame (that is, when a function call occurs in the code). Although there is no ready way to set trace functions for different threads, you can call threading.settrace to set trace functions for all newly created threading module threads.

Local tracking function (each frame): the interpreter sets the value of the tracking function to the return value when the global tracking function creates the frame. Similarly, there is no ready-made method to automatically set the local tracking function when the frame is created.

The purpose of this mechanism is to give the debugger a more accurate grasp of the frames being tracked so as to reduce the impact on performance.

Build the debugger in three simple steps (our original idea)

Relying solely on what was mentioned above, it seems impractical to build a real debugger with homemade trace functions. Fortunately, Python's standard debugger, pdb, is built on Bdb, the base class in the Python standard library dedicated to building debuggers.

The simple Bdb-based breakpoint debugger looks like this:

Import bdbimport inspect class Debugger (bdb.Bdb): def _ init__ (self): Bdb.__init__ (self) self.breakpoints = dict () self.set_trace () def set_breakpoint (self, filename, lineno, method): self.set_break (filename, lineno) try: self.breakpoints [(filename, lineno)] .add (method) except KeyError: self.breakpoints [(filename, lineno)] = [method] def user_line (self) Frame): if not self.break_here (frame): return # Get filename and lineno from frame (filename, lineno, _) = inspect.getframeinfo (frame) methods = self.breakpoints [(filename, lineno)] for method in methods: method (frame)

The full composition of this debugger class is:

Inherit Bdb, define a simple constructor to initialize the base class, and start tracing.

Add the set_breakpoint method, which uses Bdb to set breakpoints and track them.

Overloads the user_line method that Bdb calls on the current user line, which must be called by a breakpoint, then gets the source location of the breakpoint, and calls the registered breakpoint.

How efficient is this simple Bdb debugger?

The goal of Rookout is to provide a similar experience to ordinary debuggers in production-level performance scenarios. So, let's take a look at how the previously built simple debugger works.

To measure the overall performance overhead of the debugger, we tested it using the following two simple functions, which were executed 1600 times in different scenarios. Note that breakpoints are not executed in all scenarios.

Def empty_method (): pass def simple_method (): a = 1 b = 2 c = 3 d = 4 e = 5 f = 6 g = 7 h = 8 I = 9 j = 10

It takes a lot of time to complete the test in the case of using the debugger. The poor results indicate that the performance of this rudimentary Bdb debugger is far from adequate for use in a production environment.

Optimize the debugger

There are three main ways to reduce the extra overhead of the debugger:

Limit local tracing as much as possible: because each line of code can contain a large number of events, local tracing is much more expensive than global tracing.

Optimize the call event and return control to the interpreter as soon as possible: when the call event occurs, the debugger's main job is to determine whether the event needs to be traced.

Optimize the line event and return control to the interpreter as soon as possible: the main job of the debugger when the line event occurs is to determine whether we need to set a breakpoint here.

So we copied the Bdb project to simplify the features, simplify the code, and optimize for the usage scenario. Although these efforts have achieved some results, they still can not meet our needs. So we continued to make other attempts to optimize and migrate the code to .pyx for compilation using Cython, but the result (shown in the following figure) is still not ideal. In the end, after taking a closer look at the CPython source code, we realized that it was impossible to make the tracking process fast enough to meet production requirements.

Abandon Bdb and try bytecode operation instead

After going through the disappointment caused by the previous trial-failure-retest cycle of standard debugging methods, we turned to another option: bytecode operation.

The work of the Python interpreter is mainly divided into two phases:

Compile Python source code into Python bytecode: this unreadable format is optimized for execution efficiency and is usually cached in well-known .pyc files.

Iterate through the bytecode in the interpreter loop: in this step the interpreter executes instructions one by one.

The mode we chose is to use bytecode operations to set uninterrupted breakpoints with no global overhead. The implementation of this method first needs to find the part we are interested in in the bytecode in memory, and then insert a function call in front of the relevant machine instructions in that part. In this way, the interpreter can achieve our uninterrupted breakpoint without any extra work.

This method does not rely on magic, let's give a brief example.

First define a simple function:

Def multiply (a, b): result = a * b return result

In the documentation for the inspect module, which contains many practical units, we learned that the bytecode of the function can be obtained by accessing multiply.func_code.co_code:

'|\ x00\ x00 |\ x01\ x00\ x14}\ x02\ x00 |\ x02\ x00S'

These unreadable strings can be translated using the dis module in the Python standard library. After calling dis.dis (multiply.func_code.co_code), we can get:

4 0 LOAD_FAST 0 (a) 3 LOAD_FAST 1 (b) 6 BINARY_MULTIPLY 7 STORE_FAST 2 (result) 5 10 LOAD_FAST 2 (result) 13 RETURN_VALUE here The study on "how to set an uninterrupted breakpoint during Python debugging" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.