Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you an example analysis of the open source Python interpreter Pyston that Dropbox is developing. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

Hello, everyone. I am very excited to announce Pyston, an open source Python interpreter being developed by Dropbox. The goal of this project is to produce a high-performance Python interpreter so that Python can also be used in areas occupied by traditional system languages such as C++.

At Dropbox, we love Python and try to do everything we can with it. However, as the scale grows and there are more and more problems to deal with, we begin to find it extremely difficult to continue to use Python to achieve our performance goals. Sometimes rewriting in another language doesn't help much. I personally like Python very much, and I get hurt every time we decide to rewrite something, so I want to do something about it. After doing some useless work on static compilation, we looked around and found that JIT technology was very successful on Javascript, especially Chrome's V8 engine greatly improved the performance of Javascript. We hope to achieve the same performance improvement on Python with the same technology.

Pyston is still in its infancy and cannot be put into use yet. But we hope to release and open source it at the beginning of its life cycle so that we can work with the Python and JIT communities to develop it. There are too many details to write in this blog, but we'd like to talk about why we need a new Python implementation and talk a little bit about how Pyston works.

Why choose to implement a new Python interpreter

There are already a lot of Python implementations using JIT technology: PyPy uses its tracing JIT to improve performance; Jython and IronPython are built on virtual machines that widely support JIT. So why do we think it's worth starting to create a new implementation?

To put it simply, it is because we believe that the vast majority of promising technologies are incompatible with existing implementations. For example, in the Javascript world, there is a strong performance advantage in switching from tracing JIT to method-at-a-time JIT. Whether Python has the same performance advantages is debatable, but since both approaches are fundamentally incompatible with existing implementations, the answer can only be to build a new method-at-a-time JIT.

Another difference is that we use the traditional garbage collector in a planned way to support the extension module efficiently. Again, we don't know if this is a better approach, but this decision is essential for a JIT that is difficult to test under the existing implementation.

The downside of starting from scratch is that creating a new language implementation is undoubtedly a huge task. Fortunately, some tools to help with this process have begun to emerge. In particular, Pyston is built on top of LLVM, so that we do not need to deal with the details to generate high-quality code at the upper level. Still, a new Python implementation is a huge project, so Pyson will not be available right away.

How does it work?

From the top level, Pyston converts parsed Python code into LLVM intermediate code. The intermediate code is then optimized by LLVM and passed to LLVM's JIT engine to produce executable machine code. LLVM contains many optimization steps and mechanisms that allow it to produce very fast code.

The problem, however, is that LLVM cannot produce Python code because dynamic languages have to hide all the underlying behavior behind type Type Dispatch. To solve this problem, Pyston uses type inference: while it is usually impossible to prove that a variable will be of a particular type, Pyston can often predict what type an object will be based on certain facts. Once a prediction is made, Pyston will detect the prediction at run time and choose between the fast branch corresponding to the prediction and the slow branch corresponding to the failure of the prediction.

Pyston also includes many other modern technologies, such as hidden classes and inline caches designed for quick lookup of properties and quick invocation of methods. You can find more technical details on the Github page, as well as a separate blog post on these technical details.

Current situation

Pyston is still in its infancy and supports only a minimum subset of the Python language. It's not fair to talk about benchmark data, because 1) Pyston doesn't support benchmarks that are large enough, so it doesn't make sense. 2) Pyston does not support all runtime features (including some that may cause deceleration), so this is not a similar comparison. Under these two considerations, Pyston can usually beat CPython in performance, but it is still weaker than PyPy.

The above is the example of the open source Python interpreter Pyston that Dropbox is developing for you. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.