Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about how to upgrade QQ VIP's AMS platform PHP7. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

QQ VIP activity Operation platform (AMS), one of the important carriers of QQ VIP's value-added operation business, undertakes the Web system of massive activity operation. AMS is an activity operation platform which is mainly implemented in PHP language. The daily request of CGI is about 300 million, with a peak of 800 million. However, for a long time before, we all adopted the older basic software version, which is PHP5.2+Apache2.0 (2008 technology). Especially since last year, with the rapid growth of AMS business with QQ VIP's value-added business, the performance pressure is increasing day by day.

So, since May 2015, we have been planning the low-level upgrade of PHP, with the ultimate goal of upgrading to PHP7. At that time, PHP7 was still in the research and development stage, and our discussion and pre-research had already begun.

I. study and pre-research of PHP 7. HHVM and JIT

Another important role in the 2015 PHP performance optimization program is HHVM, which is open source by Facebook (HipHop Virtual Machine,HHVM is a Facebook open source PHP virtual machine). HHVM uses JIT (Just In Time, just-in-time compilation, a software optimization technique that compiles bytecode into machine code at run time) and other technologies to greatly improve the execution performance of PHP code. Rumor has it that the native PHP code of the PHP5 version can be improved by 5-10 times.

HHVM originated from Facebook, and a lot of early code in Facebook was developed using PHP. However, with the rapid development of business, the efficiency of PHP execution has become a more and more obvious problem. To optimize execution efficiency, Facebook started using HipHop, a PHP execution engine, in 2008, originally to convert a large amount of Fackbook's PHP code into Clearing codes to improve performance and save resources. The performance of PHP code that uses HipHop has been improved several times. Later, Facebook made the HipHop platform open source and gradually developed into what is now HHVM.

When HHVM became a PHP performance optimization solution, PHP7 was still in the development stage. I have seen some students' communication for HHVM, the performance can be significantly improved, but there is a certain cost for service operation and maintenance to be compatible with PHP syntax. For a while, JIT became a popular thing, and many technical students suggested that PHP7 should also optimize performance through JIT.

In July 2015, I attended China's PHPCON and listened to Hui Xinchen's technology sharing about the PHP7 kernel. In fact, in 2013, Hui Xinchen (PHP7 kernel developer) and Dmitry (one of the other PHP kernel developers) made a JIT attempt (not released) on the PHP5.5 version. The original execution flow of PHP5.5 is to compile the PHP code into opcode bytecode through lexical and syntactic analysis (the format is a bit like assembly), and then the Zend engine reads these opcode instructions and parses them one by one.

They introduce type inference (TypeInf) after the opcode session, then generate ByteCodes through JIT, and then execute it.

As a result, very good results are obtained in benchmark (test program), and the performance of JIT is 8 times higher than that of PHP5.5. However, when they put this optimization into the actual project WordPress (an open source blog project), they saw little improvement in performance. The reason is that the amount of code of the test project is relatively small, and the machine code generated by JIT is not large, while the machine code generated by the real WordPress project is too large, resulting in a decline in CPU cache hit ratio (CPU Cache Miss).

All in all, JIT is not a sharp weapon to turn stone into gold in every scenario, and the performance test results separated from business scenarios are not necessarily representative.

From the performance comparison between PHP7 and HHVM released by the official Wordpress, we can see that they are basically at the same level.

2. Performance optimization of PHP7

PHP7 is a relatively low-level upgrade, which changes a lot compared to PHP5.6. In terms of performance optimization, it can be summarized as follows:

Change the basic variable from struct (structure) to union (consortium) to save memory space and indirectly reduce the memory allocation and management overhead of CPU.

Some basic variables (zend_array, zend_string, etc.) adopt the way of continuous allocation of memory space to reduce the probability of CPU Cache Miss. The efficiency difference between CPU getting data from CPU Cache and getting it from memory can be as high as 100x. To take a similar example, the efficiency of the system to read data from memory is very different from that from disk, and CPU Cache Miss is similar to experiencing a page fault.

Through macro definition and inline function (inline), let the compiler finish part of the work ahead of time. There is no need to allocate memory when the program is running, it can achieve functions similar to functions, but there is no stack overhead of function calls, so it will be more efficient.

......

3. The background of AMS platform technology selection.

In terms of improving the performance of PHP, you can choose between HHVM, which will be available directly in 2015, or the official version of PHP7, which will not be released until the end of 2015. Member AMS is a Web system with a large access level. After four years of continuous upgrade and optimization, it has accumulated more than 800 business functional components, as well as a variety of common basic libraries and scripts written by PHP, and the code scale is also relatively large.

We have a high demand for code backward compatibility for the PHP version, so in our business scenario, PHP7's good syntax for backward compatibility is exactly what we need. Therefore, we choose PHP7 as the upgrade scheme.

II. Risks and challenges of PHP7 upgrade

For a large-scale public Web service that is already online, upgrading basic public software is usually a thankless task. If it is done well, it may not be perceived by everyone. However, if something goes wrong with the upgrade, you need to bear a heavy responsibility. In order to minimize the risk of upgrading, we must first be clear about the challenges and risks of our upgrade.

So we compiled a list of upgrade challenges and risks:

Apache2.0 and PHP5.2, the two basic software versions from 2008 to 2009, are relatively old, upgrade to Apache2.4 and PHP7, the upgrade span is relatively large, the time span difference is 7-8 years, therefore, the compatibility challenge is relatively high. In fact, many of our company's existing PHP services stay in the PHP5.2 and PHP5.3 versions, and the versions are on the low side.

AMS uses a lot of self-developed tphplib extensions, and tphplib has not been maintained within the company for a long time. This extension was previously only available in compiled so versions of PHP5.3 and PHP5.2, and some of the extensions do not support thread safety. Thread safety is supported because our previous Apache uses the prefork model, while we want to be able to use Apache2.4 's Event model (the multi-process thread management model introduced in mid-2014, after prefork and worker, has better performance for supporting high concurrency).

Syntax compatibility problem, the span from PHP5.2 to PHP7 is too large, even though PHP officially claims to achieve 99% backward compatibility, but the scale of our code is relatively large, it is still an unknown risk.

The risk of new software is to upgrade basic software such as Apache and PHP to the latest version, and some of the features of these versions may have unknown risks and defects.

Some students may suggest that Nginx is a better choice. Indeed, Nginx performs better than Nginx and Apache in terms of high concurrency. But as far as PHP's CGI is concerned, there is no big gap between Nginx+php-ftpm and Apache+mod_php. On the other hand, because we have used Apache for a long time, we have accumulated more technical familiarity and experience, so it may not be the best choice, but it is a more appropriate choice for our business scenario.

three。 Version upgrade implementation process 1. High-span version upgrade mode

From an Apache2.0 in 2008 to an Apache2.4 in 2016, this cross-over is too large, and even the configuration files of the http.conf used are many different. There are many areas that need to be updated here, and there are unknown risks. Therefore, our approach is to first try to upgrade Apache2.0 to Apach3.2, adjust the configuration, observe stability, and then try further to Apach3.4. Fortunately, Apache (httpd) is a special open source community that has maintained both branch versions of Apache (2.2and 2.4), so even Apache2.2 has a relatively new version.

Therefore, we first upgraded a PHP5.2+Apache2.2, tested and observed the compatibility, and confirmed that the upgrade between the two can be relatively smooth, and we began to upgrade the Apache2.4.

We use the same idea to upgrade PHP5.2. We first upgrade PHP5.2 to PHP5.6 (at that time, PHP7 is still the beta version), and then upgrade PHP5.6 to PHP7 to solve different problems step by step in a smoother way.

As a result, our upgrade plan becomes:

Apache2.4 is compiled into dynamic MPM mode (supports switching prefork/worker/event mode through httpd configuration), and can be downgraded in real time according to the risks of the existing network.

A brief introduction to Prefork, Worker and Event:

Prefork, multi-process mode, one process serves one user request, the cost is relatively high. However, it is the most stable and does not need to support thread safety.

Worker, multi-process and multi-thread mode, 1 process contains multiple worker threads, 1 worker thread serves 1 user request, because the thread is lighter and the cost is lower. However, in the KeepAlive scenario, worker resources are occupied by client and cannot respond to other requests (empty wait).

Event, multi-process and multi-thread mode, 1 process also contains multiple worker threads, 1 worker thread serves 1 user request. However, it solves the problem that the worker thread is occupied in the KeepAlive scenario, it manages these KeepAlive connections through special threads, and then assigns "work" to the specific processing worker, and the working worker will not cause an empty wait because of KeepAlive.

An official introduction to the Event model:

Http://httpd.apache.org/docs/2.4/mod/event.html

(some students may have the impression that event mode does not support https. That is actually a statement made by some domestic technology blogs more than 2 years ago. The current version supports it. For more information, please visit the official introduction.)

The way to turn on dynamic switching mode is to add the following when compiling httpd:

-- enable-mpms-shared=all

Upgrading from PHP5.2 to PHP5.6 is relatively easy, and our main work is as follows:

Cleaned up some old extensions that are no longer in use

Solve the thread safety problem

Compile api such as cmem to a new version

PHP code syntax is based on PHP5.6 compatibility (in fact, it hasn't changed much)

Synchronous adjustment of partial expansion. The apc extension becomes zend_opcache and apcu. The previous apc included compilation cache and user memory operations. In the newer version of PHP, it was broken down into two separate extensions.

Upgrading from PHP5.6 to PHP7.0 takes a lot of work and is relatively complex, so we have an upgrade plan for each stage:

Technical pre-research, PHP7 upgrade preparation.

Compile and build the environment, download the relevant compilation packages, and build a complete compilation environment and testing environment. (the compilation environment still needs to rely more on so)

Compatible with upgrades and tests. PHP7 extension recompilation and code compatibility work, AMS functional verification, performance stress testing.

Online grayscale. Package as pkg installation package, write relevant installation shell installation execution code (including soft links, resolve some so dependencies). Then, the grayscale is installed on the existing network and observed.

It's officially released. Expand the grayscale range, full upgrade.

Because many problems have been solved ahead of time in the process of upgrading from PHP5.2 to PHP5.6, the main difficulty in upgrading PHP7 lies in the compilation and upgrading of tphplib extensions.

The main tasks involved include:

The expansion of PHP5.6 to the relatively large-scale transformation and upgrade of PHP7.0 (where the workload is relatively heavy)

Renaming of apcu-compatible memory operation functions. In PHP5, the function of the apc prefix we use is no longer available, and synchronization becomes a function of the apcu prefix (which requires an apcu extension).

Syntax compatible upgrade. In fact, the workload is not large, and there is not much change from PHP5.6 to PHP7.

We completed the compilation of PHP7 and Apache in mid-April 2016, carried out the current network grayscale in late April, and released all of them to one of the existing network clusters in early May.

two。 Error debugging methods in the process of upgrade

When upgrading and recompiling PHP7 extensions, if the execution results do not meet expectations or the process core fails, many errors cannot be seen in the error log, which is not conducive to the analysis of the problem. You can use the following methods to locate and analyze most of the problems:

Var_dump/exit

By gradually outputting information from the PHP code layer and executing exit, you can gradually locate the location of the PHP function where the exception is executed, and then reverse check the implementation function in the extension according to the name of the PHP function to find the problem. This method is relatively simple, but inefficient.

Gdb-p/gdb c

This method is mainly used to analyze the scenario of the process core. We compile mod_php (PHP becomes a child or block of Apache) and use gdb-p to monitor the service process of Apache.

Command: ps aux | grep httpd

Gdb debugs the specified process:

Command: gdb-p

Use c for capture, and then construct a web request that can cause core:

Apache is usually in multi-process mode. In order to make it easier to reproduce the problem, you can modify the parameters in http.con to change the number of starting processes to 1 (multiple parameters in the figure below need to be adjusted to start only a single process and a single thread).

Of course, there is a simpler way, because Apache itself supports single-process debug mode.

. / apachectl-k start-X-e debug

Then it is easier to debug through gdb-p.

Use the strace command to see what the Apache process is doing, and analyze and locate the problem according to its execution.

Strace-Ttt-v-s1024-f-p pid (process id)

Note: execute these commands and pay attention to permission issues, which are likely to require root permissions.

IV. PHP5.6 to PHP7.0 extension upgrade practice record 1. Changes in data types

Zval

The birth of php7 begins with the change of zval structure. PHP7 no longer needs pointers, and most zval** needs to be modified to zval*. If PHP7 manipulates zval directly, then zval* also needs to be changed to zval,Z_*P () and zoned * (), ZVAL_* (var,...) Need to change to ZVAL_* (& var, …) Be sure to use the & symbol carefully because PHP7 hardly requires the use of zval*, in as many places as it does & and it needs to be removed.

ALLOC_ZVAL,ALLOC_INIT_ZVAL,MAKE_STD_ZVAL these macros that allocate memory have been removed. In most cases, the zval* should be changed to zval, and the INIT_ PZVAL macro has been removed.

/ * 7.0zval structure source code * / / * value field, which occupies only one size_t length, only pointer or double or long * / typedef union _ zend_value {zend_long lval; / * long value * / double dval; / * double value * / zend_refcounted * counted; zend_string * str; zend_array * arr Zend_object * obj; zend_resource * res; zend_reference * ref; zend_ast_ref * ast; zval * zv; void * ptr; zend_class_entry * ce; zend_function * func; struct {uint32_t w1; uint32_t w2;} ww;} zend_value Struct _ zval_struct {zend_value value; / * value * / union {. The extended field is mainly type information * / union {… ... Expand the field and save the auxiliary information * /}

Integer type

You can switch directly:

Long- > zend_long

/ * define * / typedef int64_t zend_long;/* else * / typedef int32_t zend_long

String type

In PHP5.6 version, char* + len is used to represent strings. Encapsulated in PHP7.0, the zend_string type is defined:

Struct _ zend_string {zend_refcounted_h gc;zend_ulong h; / * hash value * / size_t len;char val [1];}

Conversion between zend_string and char*:

Zend_string * str;char * cstr = NULL;size_t slen = 0Trachap. * the method to obtain char* and len from zend_string is as follows * / cstr = ZSTR_VAL (str); slen = ZSTR_LEN (str); / * char* method for constructing zend_string * / zend_string * zstr = zend_string_init ("test", sizeof ("test"), 0)

To extend the method, when parsing parameters, replace's' with 'strings where strings are used:

/ * for example, * / `zend_ string`` * zstr`; if (zend_parse_parameters (ZEND_NUM_ARGS (), "S", & zstr) = = FAILURE) {RETURN_LONG (- 1);}

Custom object

Source code:

/ * php7.0 zend_object definition * / struct _ zend_object {zend_refcounted_h gc;uint32_t handle;zend_class_entry * ce;const zend_object_handlers * handlers;HashTable * properties;zval properties_table [1];}

Zendobject is a variable length structure. So in the structure of the custom object, zendobject needs to be placed in the last item:

/ * example * / struct clogger_object {CLogger * logger;zend_object std;// later}; / * get the object by offset * / static inline clogger_object* php_clogger_object_from_obj (zend_object * obj) {return (clogger_object*) ((char*) (obj)-XtOffsetOf (clogger_object, std)) } # define Z_USEROBJ_P (zv) php_clogger_object_from_obj (Z_OBJ_P ((zv)) / * when releasing resources * / void tphp_clogger_free_storage (zend_object * object TSRMLS_DC) {clogger_object * intern = php_clogger_object_from_obj (object); if (intern- > logger) {delete intern- > logger; intern- > logger = NULL;} zend_object_std_dtor (& intern- > std);}

Array

The hash table in 7.0 is defined as follows, with some comments:

Hash table structure in / * 7. 0 / typedef struct _ Bucket {/ * an entry in the hash table * / zval val; / * delete the element zval type marked as IS_UNDEF * / zend_ulong h; / * hash value (or numeric index) * / zend_string * key; / * string key or NULL for numerics * /} Bucket; typedef struct _ zend_array HashTable Struct _ zend_array {zend_refcounted_h gc; union {struct {ZEND_ENDIAN_LOHI_4 (zend_uchar flags, zend_uchar nApplyCount, zend_uchar nIteratorsCount, zend_uchar reserve)} v; uint32_t flags;} u; uint32_t nTableMask Bucket * arData; / * Save all array elements * / uint32_t nNumUsed; / * how long is currently used, * / uint32_t nNumOfElements / * the number of elements actually saved in the array. Once the value of nNumUsed reaches nTableSize,PHP, it will try to adjust the arData array to make it more compact by discarding the entry of type UDENF * / uint32_t nTableSize; / * the array is allocated to the power of memory size 2 (minimum value is 8) * / uint32_t nInternalPointer; zend_long nNextFreeElement Dtor_func_t pDestructor;}

PHP7 defines a series of macros in zend_hash.h to manipulate arrays, including traversing key, traversing value, traversing key-value, and so on. Here is a simple example:

/ * Array examples * / zval * arr;zend_parse_parameters (ZEND_NUM_ARGS (), "a", & arr_qos_req); if (arr) {zval * item; zend_string * key After ZEND_HASH_FOREACH_STR_KEY_VAL (Z_ARRVAL_P (arr), key, item) {/ *... * /}} / * obtains the item, you can obtain the long, double, string values * / zval_get_long (item) zval_get_double (item) zval_get_string (item) through the following api

In the PHP5.6 version, key is found through zend_hash_find, and then the result is given to the zval * * variable. If the query is not available, you need to allocate memory by yourself. Initialize an item and set the default value.

2. Api changes in PHP7

Duplicate parameter

Many API in PHP5.6 need to enter a duplicate parameter to indicate whether a variable needs to be copied, especially for string operations. The duplicate parameter is cancelled in PHP7.0, and for string-related operations, as long as there is a duplicate parameter, you can delete it directly. Because the zval_string structure is defined in PHP7.0, the string operation no longer needs the zend_string value. The bottom layer can directly initialize a zend_string with zend_string_init, while in PHP5.6, the string is stored in zval, and the memory of zval needs to be allocated manually.

The API involved is summarized as follows:

Add_index_string 、 add_index_stringl 、 add_assoc_string_ex 、 add_assoc_stringl_ex 、 add_assoc_string 、 add_assoc_stringl 、 add_next_index_string 、 add_next_index_stringl 、 add_get_assoc_string_ex 、 add_get_assoc_stringl_ex 、 add_get_assoc_string 、 add_get_assoc_stringl 、 add_get_index_string 、 add_get_index_stringl 、 add_property_string_ex 、 add_property_stringl_ex 、 add_property_string 、 Add_property_stringl 、 ZVAL_STRING 、 ZVAL_STRINGL 、 RETVAL_STRING 、 RETVAL_STRINGL 、 RETURN_STRING 、 RETURN_STRINGL

MAKE_STD_ZVAL

In PHP5.6, the zval variable is allocated on the heap, so to create a zval variable, you need to declare a pointer and then use MAKE_STD_ZVAL to allocate space. In PHP7.0, this macro has been cancelled, variables are assigned on the stack, you can define a variable directly, you no longer need MAKE_STD_ZVAL, and just remove it where you use it.

ZEND_RSRC_DTOR_FUNC

Modify the parameter name rsrc to res/* PHP5.6 * / typedef struct _ zend_rsrc_list_entry {void * ptr; int type; int refcount;} zend_rsrc_list_entry;typedef void (* rsrc_dtor_func_t) (zend_rsrc_list_entry * rsrc TSRMLS_DC); # define ZEND_RSRC_DTOR_FUNC (name) void name (zend_rsrc_list_entry * rsrc TSRMLS_DC) / * PHP7.0 * / struct _ zend_resource {zend_refcounted_h gc / * reference count is structurally encapsulated in * / int handle; int type; void * ptr;}; typedef void (* rsrc_dtor_func_t) (zend_resource * res); # define ZEND_RSRC_DTOR_FUNC (name) void name (zend_resource * res)

In PHP7.0, to upgrade the zend_rsrc_list_entry structure to zend_resource, you only need to change the parameter name in the new version.

Secondary pointer macro, that is, Z_*_PP

All PP macros are canceled in PHP7.0, and the corresponding P macros are used directly in most cases.

Zend_object_store_get_object cancelled

According to the official wiki, the following macro can be defined to obtain object. In fact, this macro is used frequently:

Static inline user_object * user_fetch_object (zend_object * obj) {return (user_object *) ((char*) (obj)-XtOffsetOf (user_object, std));} * / # define Z_USEROBJ_P (zv) user_fetch_object (Z_OBJ_P ((zv)

Zend_hash_exists 、 zend_hash_find

For all functions that require string arguments, the way in PHP5.6 is to pass two arguments (char* + len), while zend_string is defined in PHP7.0, so only one zend_string variable is required.

The return value becomes of type zend_bool:

/ * example * / zend_string * key; key = zend_string_init ("key", sizeof ("key"), 0); zend_bool res_key = zend_hash_exists (itmeArr, key)

Reference:

1. Php5 to phpng, http://yaoguais.com/?s=md/php/php7-vm.md

2. PHP extension development and kernel application, http://www.walu.cc/phpbook/10.1.md

3. New Hashtable implementation and performance improvement in PHP 7, http://gywbd.github.io/posts/2014/12/php7-new-hashtable-implementation.html

4. Deeply understand the zval and https://github.com/laruence/php7-internal/blob/master/zval.md of PHP7

5. Official wiki, https://wiki.php.net/phpng-upgrading

6. Php manual, http://php.net/manual/zh/index.php

7. PHP7 uses resources to package the implementation of third-party extensions and its source code interpretation, https://mengkang.net/684.html

5. Performance optimization results of AMS platform upgrade PHP7

The current network service is a very important and sensitive environment, which will affect the user experience and cause accidents on the current network. So, after we finished compiling and testing PHP7 in late April, we launched a grayscale online on one of the AMS machines, observed it for a few days, and then gradually expanded the grayscale range to complete the upgrade in early May.

This is the stress test result of our stress test AMS for querying multiple activity counters, as well as the CPU load data of the current network CGI machine under the same peak TGW traffic scenario:

According to the results of our business stress test and current network, it is basically consistent with the official said that the performance has doubled.

The AMS platform has a lot of CGI machines. The upgrade and application of PHP7 bring us a performance improvement, which can effectively save the cost of hardware resources. And, through Apache2.4 's Event mode, we have enhanced Apache's ability to support concurrency.

Our PHP7 upgrade R & D project team, after continuous efforts and promotion over a long period of time in the past, finally launched the network grayscale in late April 2016 and fully upgraded in the cluster in early May, which has greatly improved the performance of our AMS activity operation platform.

PHP7's innovation is of great significance and value to the PHP language itself, which makes me more convinced that PHP will be a better and better language. At the same time, I would like to thank the developers of the PHP community for the performance improvement they have brought to our business.

The above is the editor for you to share how to upgrade the QQ VIP AMS platform PHP7, if you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.