Shulou(Shulou.com)06/02 Report--

This series of articles focuses on the core library Libuv in nodejs.

My reference book:

Park Ling's profound and simple nodejs

Windows Core programming of Jeffrey Richter

Anthony Williams's C++ concurrent programming practice

It is tentatively scheduled for four articles:

1) the initial understanding of conquest (background and important concepts, diagrams, compilation of libuv, examples) 2) the progress of conquest (internally used c language skills and the use of QUEUE) 3) the most exciting parts of conquest (thread pool, iocp, synchronization, concurrency, communication between threads, etc.) 4) Conquest Harmony (initialization of Libuv, main loop, harmonious communication between main thread and thread pool)

By conquering the Libuv series of articles, the aim is to let you know:

1) the beauty of C language (simple syntax, powerful function, close to hardware, suitable for system-level programming, there are many skills) 2) Multi-thread and thread synchronization technology 3) Thread pool technology 4) windows IOCP technology 5) understand the similarities and differences, advantages and disadvantages between ms PPL, intel TBB and libdispatch (ios gcd). 6) debug skills under multithreading in vs C++

The reason why it is called understanding, rather than mastery, is because these technologies tend to the underlying system programming, not a handful, and it takes a certain amount of time and experience to settle down. So it can only be said that it is for everyone to understand.

For a simple period, there are some restrictions:

Only focus on the implementation under windows, only focus on the analysis of libuv's general cpu-intensive computing framework, understand the cpu intensive computing framework, in fact, asynchronous io is easier to understand because under windows, asynchronous io through IOCP (completion port) will make the code implementation very simple and efficient.

While writing this blog, I suddenly felt that we should split the libuv library and extract the core part of the code to run alone. The advantage is that we need to analyze much less code, and only focus on the code we are interested in, which is good for demonstration.

There is another purpose:

Just upgrade on the basis of the simplified and refined version to see if you can achieve the dynamic balance of tasks (both ms PPL and intel TBB libraries have dynamic balance, the function of Work Stealing, which is confirmed by libuv. Libdispatch has not found that the source code is being read and has not seen it yet). This is still very difficult to achieve. Let's just practice our hands, .

Let me give it a try and see if it works!

(this article was written in 2016. after studying during this period of time, I gave up the above idea of splitting on 2017-2-10. It really involves too much code and operating system, so let's give up this idea decisively.)

1. Background:

Some time ago, I completed a Wechat project. During the background selection process, I spent nearly a month reviewing several java,php libraries, but finally chose nodejs for a simple reason:

1, the configuration of java is really too troublesome 2, php is really not familiar with, especially twist 3, nodejs uses js programming, I am still quite familiar with the basic part of js (not counting the es6.0 es7.0 standard), and npm is really easy to use, it is really rich in resources, sometimes too many choices is also a pain ah! 4. Nodejs is based on asynchronous io technology and has high efficiency. And distributed deployment is simple.

After countless comparisons, I finally chose a combination that made me very satisfied, and I felt that I had written at least 80% less code.

Backstage supporting plan:

Framework: strongloop/loopback (acquired by IBM, it can only be said to be extremely powerful)

Park Ling's: wechat / wechat-api / wechat-oauth

Supersheep: wechat-pay

Database: mongodb for tables that do not require transactions mysql for payment transactions

Because of inexperience, transaction processing is required at payment time, so it is adjusted to use mysql. However, from another point of view, it shows that loopback is powerful and supports unified api operation of multiple data sources.

Front desk supporting program:

Angularjs1.x

Jquery

Overall, the development of the foreground and backstage supporting system is very pleasant. The foreground is not good for statistics, but Wechat backstage, write at least 80% less code!

2. Evolution:

At present, asynchronous callback-based development is very popular, such as interface-based callback in gcd,android in ios. If you have this development experience, you will feel that nodejs's asynchronous event development model is quite familiar.

As I became familiar with nodejs, I fell in love with nodejs, so I wanted to learn more about how nodejs is implemented.

By understanding the code structure of nodejs, combined with the flow chart shown in asynchronous io in chapter 3 of nodejs, its correctness is verified step by step. Especially when I see the code written by an old bird with deep c language skills in libuv, I can't help but want to share the beauty of c code with you!

3. What is libuv?

Official introduction:

Libuv is a multi-platform support library with a focus on asynchronous I/O. It was primarily developed for use by Node.js

From the above introduction, we can know:

1) Cross-platform: windows linux unix supports 2) Asynchronous io:windows IOCP, linux epoll, unix kqueue3) is mainly used for nodejs. In fact, there are many libraries or programs that use libuv, such as the popular cross-platform. NetCore library (once the famous Microsoft asp.net) also uses libuv as the core 4) libuv not only supports asynchronous io operations, but also has a strong thread pool. For cpu-intensive operations that support multithreaded parallelism. This is the relationship between module 4, nodejs, google v8 javascript engine and libuv that we focus on: 1) nodejs is mainly composed of google v8 javascript engine and libuv. 2) V8 engine binds api implemented by libuv, so it can not only use ecma js standard language to execute js code, but also call libuv-related interfaces through js. 3) thus it can be seen that libuv itself is an independent c language library, which can not only be called directly with cCompact +, but also be bound to c # (.NetCore) or any other language, such as java, lua. The biggest advantage of the library implemented by cAccord + is that it can be bound and invoked by various other programming languages. Because all other programming languages are basically implemented with cripple +, there is an interface for intermodulation with cripple +.

Borrow two diagrams from nodejs (agreed by the author) to understand the whole process:

We libuv source code analysis of the above figure as a guide, in-depth mining of each detail, verify its correctness, so as to grasp the essence of the whole libuv!

Be sure to clearly understand what happens in the main thread and what happens in the thread pool!

Please read this picture 100 times and memorize it. It must be very helpful.

Nodejs is the data passed to Libuv for processing through the v8 engine (main thread: data input) (thread pool: data processing-depending on the data type, io data is processed by IOCP [windows] thread pool, general computing is processed by their own thread pool). After libuv processing, notify the v8 engine that I have finished, and you complete the processing (main thread: complete callback, information output).

In fact, the following points can be learned from the above description:

1) the V8 javascript engine is single-threaded, and the input of data and the output of information (complete callback) are processed in the main thread. We can verify this later in the source code analysis. Through the powerful debug function of vs, we can clearly see which thread the specific code is running in.

2) but the data processing module (libuv) is not single-threaded, depending on whether the data request type is an io request (socket, file read / write or pipe, etc.) or a work request (non-io request). Different requests use different processing strategies. For example, io requests, using IOCP under windows and epool under linux. Work requests, windows and linux all use a unified, self-implemented thread pool. And our source code analysis is to prove the above description.

5. Visual studio compiles and tests Libuv libraries:

1) how to compile the libuv library 2) when testing the libuv library, how to solve all kinds of link errors 3) the key point is to locate, analyze and solve the problem through msdn according to the link error number. Mastering the methodology is the key.

Compile:

1) install python2.,6 or version 2.7, and set the environment variables. Never install version 3.0 or higher.

At present, many cross-platform libraries are compiled or booted by python scripts, so python is a must. 、

2) in cmd:

Cd to your libuv directory

And enter git clone https://chromium.googlesource.com/external/gyp.git build/gyp

Clone the google gyp system to the build/gyp folder of the directory where libuv is located

The google website cannot be accessed because of GFW. As a programmer, I think you should have a way.

3) run vcbuild.bat under the directory where libuv is located to generate the visual studio solution. Double-click to run, and then F5 compile and debug, everything is under control!

If you can log on to the google website, run vcbuild.bat directly. If you don't have gyp installed, the gyp build system will be downloaded automatically. So you can omit the second step.

Double-click the uv.sln project, f5 compile, have a smooth journey, no difficulty! 2, how to test and run:

Because the result of compiling Libuv in windows is a static link library, we need to build a new exe project and link libuv.lib into the program exe, as follows:

1) create a new win32/ console / empty project with the name for example: LibuvTest

2) add the main.cpp file to the source file, implement the following simple code and run, F5, debug and run

3) if you cannot start the program by running the Times, set exe as the startup item:

4) use uv_wort_t for cpu intensive computing test code, F5 compile and debug

Link error occurred after F5 running

5) the LNK2019 link error indicates that the relevant Lib has not been introduced. By observing the relevant error content, you can make sure that the Libuv.lib has not been imported.

1. Find the directory to which Libuv.lib is compiled

2. Right-click on the LibuvTest project, select the properties menu, pop up libuvTest Property Pages, and then select the Linker/input/Additional Dependencies interface

3. Relative path mode, add libuv.lib library

Referenc

6) after adding libuv.lib, continue with F5 debugging, there are still a large number of link errors:

7) resolve the LNK4098 error, which is caused by the LIBCMTD library, which we ignore as follows:

8) resolve the LNK2019 error, which was also mentioned earlier, because the corresponding lib was not introduced.

1. Check the specific description of the link error, and you can see the socket-related

2. Msdn is a magic weapon. Look for the closesocket function in msdn to see which lib library closesocket belongs to.

3. Add Ws2_32.lib in the way you added libuv.lib () before. Or you can refer to this document and use another way to link: # pragma comment (lib, "path").

9) go round and round, keep using 8) to resolve all LNK2019 link errors

10) all link libraries that are ultimately needed

11) F5 continue to compile and debug, run the output result correctly

12) so far, the Demo of libuv's uv_work_t compiles and runs correctly. We can take a break, too! It may be too basic, but it also reflects the way to solve the problem, and write it down together for your reference.

In the next article, we focus on an important data structure in libuv: the implementation and use of QUEUE. This structure transfers data between multiple threads, so it is necessary to have an in-depth understanding of its principle and implementation.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.