Shulou(Shulou.com)06/03 Report--

From the main road to simplicity, we will return to nature.

Preface

On the eve of this blog post, some other friends are asking the following related questions:

How's the performance?

Is it easy to use?

Where are the development goals?

How to feedback the problem?

What are the advantages over lua open source projects in the industry?

Wait, the above questions will be introduced one by one in this article.

The cause of CF

First of all, let's talk about feelings! I believe that practitioners in every industry have more or less had a dream, this dream is called: "I will develop a XXX"! In fact, the author was the same at the beginning.

Whenever you work overtime, read documents, and debug in the middle of the night (early in the morning), you always search for frames or entry-level demo from a few years ago or more than a decade ago. For example: tinyhttp, the source code of the link is the mirror station of some students' fork.

Every time I see these contents, I will more or less arouse a trace of dying enthusiasm in my heart, maybe this is the last desire for technology?

Is to ask yourself again and again if you want to do it before you start to create a project? Can you hold on? Maybe being sprayed is an extravagant hope?

After answering these questions one by one, the project was created at the end of 2018.

Tell me the truth! The most difficult thing about a web development framework is not to achieve a function, but to build wheels step by step from scratch!

As a web development framework, the two most important functions are definitely needed! Timer library, event-driven library. How to choose? There are two options: libev / libuv.

Libev is mature and stable, lightweight, unix like support, easy to embed; libuv is better than libev, adding many functions (thread pool, signal, synchronization, lock, etc.), better encapsulation, and added windows support

From the initial selection of cf framework development, libuv is definitely the best solution at present. But the author chose libev. From then on, the arduous road of underlying development unfolded.

First of all, the author does not allow users to carry out actual business development with CumberCraft +! This will make users have higher development and learning costs, and it is particularly important to choose a better scripting language.

The author is a little familiar with Lua, so he chose Lua as the business scripting language. As for the advantages of Lua language, we won't talk about it here. There are a lot of articles praising it online.

Now that the scripting language has been selected, start writing code! Let's Lua.

The way to write CF 1. Network layer

First, let's look at a piece of API code encapsulated by C for Lua calls:

LUAMOD_API intluaopen_tcp (lua_State * L) {luaL_checkversion (L); / * add SSL support * / SSL_library_init (); SSL_load_error_strings (); / CRYPTO_set_mem_functions (xmalloc, xrealloc, xfree); / / OpenSSL_add_ssl_algorithms (); / * add SSL support * / luaL_newmetatable (L, "_ _ TCP__") Lua_pushstring (L, "_ index"); lua_pushvalue (L,-2); lua_rawset (L,-3); lua_pushliteral (L, "_ mode"); lua_pushliteral (L, "kv"); lua_rawset (L,-3) LuaL_Reg tcp_libs [] = {{"read", tcp_read}, {"write", tcp_write}, {"ssl_read", tcp_sslread}, {"ssl_write", tcp_sslwrite}, {"stop", tcp_stop}, {"start", tcp_start}, {"close", tcp_close}, {"listen", tcp_listen}) {"connect", tcp_connect}, {"ssl_connect", tcp_sslconnect}, {"new", tcp_new}, {"new_ssl", ssl_new}, {"free_ssl", ssl_free}, {"new_server_fd", new_server_fd}, {"new_client_fd", new_client_fd}, {NULL, NULL} LuaL_setfuncs (L, tcp_libs, 0); luaL_newlib (L, tcp_libs); return 1;}

The above is a snippet of the C code implemented by TCP. If you are interested in reading the source code, please click here.

It is well known that Lua does not have native Socket. Then it is necessary for the framework writer to abstract the underlying logic and reimplement a set of API.

Simple encapsulation of Lua C libraries can be done by anyone, and it's not that difficult. But our goal is to make the underlying synchronous blocking Socket hook non-blocking, and that's when the difficulty comes!

We all know that libev is an event-driven network library based on react model, and all business logic after registering events is triggered in the form of callback. Doesn't that turn into node-lua code? (laughter)

At this time, the author came up with an idea to solve this problem! The execution process is as follows:

Every time you need to do some synchronization, call C API to register the callback event.

Create a Lua co-program for all currently registered events to save the context and relinquish the current co-program execution rights.

After the registration event is triggered, the C API recovery protocol is called to continue.

To put it simply, the asynchronous callback logic at the C level is encapsulated as the synchronous non-blocking of the Lua layer to ensure that the thread is not blocked because of IO problems.

The following provides a piece of pseudo code for socket synchronization and non-blocking, for reference:

Function TCP:recv (bytes) local current_co = co_self () self.read_co = read_ev (function ()-do action-stop timer_ev-wakeup (current_co) reinstatement end) self.timer_co = self.timer_ev (function ()-do action-stop read_ev-wakeup (current_co) Restoration Enforcement end) tcp_start (io, EV_READ) Self.read_co) timer_start (timer, 3 second timeout, self.timer_co) return co_yield ()-relinquish the right to execute end

A Lua version of Socket EV_READ pseudocode processing flow is roughly like the above, to see the actual processing logic, please see here.

In the same way, Socket write/connect/listen and other API can copy directly (UDP is more or less the same). As a matter of fact, there is a small episode here, which is the trap of SSL SOCKET, but I won't talk about it because of the space.

A careful partner may find that the code registers both Socket and Timer events, and Socket non-blocking operations cannot solve the problem of read and connect timeouts. So the cf framework is simply a little more thoroughly packaged.

So far, Socket can be regarded as the completion of basic hook and encapsulation. Then you can start writing application layer protocols.

two。 Application layer protocol

Now that Socket is finally working properly, there are new problems.

Libev does not have its own asynchronous dns

Dns still needs to be packaged by users, and this hole is really hard to fill! Fortunately, there are predecessors on the network to achieve the Lua version of asynchronous dns, the author after a little understanding, borrowed to encapsulate the internal use.

In this way, cf also has a deeply customized asynchronous dns library. (it is not perfect, but it is sufficient to use)

Whether a network library is popular or not basically depends on the ecology. Then the wheels of the protocol layer have to be built again:

Httpd and httpcmailmysqlredismqttwebsocket

Some of these protocols are borrowed and customized by seniors, while simple protocols are directly handwritten in 1-2 hours.

3. Encapsulation and ease of use

In order not to make API so closed and improve the usability of cf, the author decided to initially package mysql and redis.

Encapsulation includes commonly used functions, connection pooling, object-oriented operations, no need to manually manage the session life cycle, and so on. Simplify the burden of programming ideas to improve development efficiency.

As for the internal Socket, it is possible to use the framework to solve the release problem to ensure that the number of file descriptions is limited. (actually, I don't like to rely on gc passive close fd and free memory)

What is CF?

If you have patiently read the first part of the introduction, then you should have a general understanding of cf.

Cf, whose full name is CoreFramework, is a Lua network development framework based on libev. A variety of network protocols and third-party libraries are implemented to help users develop project prototypes rapidly.

Cf respects the separation of front-end and back-end solutions in httpd, which only implements basic view routing and does not support rest-style API routing. Although this may attract criticism from many people.

Cf httpd embedded websocket support, convenient for users to reuse the port at the same time can enjoy the fun of long-connection writing.

For more information, please refer to the Wiki of the project address.

What can CF do?

Micro-service scenario based on container technology (swarm/kubernetes);-- recommended

The front-end agent layer of the game server;-- recommended

CVM whose memory / CPU resources are scarce;-- recommended

Stateless clusters with high performance requirements;-- recommended

Mass persistent connection (websocket) Agent cluster;-- which technology stack is recommended for CF?

Transport layer: TCP/UDP

Session layer: SSL Client support

Protocol layer: dns/webocket/http/mqtt/redis/mysql/smtp

Tool library: Timer/TASK

Third-party libraries: Libev, openssl/libressl, lua-5.3, jemalloc/tcmalloc (optional)

How to install CF?

Cf currently supports most Unix like operating systems, and the author develops on Mac, so Mac support is necessary.

The Linux tested by cf is Centos, so basically there is no problem with the compiled operation of the operating system based on the Linux kernel (export added / usr/local/lib)

At the same time, the author also made a simple Dockerfile for you. The file is in the root directory of the project, you can download it and use it directly.

Of course, if you don't want to make Dockerfile, you can also use the Docker command to pull out the image created by the author and put it in docker hub. Candymi/cfweb

For more information and instructions, please refer to Docker installation and compilation installation.

How does CF work? Test run

Bash#:. / cfadmin

Background operation

Bash#:. / cfadmin

Quit

Killall cfadmin

Ctrl + c

Where's the file?

The author wrote a document that is too detailed to be more detailed, so as to get everyone's likes and attention.

The author also prepares sufficient Chinese and English notes for students who like to read the source code, so that you can quickly understand the working mode of CF (it is easy to understand some special words with Chinese / English notes).

Answer the previous question:

q. How's the performance?

a. The performance is good, but please test the specific value by yourself.

q. Is it easy to use?

a. One-hour start to learning lua-> one-hour start to cf

q. What is the original purpose of this project?

a. In fact, it has already been answered earlier.

q. Where are the development goals?

A. there is a TODO item in Wiki

q. How to feedback the problem?

A. there are Q & An items in Wiki.

q. What are the advantages over lua open source projects in the industry?

A. CF changes users' habits more deeply than other lua development projects! Simplify the difficulty of the frame and make the frame black box transparent. There is no need to learn complex design patterns and concepts.

Q. what is the development philosophy of CF?

A. the goal of the CF project is not to compete, but to understand that simplicity is beautiful. When you get used to it, maybe you will become addicted.

Use the example

Container deployment

Rapid Construction of Development Environment by Dockerfile

Application examples in cf k8s

Initialization and use of cf web

Cf web Websocket Application Guide

Wonderful screenshot of cf web middleware development guide

hope

You may be using other development frameworks, but that doesn't stop you from pushing cf.

Maybe you are trying it, which does not prevent you from communicating your ideas with the author.

Maybe you are complaining about its shortcomings, please come to issue to complain.

Document and address

Project documentation

Project address

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.