Shulou(Shulou.com)06/02 Report--

I. basic concepts of WWW

WWW is the abbreviation of World Wide Web, which means the World wide Web. To understand what the World wide Web is, you need to understand the concept of hypertext. Hypertext is a kind of text used to display information, and this text can contain hyperlinks to other text, through which you can access other text associated with the text. in this way, the text associated with two or more text is called hypertext.

When a user visits a site, there are many links after opening a hypertext, and clicking a link can jump to other text, that is, the ability to jump back and forth between multiple texts, thus forming a web similar to the "spider web", called the World Wide Web, WWW, or Web.

WWW defines three important concepts, which are used to identify (describe) the location of a particular resource on the server (URL, Uniform Resource Identifier), the representation of information (HTML, HyperText Markup Language), and the way in which information is transmitted to users (HTTP, HyperText Transfer Protocol).

With URL, users can access the website directly through a browser, and the useful information provided on the site can be called "Web resources" or "resources". HTML is a hypertext markup language and a programming language for developing hypertext documents. HTTP is a protocol specification that specifies how information is transmitted between the Web client and the Web server.

As shown in the figure, users can type URL in the browser to access the page.

The hypertext file framework developed using HTML is as follows:

TITLE

Blabla... Blabla...

...

II. Introduction of HTTP protocol

HTTP is the abbreviation of HyperText Transfer Protocol, that is, Hypertext transfer Protocol, belongs to the application layer protocol. HTTP is a protocol specification written on documents, which specifies the way of network communication between the Web client and the Web server, so it is based on the architecture of CAccord S. Like other application layer protocols, HTTP protocol is to implement a specific application protocol, which requires programmers to develop programs that follow this protocol specification to achieve its functions. The same is true of the HTTP protocol, which is implemented in two ways, namely, the client-side implementation program and the server-side implementation program. The client-side implementation is mainly Web browser, including Microsoft's Internet Explorer, Mozilla Foundation's Firefox, Google's Google Chrome, Netscape's Navigator, Opera Software's Opera, Apple's Safari, etc., and the client's command line tools include curl, elink, etc., and the crawler is also a kind of Web client program. The server side is implemented as Web server, such as httpd, nginx, tomcat, IIS and so on.

III. MIME mechanism

Since the HTTP protocol is called Hypertext transfer Protocol, literally, HTTP can only transfer document files, while users should only get document resources on the browser. It is true that the early HTTP protocol could only send files in text format (hypertext is also plain text files), but later users were able to access images, audio, video, programs and other resources through browsers because HTTP borrowed the MIME extension in e-mail. MIME is the abbreviation of Multipurpose Internet Mail Extensions, which means multi-purpose Internet mail extension. For a long time, e-mail can only send files in text format. With the introduction of the MIME extension, e-mail can send data in the form of images, audio, video, programs, etc. HTTP draws on this and introduces the MIME mechanism, so you can transfer files in a non-text format through HTTP.

The main function of the MIME mechanism is to inform the client of the encoding format of the content of a resource, so that the client knows how to display the resource (perhaps with the help of tools or browser plug-ins on the operating system). This encoding format is marked by the multimedia resource type (Content-Type header), and the tag format is the primary type / secondary type (major/minor). Common MIME types are: text/plain, text/html, p_w_picpath/gif, p_w_picpath/png, audio/basic, video/avi and so on.

Resources in binary format (such as pictures, videos, etc.) need to be encoded and re-encoded into text format (you can use Base64 encoding), because the HTTP protocol can only transmit data in text format.

IV. The working mode of HTTP

Web clients (such as Web browsers) usually request access to Web resources on the Internet, while Web resources are stored on Web servers. Web clients can send HTTP requests to Web servers, while Web servers respond to the data requested by clients through HTTP.

For example, when we browse the Baidu page (http://www.baidu.com/index.html), the browser will send a HTTP request to the Baidu server www.baidu.com, and after receiving the request from the client, the Baidu server will go to its local disk to find the object (/ index.html) expected by the client. If the server finds the file, and the Web server process has permission to read the file. The data such as the object (/ index.html) and the attribute information of the object content (object length, object type, etc.) can be returned to our browser through the HTTP response. As shown in the picture.

Of course, a normal Web page is usually displayed after multiple requests / responses.

The HTTP communication process consists of the request sent by the Web client to the Web server and the response returned by the Web server to the Web client, while the request sent from the Web client to the Web server consists of the HTTP request message, and the response returned from the Web server side to the Web client consists of the HTTP response message.

5. HTTP transaction

A communication process consisting of a request and a corresponding response is called a HTTP transaction process.

VI. Web resources

Web resources are divided into static resources and dynamic resources.

① static resources: resources returned directly to the client without additional processing on the server side. Common static resource formats are .txt, .html, .jpg, .png, .gif, .js, .css, .mp3, .avi, etc.

② dynamic resources: resources that the server needs to send the running results of the program to the client by running the program. Common dynamic resource formats are .php, .jsp and so on.

The text, image, audio, video and other resources we usually get when we visit the website are static resources. They are a constant data stream stored on the server disk. When requested by the client, the server only needs to package and send the data to the client. On the other hand, dynamic resources need to generate results by running programs as needed, such as the search function we achieve when shopping on Taobao, as well as Internet search engines and so on.

The so-called Web server, that is, the host that stores Web resources (web resource), is the program implementation of the HTTP server, which is responsible for providing static resources requested by the other party to the request, or the results generated by the running of dynamic resources. These resources should usually be placed under a path of the local file system (the path is called DocRoot). On the Internet, these resources can be identified by URL.

7. The mechanism for browsers to speed up the opening of pages

We know that when we use a browser to access a page on a site, a page may display multiple resources, each of which needs to be requested separately. In order to speed up our access to the site, two acceleration mechanisms have been implemented on client browsers to provide a better user experience:

(1) the browser requests resources with multiple threads. When a user opens a browser, the browser starts multiple threads in the background, and one thread requests a resource.

(2) the browser caches static resources.

8. A complete HTTP request processing process

For the Web server side, after receiving a request from the client, the following processing is done.

Step 1: establish or process a connection-receive or reject a request.

Step 2: receive a request-the process of receiving a request for a particular resource from a host request message on the network.

The third step: processing the request-parsing the request message to obtain the resources and methods requested by the client and other related information.

Step 4: access the resources-get the requested resources in the request message.

Step 5: build the response message.

Step 6: send a response message.

Step 7: keep a log.

As shown in the picture.

Next, explain each step in detail.

8.1 establish or process a connection-receive or reject a request.

When a client initiates a TCP connection to Web, the Web server can establish a connection and add a new connection to the Web server connection list to monitor whether there is data transfer on the connection.

8.2 receive request-the process of receiving a request for a particular resource from a host request message on the network.

Now for high-performance Web servers, tens of thousands of connections may need to be established during peak periods, through which the server can communicate with clients around the world to exchange resources, and each client may open one or more connections to the server. In this regard, different Web servers will have different models for receiving requests, and the models for receiving requests mainly use the concurrent access response model, which has the following four types.

(1) single-process Istroke O model

The single-process I / O model starts a process to process user requests, and the Web server can only process one request at a time, and multiple requests are responded to serially. This structure (model) is easy to implement, but it can cause serious performance problems because multiple requests cannot be processed in parallel, so this model is only suitable for low-load Web servers.

(2) Multi-process Istroke O model

The multi-process Icano model starts multiple processes in parallel, and each process responds to a user request. The Web server of this structure (model) can create processes as needed, or it can create some idle processes in advance to wait for user requests. However, when the Web server needs to process thousands of requests at the same time, the number of processes required is extremely large, which consumes too many system resources. Therefore, many multi-process or multi-threaded Web servers limit the maximum number of processes / threads.

(3) the reused Istroke O model

The reused Icano model can respond to multiple requests with a single process, and it can be implemented in two ways:

① multithreading mode: one process generates n threads, and one thread processes one request.

② event driven (event-driven): a process responds directly to n requests.

In order to support a large number of connections, many Web servers use a reuse structure. In the multiplexing structure, each thread (multithreaded mode) or process (event-driven) needs to monitor the activity on all connections at the same time, and some state information needs to be recorded for each connection, such as request processing time, current processing phase and other data. Because for Linux, both processes and threads are very lightweight, there is not much difference between threads and processes on Linux, so there is not much difference between the multithreaded model and the multi-process Icano model.

(4) reused multi-process and multi-thread Icano model.

The reused multi-process and multi-thread Ithumb O model starts m processes and each process generates n threads to respond to client requests. The reuse multi-process and multi-thread Istroke O model combines multi-thread and reuse functions together, and makes full use of multiple CPU on the computer platform. So the number of requests that the Web server can respond to is MFN.

These four models are shown in the following figure.

8.3 processing the request-parsing the request message to obtain the resources and methods requested by the client and other related information.

In this process, the HTTP server process mainly analyzes the header of the request message, and the communication through the HTTP protocol is to negotiate how the client and the server exchange information according to the header of the HTTP request message and the header of the HTTP response message.

The header of the HTTP request message is structured as follows:

Headers

8.4 access resources-get the requested resources in the request message.

As mentioned earlier, the Web server is the host that stores Web resources, which is responsible for providing the client with the static resources requested by the other party or the results generated after running the dynamic resources requested by the other party. These resources are generally stored in a path of the server's local file system (that is, under the document root path DocRoot). When the server process obtains the information such as the resource and method that the client wants to request, it usually loads the resource on the local disk, so it initiates the disk IO (system call).

DocRoot is one of the resource path mapping mechanisms of Web servers. The resource mapping mechanism means that the Web server maps the client's request URL to a resource under a path on the local file system. For example, the client requests that you type on the browser

After http://www.baidu.com/index.html, the Web server can map to / var/www/html/index.html on the local file system after receiving the request, and the document root path DocRoot is the / var/www/html directory path.

Of course, the request URL can be either a directory or a file, and the above request URL is a file. What if the request is a directory? For example, the user enters http://www.baidu.com/ on the browser (if the last'-'is not added, it will be automatically added by redirection). In the case of Apache's httpd server program, the DirectoryIndex directive can be used in the httpd configuration file to define which page file in that directory should be returned to the client by default when the request URL is a directory. Here is an example.

DirectoryIndex index.html index.htm index.cgi

In this example, DirectoryIndex defines three page file names, and the Apache Web server looks for the three files in order, returning the second file if the first file exists, continuing to find the second file if the first file does not exist, and so on. If none of the files defined in the DirectoryIndex directive exist, the Web server lists all the url in this directory (including the file name, size of each file, last modified date and description, and links to each file), that is, the directory index file, which risks exposing the source code of the site. As shown in the picture.

Automatic generation of catalog index files can be disabled on the Apache Web server with the following instructions:

Options-Indexes

In addition, there are four resource path mapping mechanisms for Web servers:

(a) document root path docroot

(B) path alias alias

(C) docroot of the virtual host

(d) docroot of the user's home directory

8.5 build a response message.

The constructed response message includes the HTTP protocol version, the response status code, the reason phrase describing the status code, the response header and the optional response body.

8.6 send a response message.

Respond to the client with a built response message. In this process, it is necessary to restart the system call to send the HTTP response message (including the HTTP response start, the response header and the optional response body) through the layers of the TCP/IP protocol stack (transport layer header, IP header, MAC header), and then send it to the client through the communication line, in which the MAC header may be changed many times.

8.7 keep a log.

Logging is a very important part of Web server processing requests. It can record the status information of HTTP connections, which is convenient for later log analysis. Some websites (such as Taobao) can use logs to analyze the commodity sites that users often visit, and then analyze users' preferences, so that they can recommend specific types of products to users.

IX. Connection methods in HTTP request processing

9.1 persistent connection (Persistent Connection)

In the earlier version of HTTP, every time you finished HTTP communication, the TCP connection was disconnected. As shown in the picture.

However, this connection method is only suitable for early documents with small transmission capacity. With the popularity of HTTP, especially after the introduction of the MIME mechanism, a web page contains not only documents, but also a large number of pictures and other resources (web designers in order to make the page more popular), which makes each open a web page need to make a large number of requests to the server, and each request has to be TCP connection establishment and disconnection, increasing a lot of additional communication overhead, as shown in the following figure.

In order to save traffic, HTTP/1.1 and some HTTP/1.0 introduce persistent connections (Persistent Connection, also known as HTTP keep-alive or HTTP connection reuse), also known as persistent connections. Therefore, a connection that does not use the keep-alive mechanism is called a non-persistent connection, or a short connection.

Persistent connection (keep-alive) means that after a TCP connection is established, each resource will not be disconnected after it has been acquired, but will continue to wait for other resource requests. The characteristic of persistent connection is that TCP connection is maintained as long as disconnection is not explicitly proposed by either end.

The advantage of persistent connections is that they reduce the additional overhead (network bandwidth, CPU, memory and other system resources) caused by the establishment and disconnection of TCP connections; in addition, reducing that part of the time can make the HTTP request and response time end faster, so that the display speed of Web pages is improved accordingly.

However, if all the resources requested by the user's client browser have been obtained, but because of the use of keep-alive, the TCP connection has occupied the resources of the server until it is disconnected, which will also cause unnecessary resource consumption. In order to maximize the advantages of keep-alive while avoiding its disadvantages as much as possible, you can disconnect and control it from two dimensions:

① request time limit: the timing starts after the TCP connection is established. Once a certain period of time has elapsed, the TCP connection will be disconnected automatically, and the connection will be reestablished if necessary.

Limit on the number of resources requested by ②: the number of requested resources is counted after the TCP connection is established. Once the number of resources requested on a TCP connection exceeds a certain value, the TCP connection is automatically disconnected and re-established if necessary.

These are the two ways to disconnect keep-alive. In this way, it is generally the Web server, not the Web client, that is the first to actively disconnect the TCP connection. Even so, using keep-alive as a connection method still has its side effects. For servers with large concurrent visits, using a long connection mechanism will make some subsequent requests unable to get a normal response. For example, if a Web server can only establish a maximum of 10, 000 TCP connections at the same time, subsequent connection requests will not be established when the 10, 000 connections are full and all are keep-alive connections. Therefore, the compromise is to use a shorter duration of persistent connections (request time limit) and a smaller number of requests (limit on the number of resources requested).

In HTTP/1.1, all connections are keep-alive connections by default, while in HTTP/1.0, keep-alive has not been standardized, but some servers or browsers have implemented keep-alive in a non-standard way. However, in order to use keep-alive connections for HTTP communication, both the client and server sides must support keep-alive.

If the browser supports keep-alive, it adds the following to the header of the HTTP request message:

Connection: Keep-Alive

When the server receives the request, if the server supports keep-alive, it will add the following to the header of the HTTP response message:

Connection: Keep-AliveKeep-Alive: timeout=10, max=500 (this is an example, parameter values can be changed)

When the server wants to disconnect explicitly, specify the value of the Connection header field to be close:

Connection: close

9.2 pipelining (HTTP pipelining)

HTTP/1.1 allows the optional use of pipelining (pipelining) on persistent connections, so the use of pipelining technology must ensure a persistent connection between the two parties, which is another performance optimization over keep-alive connections. Before the advent of pipelining technology, the client had to wait and receive a response from the server after each request was sent before it could send the next request. With the advent of pipelining technology, the client can send the next request without waiting for a response from the server.

Simply put, pipelining allows the client to send multiple requests to the server in parallel without waiting for a response one by one. This can save the time of network loopback and improve performance. In the high latency network environment, and when the number of requested resources is large, the effect of using pipelined acceleration is more obvious. In the broadband environment, the acceleration effect is not obvious.

Pipelining is allowed for request methods such as GET and HEAD, but not for non-idempotent request methods, such as POST, because in the event of an error, the client may not know which of a batch of pipelined requests is being processed by the server.

State management using Cookie

HTTP is a stateless protocol (stateless), so when communicating over HTTP, the server cannot continuously track the source of visitors. That is, HTTP itself cannot manage the status of previous requests and responses.

When a Web page requiring login authentication cannot manage the status of previous requests and responses (without recording the logged-in status), visitors need to re-login authentication every time they refresh the page or jump to another page on the server.

Of course, HTTP as a stateless protocol also has its advantages. Because there is no need to save state, it naturally reduces the consumption of resources such as server CPU and memory. In addition, because the HTTP protocol itself is very simple, it is suitable for a variety of application scenarios.

However, for websites that need to continuously track the source of visitors, such as large shopping sites, forums, etc., it is necessary to manage previous requests and corresponding status. Therefore, other mechanisms are introduced to track users, such as cookie, url mechanism and so on. Cookie is usually implemented with session mechanism. Cookie and session are important technical means to track and save user access behavior in dynamic sites, and they are the best way to identify users and achieve persistent sessions.

Cookie means "cookie" or "small text file". Cookie was originally developed by Netscape, but now all major browsers support it. According to the different storage location of cookie in the client, it can be divided into: memory cookie and hard disk cookie. The memory cookie is maintained by the client browser, which records the behavior and preferences of the user when visiting the current site. When the user exits the browser, the memory cookie is deleted, and the memory cookie, also known as the session cookie; hard disk cookie, is stored on the hard disk and still exists when the browser exits or the computer restarts. The hard disk cookie is also known as persistent cookie. Therefore, according to the different storage time of cookie, it can be divided into: session cookie (non-persistent cookie) and persistent cookie.

11. Session Management and Cookie Application

As mentioned earlier, HTTP cannot manage the status of previous requests and responses, that is, it cannot track the source of visitors persistently, so we need to introduce Cookie to manage Session to make up for the state management capabilities that are not available in HTTP.

11.1 the concept of Session

Session is a tiny data structure specially maintained by the server for each browser process, which records the user's previous visit to the current website. According to the Cookie sent by the client, the server is associated to a specific Session to track the user. On the server side, a certain amount of memory space can be used to save the behavior of a user browsing the website before.

11.2 the working mechanism of Cookie and Session

When the user visits the website for the first time, the server will send a Cookie ID to the client; later, when the same user visits the same website, the server will bring his own Cookie logo, and the server will find and associate it to a local Session according to the Cookie ID sent by the client to know which user visited the website.

The Cookie here can be understood as "token". After we live in the community, we need to apply for several community cards similar to tokens to the local community property, each person uses a community card, and then we can use this card to get in and out of the community. The property here is actually the equivalent of a server.

11.3 Session Management and Cookie status Management steps

Step 1:

The client puts the login authentication information such as user ID and password into the main body of the request message, which is usually sent to the server by POST request method. At this time, HTTPS is usually used to send the form interface to the client and the login authentication information filled in by the client to the server.

Step 2:

The server sends out the Session ID that identifies the user. In this step, the server authenticates the login authentication information sent by the client, binds the authentication status to Session ID and saves it locally on the server. Then, when responding to the client, the Set-Cookie header is added to the response message, and the Session ID (for example, 0019e0f) is recorded in the header value. You can think of Sesssion ID as an identification code used to distinguish between different users.

Step 3:

After receiving the Session ID sent by the server, the client saves it locally as a Cookie. When subsequent users visit the server, the browser will automatically send the Cookie header to the server, so the Session ID will also send it to the server. The server looks up and associates to the local Session based on the Cookie sent from the client, so that it knows which user accessed it.

In addition, in the second step, because the Session ID is easy to be camouflaged, stolen, and guessed, it should be given as an unpredictable string, and the server also needs to manage the validity of the Session ID to ensure its security.

XII. HTTP protocol version

HTTP/0.9

HTTP/0.9, launched in 1991, is a prototype version with rudimentary functions. HTTP/0.9 only supports GET methods, and does not support MIME types, various HTTP headers, and version numbers of multimedia content. HTTP/0.9 was originally designed to transfer documents.

HTTP/1.0

HTTP/1.0 is the first widely used version.

① supports many methods (method), including GET, POST, HEAD, PUT, DELETE, TRACE, OPTIONS, and so on.

② supports caching (cache) mechanism, but it is relatively weak.

③ supports the MIME type of multimedia.

HTTP/1.1

In particular, the caching function is enhanced.

HTTP/2.0

Drawing lessons from the spdy protocol developed by Google, HTTP/1.1 has been improved in many aspects, so that its performance has been greatly improved.

XIII. Web page development technology

The three technologies that front-end developers must master are HTML, CSS, and JavaScript (js for short).

① HTML

HTML (HyperText Transfer Markup Language, Hypertext markup language) is used to modify documents and can define the presentation of document information. HTML embellishes a document by adding a specific string tag to a part of the document. Usually the Web page we browse is almost written by HTML, and the browser can parse and render the HTML document sent by the server, and finally display the Web page.

HTML instance:

Example.com.logo {padding: 20px; text-align: center;}

Example.jp

② CSS

CSS (Cascading Style Sheet, hierarchical (cascading) stylesheet) is used to define the style of some content in a web page, which can specify how to display the various elements in the HTML document. Therefore, even if the same HTML document has different styles defined by CSS, the appearance displayed is different.

CSS instance:

.logo {padding: 20px; text-align: center;}

③ JavaScript

JavaScript is a client script development language, abbreviated as JS. JS acts as a client script, that is, you need to download the content to the client browser, run it on the client browser engine and display the results. Therefore, the source file needs to be transferred from the server to the client, and the client is responsible for execution.

JavaScript instance:

Var content = escape ([xss_clean]); [xss_clean] ("

")

There are two development techniques for Web services:

(1) client technology: js and other client scripts, which need to be run on the client.

(2) Server technology: php, jsp and other server scripts, which need to be run on the server.

XIV. Reference materials

HyperText Transfer Protocol Version 2 (HTTP/2)

HyperText Transfer Protocol-HTTP/1.1

"HTTP: The Definitive Guide", by David Gourley / Brian Totty

"graphic HTTP", by Ueno Xuan

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.