Shulou(Shulou.com)06/03 Report--

For more information on cloud native technology, please follow Alibaba Cloud native technology circle.

What is Puppeteer?

The official website of puppeteer is as follows:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium.

The popular description is that Puppeteer can run Chrome or Chromium in a * mode (of course, it can also be run on a server with an interface), and then it can control the behavior of the browser through code. Even if it is run in a non-interface mode, Chrome or Chromium can correctly render the content of the web page in memory. So what can Puppeteer do?

Generate screenshots of web pages or PDF capture SPA (Single-Page Application) for server rendering (*) advanced crawlers that can crawl a large number of asynchronously rendered web pages to simulate keyboard input, form automatic submission, login pages, etc., to achieve UI automated testing to capture the timeline of the site, in order to track your website and help analyze website performance problems.

In this paper, we choose the screenshot scene as the demonstration.

No more nonsense, let's directly show you how to use functional computing products to quickly deploy a Puppeteer Web application.

How to quickly deploy a distributed Puppeteer Web application?

In order to rapidly deploy distributed Puppeteer Web applications, this paper takes the function computing service as an example.

Function computing (Function Compute): function computing is an event-driven service. Through function calculation, users do not need to manage the operation of the server, but only need to write code and upload. The function calculation prepares the computing resources and runs the user code in an elastic way, while the user only needs to pay according to the resources consumed by the actual code. Function to calculate more information.

With the function computing service, our goal here is to build a distributed application, but what we do is very simple, that is to write the business code and deploy to the function computing, that's all.

After calculating using the function, the architecture of our system is as follows:

Effect demonstration

You can view the results directly through the following link: https://1911504709953557.cn-hangzhou.fc.aliyuncs.com/2016-08-15/proxy/puppeteer-test/html2png/?url=https://www.aliyun.com/product/fcPS: the first request may have a few seconds of cold start time, and the cold start can be completely removed by using reservation mode.

Build steps:

The overall process is shown in the following figure:

Among them, the only commands we need to operate are Fun Init, Fun Install, and Fun Deploy, and the contents of each step are automatically completed by these three commands.

1. Tool installation

Install the Fun tool:

It is recommended to download the binary executable program directly from here and use it directly after decompression. Download address.

Install Docker:

You can install it according to the method described here.

two。 Initialize the project:

Using the Fun tool, you can quickly initialize a scaffolding for a Puppeteer Web application using the following command:

Fun init-n puppeteer-test http-trigger-node-puppeteer

Where-n puppeteer-test represents the directory name of the initialization project, and http-trigger-node-puppeteer represents the name of the template to be used, which can be omitted. After omitting, you can select the desired template from the list prompted by the terminal. After execution, you can see the following directory structure:

. ├── index.js ├── package.json └── template.yml

Compared with traditional puppeteer applications, there is only one more template.yml file to describe the resources of function computation. Index.js is our business code. You can write your own business code in accordance with the requirements of the Puppeteer official help document. Without repeating it here, the core code is as follows:

Const browser = await puppeteer.launch ({headless: true, args: ['--no-sandbox','--disable-setuid-sandbox',]}); const page = await browser.newPage (); await page.emulateTimezone ('Asia/Shanghai'); await page.goto (' https://www.baidu.com', {'waitUntil':' networkidle2'}); await page.screenshot ({path:'/ tmp/example', fullPage: true, type: 'png'}) Await browser.close ()

The package.json content is as follows:

{...... "dependencies": {"puppeteer": "^ 2.0.0"},.}

As you can see, the dependency of puppeteer is declared in package.json. This is also the standard practice when we use node development, and there is nothing special about it.

3. One-click installation dependency

Puppeteer installation, even on traditional linux machines, is not so easy. Because puppeteer itself depends on a lot of system libraries, which system libraries to install and how to install these system libraries have become a headache.

Fortunately, the function calculation command line tool Fun has integrated the Puppeteer solution, as long as the package.json contains puppeteer dependencies, and then use fun install-d to install all system dependencies with one click.

Fun install-d4. Run and debug functions locally

The local running and debugging methods of Puppeteer are exactly the same as those described here, so we will not repeat them. We will only demonstrate the running effect here:

5. One-click deployment of applications

Basically all FaaS platforms set code package restrictions in order to reduce the cold start of applications, and function calculation is no exception. Puppeteer itself has reached about 350m, along with its system dependency has reached 450m. How to deploy the 450m function to the FaaS platform is a headache and tedious problem.

Fun, the command-line tool for function computation, now natively supports this large dependency deployment (version 3.1.1 only supports node runtime). No additional action is required, just fun deploy:

$fun deploy

Fun automatically completes the deployment of dependencies. When it is detected that the packaging dependency exceeds the platform limit, it will go to the configuration wizard to help the user configure automatically.

The path we recommend here is when prompted whether Fun will automatically help NAS manage yes, enter yes, and then prompt whether to use NasConfig: Auto to automatically handle NAS, and then choose Yes, then you don't need to do anything else and wait for the function to be deployed successfully.

If you have other requirements, such as if you want to use your existing NAS service, you can enter no when prompted to use NasConfig: Auto, which will lead you to the appropriate process. For more detailed instructions, please refer to the FAQ below.

What does Fun do when FAQ installs puppeteer?

Puppeteer itself is a npm package, and its installation is very simple, via npm install. The problem here is that puppeteer relies on chromium, while chromium relies on some system libraries. So after npm install, it will also trigger the operation of downloading chromium. The problems often encountered by users here are as follows:

Due to the large size of chromium, we often encounter network problems that lead to download failure. Npm only downloads the system libraries that chromium,chromium depends on and does not install them automatically. Users also need to find the missing dependencies themselves to install.

The main optimizations made by Fun are:

By testing the network environment, for domestic users, it will help configure Taobao NPM image to accelerate download. Automatically installs dependent libraries missing from chromium for users. How does Fun deploy large dependencies to function calculations? Isn't there a limit on the size of the code package?

Basically all FaaS in order to optimize the function cold start, will add the function code package size limit. Function calculation is no exception. However, through the built-in NAS (Ali Cloud File Storage) solution, Fun can help users create, configure and upload NAS to NAS with one click. At run time, function calculations can be automatically read from NAS to functional dependencies.

To help users automate these operations, Fun has a built-in wizard (only node is supported in version 3.1.1, and more will be supported later. Github issue is welcome to raise requirements). When it is detected that the size of the code exceeds the platform limit, it will prompt whether Fun will transform it into a NAS solution. The logic of the wizard is as follows:

Ask if you want to use Fun automated configuration NAS to manage dependencies? (if the answer is yes, enter the wizard, answer no, and continue the publishing process) detect whether NAS has been configured in the user's yml. If so, prompt the user to select the configured NAS storage function dependency. If not, prompt the user whether to use NasConfig: Auto to automatically create NAS configuration. If yes is selected, it will help the user configure nas and vpc resources automatically. If no is selected, the NAS resources already available on the user's current NAS console will be listed. No matter which method is used above, the NAS will eventually be generated in template.yml and the VPC-related configuration will be detected according to the language, such as node runtime. The node_modules and .fun / root directories will be mapped to the nas directory (implemented through .nas.yml) to automatically execute fun nas sync to help users upload local dependencies to the NAS service to automatically execute fun deploy, and to help users upload the code to the function calculation prompt help. For HTTP Trigger, prompt the function Endpoint, open the browser directly to see the effect can you specify the version of puppeteer?

Yes, you just need to modify the version of puppeteer in package.json and reinstall it.

Is there a way to change the UTC used in the time zone in the example of function calculation to Beijing time?

The display effect of some web pages is linked to the time zone, and different time zones may lead to differences in the content displayed. Using the methods described in this article, it is very easy to use the latest version of puppeteer, while the latest version of puppeteer provides a new API page.emulateTimezone (timezoneId) that can easily modify the time zone.

If subsequent versions of Puppeteer are updated and rely on more system dependencies, is the method described in this article still applicable?

Fun has a built-in .so missing detection mechanism. when debugging and running locally, it will intelligently identify the missing dependent library according to the error report, and then accurately give the installation command, which can be installed with one click.

If a new dependency is added, how to update it?

If you add a new dependency, such as a new dependency library in the node_modules directory, you just need to re-execute fun nas sync to synchronize.

If you change the code, you just need to redeploy using fun deploy. Because large dependencies and code are separated through NAS, dependencies usually do not need to change frequently, so the frequency of calls is relatively low, while that of fun deploy does not have large dependencies

In addition to the methods described in this article, are there any other ways to install puppeteer with one click?

Fun provides a lot of dependency installation methods. In addition to declaring dependencies directly in package.json and then installing them through fun install-d, there are many other methods, all of which have their own applicable scenarios:

Imperative installation. Such as fun install-f functionName-p npm puppeteer. The advantage of this installation is that even users who don't know anything about fun can use it foolishly. Declarative installation. The advantage of this installation is that it provides a Dockerfile-like experience, where most of the Dockerfile instructions can be used directly. Dependencies declared in this way can be submitted directly to the version repository. After others pull the replacement code, you can also install all dependencies with one click. Interactive environment installation. The advantage of this installation method is that it provides an installation experience similar to that of traditional physical machines. In an interactive environment, most linux commands are available and can be kept on trial and error. Summary

This paper introduces a relatively simple method to build distributed Puppeteer Web services from scratch. Using this method, you can smoothly complete the deployment without caring about how to install the dependency or how the relationship uploads the dependency.

After the deployment is completed, you can enjoy the advantages of function calculation, namely:

Without the need for infrastructure such as procurement and management servers, we only need to focus on the development of business logic, which can greatly shorten the project delivery time and labor cost, provide rapid troubleshooting and maintenance-free operation and maintenance, such as log query, performance monitoring, alarm and other functions. Millisecond scale-up, rapid bottom expansion to cope with peak pressure, excellent performance and competitive cost

"Alibaba Cloud Native focus on micro-services, Serverless, containers, Service Mesh and other technology areas, focus on cloud native popular technology trends, cloud native large-scale landing practice, to be the best understanding of cloud native developers of the technology circle."

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.