Shulou(Shulou.com)06/01 Report--

This article shows you how to use the full stack of JavaScript to build business-grade applications. The content is concise and easy to understand, and it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

What is a sparrow?

Yinque is a professional cloud knowledge base, which provides unique knowledge management for individuals and teams, and creates easy and smooth work collaboration. It provides editing capabilities for online documents (rich text, tables, design manuscripts, etc.) in various formats. Support real-time online multi-person collaborative editing, data cloud preservation is not lost. The biggest difference between Voice and other documentation tools is that it organizes documents through a knowledge base to enable knowledge creators to better manage knowledge.

The prototype stage of the evolution of Finch technology architecture

The Finch was born in 2016, when Ant Financial Cloud needed a tool to carry its documents. The technical students in charge at that time used their spare time to build this documentation tool. At the beginning of the project, there is no personnel and resources to support, but also in order to quickly verify the prototype, the technology selection chose the lowest cost solution.

The underlying services are based entirely on the BaaS services and container hosting platform provided by the experience Technology Department:

Object service: a MongoDB-like data storage service

File service: a file storage service encapsulated on Aliyun OSS

DockerLab: a Container managed platform

These services and platforms are implemented based on Node.js and are designed for internal innovative applications, and it is precisely because of these internal services that reduce the cost of innovation that provide engineers with a better innovation environment.

The server side of the application layer naturally chooses the open source Node.js Web framework Egg (the encapsulated Chair inside the ant) of the experience Technology Department, and implements the server side through a single Web application. The application layer client also chooses the React technology stack, combined with the internal antd, and uses CodeMirror to achieve a powerful and elegant markdown online editor.

At this time, it can be regarded as the "prototype phase" of the Finch, which is just an amateur project of an engineer, using internal BaaS services for innovative applications and a series of open source technology solutions to verify the product prototype of online documentation tools.

PS: I wasn't on the Whisperbird team at that time, but coincidentally, I was providing BaaS services such as Object and File and Egg.js Web framework support to Whisperbird.

Internal service phase

As the online documentation tool has been recognized within the team, the goal of the Finch is not only the documentation tool of the financial cloud, but also to replace competitive products such as confluence to become a knowledge management platform for 100, 000 employees within Ali. To target knowledge creators, providing only Markdown editors will certainly not enable non-technical people to use Whisperfinch more efficiently. Although many true love fans began to learn and even fell in love with Markdown, we are duty-bound to step into the pit of rich text editors. At the same time, unlike rich text editors such as Word, we choose a more "Web" route, adding special functions such as formula, text drawing, mind mapping and so on. With the continuous exploration of Finch in the field of knowledge management, the three-tier structure of knowledge management (team, knowledge base, document) began to take shape. On this basis, the functions of collaboration, sharing, search and message dynamics are becoming more and more complex. Relying solely on BaaS services can no longer meet the business needs of Voice.

In order to meet the challenges brought by business development, we mainly carry out the transformation from the following points:

Although BaaS services are simple to use and low in cost, the functions they provide are not enough to meet the development of Finch business, and there are also deficiencies in stability. So we replaced the underlying services from BaaS with internal IaaS services (MySQL, OSS, caching, search, etc.).

The Web layer still adopts the Node.js and Egg framework, but the business layer begins to become a large single application by learning from the practice of the rails community. By introducing ORM to build the data model layer, the layering of the code is clearer.

The front-end editor migrates from codeMirror to Slate. In order to better realize the function of the Finch editor, we have carried out in-depth development of fork Slate and customized an independent content storage format to provide more efficient data processing and better compatibility.

In the internal service stage, the Finch has become a formal product, which is no different from other projects of ants. Through the internal training in Ali, the product form of the Finch is basically finalized.

Commercialization stage

With the increasing internal influence of the Finch, some Ali alumni who left to start a business began to find Yu Bo: "the Finch is very useful. Have you considered making it available to outside companies after commercialization?" After half a year of gestation and reconstruction, at the beginning of 18, the Finch began to provide services to the outside world for commercialization.

When an application comes out of the company and into a commercial environment, the technical challenges it faces suddenly become greater. The function of the core part of knowledge creation management is becoming more and more complex. With the addition of new formats such as tables and mind maps, the demand of multi-person real-time collaboration poses a higher challenge to the editor technology. In order to better serve corporate users and individual users, Yuque has also invested a lot of energy in corporate services, membership services and so on. With the rapid development of business, service commercialization also puts forward higher requirements for quality, safety and stability.

In response to the development of the business, the architecture of the Finch has evolved:

We completely put the underlying dependence on the cloud, and all migrated to Aliyun, which not only provides basic storage and computing capabilities, but also provides richer advanced services, while ensuring stability.

Rich cloud computing basic services ensure that the server can choose the storage, queue, search engine and other basic services that are most suitable for the Finch business.

More artificial intelligence services have brought more possibilities to the Finch's products, including OCR map recognition, intelligent translation and other services, which have been directly transformed into the characteristic services of the Finch.

In the application layer, the server of Voice is still based on a large-scale Node.js web application based on Egg framework. However, as there are more and more functions, some relatively independent services are also being detached from the main service, and these services can be divided into several categories:

Micro-service: for example, multi-person real-time collaborative service, because it is relatively independent and long-connected service is not suitable for frequent release, so we split it into a separate micro-service to maintain its stability.

Task services: a large number of local file preview services such as Finch will result in some tasks that consume more resources and depend on complex ones. By stripping it from the main service, we can avoid the impact of uncontrollable dependency and resource consumption on the main service.

Function calculation class: tasks such as plantuml Preview and mermaid Preview are not sensitive to response time, and dependencies can be packaged into Ali Cloud function calculation. We will run it in function calculation, which saves money and is safe.

As the editor becomes more and more complex, there are more and more problems in developing on the basis of slate. Finally, the Finch embarked on the road of self-research editor, realized the rich text editor based on browser contenteditable, the table editor through canvas, and the mind map editor through SVG.

For an introduction to the rich text editor, take a look at the sharing of long Hao, the father of Lake Editor: the technological evolution of the rich text editor.

This stage (and the current stage) of Whisperfinch is the commercial stage, but we still maintain a very small team to develop through the full stack of JavaScript. The underlying services are fully available on the cloud, with the help of cloud services to create the unique features of Whisperfinch. At the same time, knowledge creation and management tools are provided for enterprise users and personal knowledge workers.

JavaScript full stack

On social networks, people seem to have a negative view of the whole stack of JavaScript. "everything is good, everything is loose" may be your first impression when you hear the term "full stack engineer". So why did the Finch choose the direction of the full stack of JavaScript?

JavaScript full Stack and Product engineer

At Whisperfinch, we do not define an engineer who develops with the full stack of JavaScript as a full stack engineer, but a "multi-purpose" product engineer:

They are the "technical partners" of the product, they have an owner sense of the product, participate in the product discussion and design with the product manager, make suggestions on the product design plan from a technical point of view, independently complete the full stack research and development of the product function, and track the product results after release.

At the same time, they are also domain experts in a certain technical field, for example, someone may be an expert in the server side, an expert in the testing field, an expert in the front-end build field, and an expert in the CSS field. They can use their professional domain knowledge to optimize the team R & D tool chain and improve the efficiency of product development.

At the Finch, the product development process for product engineers is as follows:

In the product design stage, the product engineer will participate in the discussion and eventually produce a product design draft of final design. As the early product engineers participate in the full discussion, generally speaking, the product design draft decided here will not encounter technical problems in the later research and development process.

* * then a documented system analysis and design will be carried out on the Finch. * * an asynchronous review will be initiated on the whisperer. Some large technical solutions will be reviewed by experts from other fields to ensure that all the technical difficulties are sorted out clearly.

After the design of the system is clear, it enters the research and development stage.

For all code, automated test coverage is required. Full coverage of unit tests is required for all new code and modified business logic, as well as end-to-end testing for the functions of critical links. Writing automated tests is a necessary process before entering code review.

Asynchronous code review will be initiated after the periodic function development and test writing are completed. The person in charge of the relevant business and some corresponding domain experts will be invited to conduct code review. Code review is carried out from the perspectives of correctness, security and maintainability of business logic.

Finally, when the release is launched, we must follow the three principles: grayscale, emergency, and monitoring. Avoid the bug that the function change may bring to affect a large number of users.

How does the Finch conduct a full-stack JavaScript test? Interested students can take a look at the sharing of Big Niu Dafeng in the big front-end automated test of the Finch team: the thinking of the big front-end test and the practice in the language Finch.

Through the full stack of JavaScript, Voice Finch team can complete product research and development more efficiently and with high quality:

At the code level, there is a lot of code that can be reused, such as the editor, which can be used not only on the Web side, but also on the desktop side. At the same time, many data processing capabilities can also be used on the server side.

In terms of product R & D efficiency, full-stack R & D reduces a lot of communication costs, which is very efficient in the current stage. The JavaScript full stack avoids developers switching between different languages, regardless of what lodash / moment and other tool classes used in the front end should be used in other languages, which greatly improves the research and development efficiency of the whole stack.

Finally, from the engineer's point of view, full-stack R & D gives engineers the opportunity to participate deeply in the whole process of product research and development, and people will spontaneously think about what optimization points the product has and what it can help the product to do in terms of technology. For example, the new OCR search function of Voice is that the whole stack engineer spontaneously completes the whole product optimization from technical pre-research to product landing.

JavaScript full Stack and Node.js

When it comes to the full stack of JavaScript, one technology that can't be bypassed is Node.js. When running as a server that is closely integrated with the front end, it basically becomes the spokesman of the whole stack. So is Node.js really a language suitable for large-scale commercial projects? Everyone has a lot of doubts about it:

In fact, with the development of the JS language, many problems have been solved, such as the emergence of Async Function, which allows developers to write asynchronous code synchronously, making it easier to understand and easier to handle exceptions. At the same time, with the further improvement of the community, a large number of high-quality tool modules and frameworks emerge. The server part of the Finch is based on the Egg framework and has integrated a large number of modules and services needed for Web development. At the same time, the programming model based on Async Function is also more simple. The emergence of TypeScript has also dispelled many people's misgivings about the large-scale project development of JavaScript. In addition, there are other ways to ensure code quality and maintainability (even a pure JavaScript project without a single line of TypeScript code).

The first thing the Finch does is to determine the boundary between the core system and the external system. Through the hexagonal architecture (also known as the port adapter architecture), we fix the interaction between the core system and the external system and the user. Through the "port" form, to determine the input and output. The external system connects the system to the exposed port through the "adapter", which only needs to be implemented according to the definition of "port", and the external system can be replaced freely.

In this model, Controller is the HTTP adapter that the Finch exposes to the user interface. In Controller, we do a format checksum conversion on the user request parameters, check the user permissions, and format the output.

We define the interaction between Finch and third-party platforms and services (usually a series of methods), encapsulate different services in different environments into a unified method through adapters, and record the call log during the call.

The data model layer is the Model of the data layer. Take the Doc model as an example, its meta information data is stored in MySQL, while the document body data is encrypted and stored in OSS. For the core business logic, it is completely unaware of where the underlying storage is. Furthermore, as long as Voice uses SQL to interact with the database, the underlying data can be seamlessly migrated to other databases that support full SQL syntax, such as OceanBase, and even minor modifications can be encapsulated in the Model layer.

Finally, take a document release as an example, the user interacts with Voice by calling the HTTP interface, and the data is written to the storage through the Model layer, including MySQL and OSS, to update the document cache. At the same time, send asynchronous messages to other systems, trigger the nail WebHook, and synchronize the data to the search engine. These interactions with external systems are encapsulated by adapters to perform their own functions, parameter conversion, authority verification and logging, which not only ensure the simplification of core logic, but also make system call link tracking easier.

Hybrid application architecture

When the system is developed to a certain extent, should we continue to add functions to large monomer applications, or split into micro-services? Since these two architectures exist, they must have their own advantages and disadvantages, and the specific choice of architecture form should be determined by the current business scale and team distribution. Therefore, the technical architecture of the Finch has become a hybrid technical architecture with the business form of the Finch.

The main service of Voice is a large Node.js service, which centralizes all the application business logic. In addition to the main service, there are some other services in different forms.

Microservices: some independent and stable functional modules, or services with additional deployment architecture requirements, will be deployed independently in the form of microservices, and systems will temporarily interact through HTTP interfaces. For example, real-time collaborative service is deployed as an independent micro-service because it is relatively independent and stable, and it is a long-connected service, so it can not be released and restarted frequently.

Task clusters: some CPU-intensive tasks, or relying on complex third-party dependent services, are put into a separate task cluster. For example, various file preview services may rely on other services and require a lot of computational costs, so it is most appropriate to put them in a task cluster after eliminating concurrency through queues.

Function calculation: for some services with high response time and can be functionalized, we will try our best to migrate to Aliyun's function computing, such as plantuml, mermaid and other text drawing services.

Take the rendering of mermaid as an example. The user enters a piece of mermaid code to call the Finch, which calls a function deployed in the Ali cloud function to calculate, and runs puppeteer in the function to render svg to return.

Why did you take Serverless out and say it alone? Remember when we said that Node.js is single-threaded and not suitable for CPU-intensive tasks? Due to the emergence of Serverless, we can migrate these tasks that have security risks and consume a lot of CPU computing to function computation. It runs in a sandbox environment without worrying about the security risks caused by users' malicious code, while stripping these CPU-intensive tasks from the main service to avoid blocking the main service when concurrency occurs. Pay-on-demand can also save a lot of money, eliminating the need to deploy a resident service for low-frequency scenarios. Therefore, we will try our best to migrate all these services to Serverless (such as Ali cloud function computing).

A general domain other than language

In addition to language, there are more aspects to consider in any commercial system, the two most important of which may be security and stability.

There are all kinds of security risks in the dependence of a system from the front end, the server side to the bottom layer:

Front-end security risks: XSS, jump fishing, cross-site requests, etc.

Server security risks: horizontal privilege issues, unauthorized access, disclosure of sensitive information, SSRF, SQL injection, etc.

Security risks of cloud services: SMS / email bombardment, risk of data leakage, content security, etc.

There are basically no silver bullets to solve these security problems, and they can only be dealt with individually, but there are some basic principles:

Do not trust any input from the user

Any place that renders rich text needs to guard against XSS, and the content may not be entered through IDE

If you want to execute the user's code on the server, you must put it in a sandbox.

To request the resources passed by the user from the server, it must be filtered by SSRF

Precipitate the standard coding paradigm to deal with security risks, and need to focus on Code Review

All interfaces must have permission to verify

Response serialization method to filter sensitive information

Splicing of SQL is not allowed

Since the beginning of commercialization, Finch has worked closely with the security team, from internal security awareness training and internal security team testing to internal red and blue attack and defense, external white hat penetration testing, security is a protracted war.

In order to ensure the stability of Voice, we have done a lot of work from the front end to the server and cloud services. like security, stability is a long-term project from front to back. The guarantee of the stability of the sparrow is mainly in two dimensions:

Ensure service availability: from the architecture design to eliminate a single point, the underlying data need to be disaster recovery and backup, services need multi-unit, availability zone deployment. And avoid introducing unnecessary strong dependencies.

Exception monitoring and traceability: from the front-end business burial log, exception log monitoring, to the server-side full-link log tracking and collection, system performance monitoring and analysis. In the end, we can achieve that exceptions can be perceived and traced in time, and performance problems can be located and analyzed.

What does it mean to avoid introducing unnecessary strong dependencies? In the case of Whisperer, for example, MySQL is a strong dependency that cannot be removed, and cache should not be a strong dependency, but the earliest Whisperer's session is stored in Redis. Once there is a problem with the Redis cluster, users cannot log in due to the inability to obtain user information. This turns the cache into a strong dependency. So when we put the session storage in MySQL, Redis becomes a weak dependency, and the system still works when it dies. Another example is that some time ago, Voice launched the function of multi-person real-time collaborative editing, and before this function was launched, it was through document locking to prevent multiple people from editing the same document at the same time. However, multi-person real-time collaboration introduces another service. Once the real-time collaborative service is dead, the user cannot edit the document, and it becomes a strong dependence of the Voice system. In order to solve it, we automatically switch to the old lock mode when the user fails to connect to the collaborative service. In this way, collaborative services have also become a weak dependence of Voice Finch.

How does the Finch choose the Technology Stack

The lark has developed step by step over the past few years, and the technology behind it has been evolving, but it has always followed several principles:

The selection of technology stack should match the product development stage. Products have different requirements for technology at different stages. The earlier the stage, the higher the requirements for iterative efficiency. After commercialization, the requirements for stability and performance will become higher. You don't need to use the most advanced technical solutions in the first place, but you need to consider and weigh them with the product phase.

The selection of technology stack should be combined with the technical background of team members. The reason why Voice chose the full stack of JavaScript is that the team that incubates Finch is mostly programmers with JavaScript background. At the same time, Node.js is a first-class citizen in Ant, and the supporting facilities are relatively perfect.

The most important point is that no matter which technology stack you choose, security, stability, and maintainability (extension) should be carefully considered. What language and service will change, but these basic security awareness, stability awareness, and how to write maintainable code are important factors that determine the long-term development of the project.

"Alibaba Cloud Native focus on micro-services, Serverless, containers, Service Mesh and other technology areas, focus on cloud native popular technology trends, cloud native large-scale landing practice, to be the best understanding of cloud native developers of the technology circle."

The above is how to use the JavaScript stack to build business-grade applications. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.