Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about the idea of .net distributed system architecture, which may not be well understood by many people. in order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Recently, I have seen some recruitment messages, asking candidates to talk about the idea of distributed system architecture. This morning happens to have some time, I also sort out the evolution of our actual. Net website architecture, just some of my own ideas, you are welcome to criticize and correct.

First of all, it is stated that there is less open source content under .net, and it is not to do basic services such as parallel database, so we will not consider Hadoop, Spark, ZooKeeper, dubbo and so on here.

First, in the initial hypothetical website, we put the application website, file and database on one server, and one server dominates the world.

Second, with the expansion of business, a server can not meet the performance requirements, so applications, databases and files are deployed on different servers, and different hardware is configured according to the use of the server to achieve the best performance.

Third, with the expansion of business, a database, website, file server no matter how high performance can not be a large number of data processing, high concurrent user access, we must consider using the cluster mode.

1. The application server, as the entrance to the website, will bear a large number of requests, and we often share the number of requests through the application server cluster. The load balancing server deployed in front of the application server dispatches user requests and distributes the requests to multiple application server nodes according to the distribution policy. The commonly used load balancing technology hardware is F5, the price is relatively expensive, and the software is LVS, Nginx, HAProxy and so on.

2. With the increase of the number of users, the database has become the biggest bottleneck. The common means to improve the performance of the database is the separation of read and write and the separation of tables. As the name implies, the database is divided into read database and write database. Data synchronization is realized through the main and standby functions. Sub-database sub-table is divided into horizontal segmentation and vertical segmentation, horizontal switching is to split a database of large tables, such as orders, logistics information tables and so on. Vertical segmentation is to switch according to the business, such as orders, taxes, and other different topics in different databases. In this case, there is no association query, which can be easily solved by the program, and distributed transactions are used to ensure the consistency of the data. We also have a practice here, a large data table is divided into the current operation table and the history table, the current operation table only retains the data being operated, and then transferred to the history table, which can improve the efficiency of the current operation data.

3. The number of users increases day by day, the volume of business becomes larger and larger, and more and more documents are generated. In general, it is recommended that there are no more than 10,000 files in a directory, otherwise the search and polling of files will be very slow, resulting in the normal operation of the whole system. We are generally organized according to the directory structure of "\ application name\ module name\ date". For applications where the number of files is still large, we should subdivide them. When a single file server can no longer meet the demand, it needs distributed file system support. A commonly used distributed file system is NFS. We use MS's distributed file system (DFS), which is closely related to the AD domain.

4. Because the application server is clustered, users may not access one server before and after two requests. As a result, state can no longer be used as before (Application, Session, Cache, ViewState, etc.), and the application system must be stateless (of course, when the load balancing used has session persistence, a user will only locate to one server). The cache of the system should be kept on a dedicated cache server, and if there must be a state, it should also be saved on a dedicated cache server. As the first crab eaters, we used Microsoft's AppFabric as the caching server, because the version was very low and there were a lot of problems, and then we abandoned AppFabric and used Redis as the caching service. Now that AppFabric has improved a lot, running on the Azure cloud should not have the same problems as before.

Put a little bit in the middle. For various governments, units and other departments that can not deploy the system to the Internet, and there are corresponding branches in each province and city. Because the price of the network dedicated line is still relatively high, at least much lower than the network bandwidth of the Internet, of course, there is no shortage of money. In this case, the centralized and cluster deployment method mentioned above is generally not adopted, but the distributed deployment method is adopted. the first kind of distributed deployment is for each branch office to build a complete system to synchronize the data on a regular basis (for example, every day). The branch data is collected to the headquarters and sent back to each branch; the second distributed deployment method is for each branch to deploy middleware, but the data is concentrated in the headquarters.

Fourth, with the further expansion of the business, the application becomes very bloated, so we need to split the application into business. for example, the integrated business management system we do is divided into portals, contact disposal, business information, indicators, data query and analysis and other business sectors. Each business segment is an independent application responsible for relatively independent business operations. Business plates communicate with each other through message queues. The database is also split accordingly, and different topics are put into different databases. At the same time, it is best to build a static resource server and store the common css, js, images and so on in the static resource server.

Fifth, for the query of massive data, we can achieve better performance by using nosql database and search engine. Not all data should be placed in relational data. The commonly used NOSQL is mongodb and redis, and the search engine is lucene. We use Solr, ElasticSearch and other easier-to-use search engines based on the Lucene kernel. If the amount of data is large, Solr and so on should also be made into clusters.

6. Further down, the system needs to interact with other systems, and the system also provides services to various front-ends (such as websites, Android, IOS), so we have to build an application service layer on top of the logic layer to provide client and external SOA service interfaces. This also involves the concepts of DTO, WebService, WCF and WebApi (Rest). But the most important thing is that under SOA mode, including the previous MQ mode, transaction consistency can not be guaranteed, and certain mechanisms such as transaction compensation mechanism must be adopted to ensure the final consistency of transactions. The pressure on the servers where each business segment is located is also different in different periods of time. In order to spread the pressure among the servers in the server cluster as much as possible, we also need to provide a better mechanism. Record the pressure of each server, resources, the number of connections, and so on, so that new requests can be directed to the server with the least pressure.

7. The business continues to develop, that is, CDN. The next step is to set up several centers and deploy the system in each center. Users from all over the country will visit the center closest to him, and the data between the centers will be kept synchronized.

Eighth, it is mentioned above that there are many aspects of application systems, and a lot of work needs to be done in terms of data. The method of subdatabase and table has been introduced above. With the expansion of the application system, there are bound to be a lot of data resources, especially when the term big data is very popular, data analysis and processing is something a system must do. The advantage of this is that the query and analysis of the data will be independent, and the formal running system will not be affected. in addition, a lot of unexpected value can be obtained through analysis and mining.

At this time, the main work is to build a data warehouse, and then carry out follow-up analysis and processing. Use ETL/ELT to import data from the formal environment into the data warehouse on a regular basis, and build data marts according to different topics. For systems with relatively small amount of data, we can use relational database + multidimensional database; for large systems, we should use column storage, parallel database and so on. The analysis of data can provide upper leaders' decision-making by means of report, KPI, dashboard cockpit, etc., and can also use data mining, machine learning and training to realize value discovery, risk control and so on.

In general, enterprises do not have so much financial resources and personnel to do the above, so the use of cloud has become a choice for enterprises. Whether it is Azure, Aliyun, Amazon and so on, they will provide services one by one. Let's take Aliyun as an example. ECS provides virtual servers, SLB provides load balancing, RDS provides database services, OSS provides storage services, DRDS is a distributed data service, ODSP (now renamed MaxCompute) provides big data's computing services, RocketMQ provides MQ, OCS provides distributed cache services, and CDN, OTS, ADS, and so on.

By the way, there is also a sharp weapon called Docker, which can be used both in the enterprise and in the cloud. The Redis, Memcached, RabbitMQ and Solr we use internally are all deployed in Docker, which is indeed quite convenient.

There are a lot of things mentioned above, but no matter how well the architecture is done, it still needs to be implemented at the bottom. At present, the popular language is still object-oriented OO Java,. Net and so on, that is to say, it still uses the ideas and ideas of OO to program. Although abstraction, encapsulation, inheritance and polymorphism are literally easy to understand, in-depth understanding does require the accumulation of a certain amount of program. several major object-oriented principles and design patterns are still an important guarantee for writing codes with higher extensible, replaceable, configurable, maintainable and other software quality indicators.

After reading the above, do you have any further understanding of the idea of. Net distributed system architecture? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.