Shulou(Shulou.com)06/03 Report--

The change of Taobao Technology Architecture

Since its inception in 2003, Taobao business has developed rapidly, growing at a rate of almost 100% every year. At the beginning of its establishment, in order to quickly launch and seize the market, I chose the popular LAMP architecture at that time, using PHP as the website development language, Linux as the operating system, Apache as the Web server and MySQL as the database. Taobao was launched in less than three months. At that time, there were about 10 application servers in the whole website, and the MySQL database adopted the deployment mode of separation of read and write, one master and two standby.

In 2004, driven by the development of Taobao business, we referred to some enterprise solutions of telecom operators and banks, and transformed the LAMP architecture into the database architecture and EMC storage mode of Oracle+IBM minicomputer (figure 2). Although the scheme is expensive, the performance is very good. At the same time, with the increase of website traffic, the system appears to be overburdened. The biggest concern at that time was how to design the system architecture of the website if the traffic of the website continued to increase and the transaction volume continued to increase. How do I select a database? How do I choose caching? How to build a business system? Later, referring to eBay's Internet design architecture, I designed a technical solution for Java, and used a lot of Java open source products. For example, choose JBoss, which is popular at that time, as the application server; choose an open source IOC container Spring to manage business classes; encapsulate a database access tool IBatis as an Object-Reletionship mapping tool for database and Java classes. In addition, for the commodity search function, we use the self-developed ISearch search engine to replace the search in the Oracle database to reduce the pressure on the database server. The practice is relatively simple. Every night, all the data of the Oracle minicomputer is dump, and the Build becomes the index of ISearch. At that time, the commodity volume was not large. A server with ordinary configuration could basically put all the indexes into it without segmentation, and directly made a peer-to-peer cluster.

Since 2006, Taobao has started to build its own CDN site in order to improve its user experience. Since Taobao's main traffic comes from static data such as various product pictures and descriptions, self-built CDN can make these resources closer to users, improve users' access speed and improve users' experience of browsing the site.

In 2007, Taobao's annual transaction volume exceeded 40 billion yuan, an average of nearly 100 million a day, and more than 1 million transactions were created every day. The main problems faced at that time are: the traffic of some systems is very large, such as commodity details, if you access the database directly, it will lead to great pressure on the database; for example, if you visit a page, you need to query the buyer's information, the seller's information, show the buyer's credit, the seller's service star, and so on. At this time, Taobao uses distributed cache TDBM (the predecessor of Tair) to cache these hot static data in memory to improve access performance. In addition, the distributed file system TFS developed by ourselves is deployed on multiple x86 servers to replace commercial NAS storage devices to store all kinds of file information on Taobao, such as commodity pictures, commodity description information, transaction snapshot information, to achieve the purpose of reducing costs and improving the capacity and performance of the overall system, while achieving more flexible scalability. The first phase will be launched with about 200 TFS servers. In addition, the ISearch search engine is changed to a distributed architecture, supporting horizontal expansion, and 48 nodes are deployed. Figure 3 shows this architectural idea.

At the beginning of 2008, in order to solve the bottleneck problem of the centralized structure of Oracle database (connection limit, Imax O performance), the system was split according to user domain, commodity domain, transaction domain, store domain and other business areas, and more than 20 business centers were established, such as commodity center, user center, trading center and so on. All systems with user access requirements must be accessed by the remote interface provided by the business center, and can not directly access the underlying MySQL database. The service interface of the business center can be called through the remote communication mode of HSF, and the call between business systems can be completed asynchronously through Notify message middleware. Figure 4 shows the distributed architecture of Taobao.

Since 2010, Taobao has focused on a unified architecture, considering the requirements of development efficiency, operation and maintenance standardization, high performance, high scalability, high availability and low cost from the overall system level. the underlying infrastructure has adopted Ali cloud computing platform (figure 5), using Ali cloud computing services such as SLB, ECS, RDS, OSS, ONS, CDN, and providing high availability features through Ali cloud services. Achieve dual computer room disaster recovery and remote computer room unit deployment to provide stable, efficient and easy-to-maintain infrastructure support for Taobao business.

In the process of the final transfer from IOE architecture to cloud computing platform technology architecture, we are mainly faced with the following technical challenges.

■ availability: whether a cloud computing platform based on PC server distributed architecture can achieve high availability without the high redundancy mechanism of minicomputer and high-end storage.

■ consistency: Oracle is based on the physical level consistency achieved by RAC and shared storage, based on whether RDS for MySQL can achieve the same effect.

■ high performance: high-end storage has strong RDS O capacity, whether RDS based on PC server can provide the same or even higher Imax O processing power, and whether MySQL and Oracle have the same processing performance for SQL.

■ extensibility: how to split the business logic, how to service, how much data is divided, how many tables are divided, what dimensions are divided, and how it is more convenient for the second split in the later stage.

Based on Ali cloud computing platform, by adopting appropriate technical strategies and best practices, including: application stateless, effective use of cache (browser cache, reverse proxy cache, page cache, local page cache, object cache and read-write separation), service atomization, database segmentation, asynchronously solving performance problems, minimizing transaction units, and appropriately abandoning consistency. And automatic monitoring / operation and maintenance means include monitoring and early warning, configuration unified management, basic server monitoring, URL monitoring, network monitoring, inter-module call monitoring, intelligent analysis monitoring, integrated fault management platform, capacity management. It can solve the above problems well, so as to achieve the realization effect of high scalability, lower cost, higher performance and availability of the whole system.

Best practices for moving Cloud Architecture

The technical architecture of Taobao is a process of gradual evolution with the gradual development of the business, in which there are a lot of valuable architectural best practices. For most enterprise customers, they can choose the appropriate technical architecture to realize the Internet design of the overall IT system according to their own business scenarios. Cloud migration architecture in different application scenarios, including file storage, application services, OLTP database, OLAP database.

For file storage, we can directly use OSS instead of EMC storage to store massive data files, and the maximum capacity of OSS storage can reach 40PB. At the same time, because OSS is a distributed storage mode, it can significantly improve data access performance through parallel reading and writing of multiple nodes. For large files, we can also transfer and store large files in blocks in parallel through Multipart Upload to achieve high performance.

For application services, the IBM minicomputer can be replaced by the combination of SLB+ multiple ECS instances (figure 6), or it can be directly deployed based on Ali Cloud middleware cloud services such as ACE, ONS, OpenSearch, etc., according to different application types.

The migration of OLTP applications is relatively complex. Currently, the maximum RDS instance of Aliyun is 48GB memory, 14,000 IOPS has a storage capacity of 1 TB (SSD storage), and supports MySQL and SQL Server. This configuration can be used as a single database server to meet the database application needs of many scenarios, and can directly replace IBM minicomputer + Oracle database + EMC storage in most scenarios.

For applications with higher performance requirements, we can consider introducing the open cache service OCS to load part of the query data into the distributed cache, reduce the number of data queries in RDS, improve the concurrent efficiency of data queries and reduce the response time of the system, as shown in figure 7.

For scenarios where the read request is much larger than the write request, you can consider using multiple RDS databases to achieve the separation of read and write in a distributed manner. The write transaction mainly occurs in the main database, and the read request accesses the standby database. The read database can be expanded according to the requirements to improve the overall request performance.

For database tables with large data scale, we can split the data horizontally, distribute the data on multiple RDS instances, and improve the performance and capacity through parallel distributed database operations.

Generally speaking, through the migration to RDS, the introduction of data cache, sub-database sub-table, read-write separation and other ways to replace the original IOE architecture with Scale-Out, and achieve better performance and scalability.

For OLAP applications, the ODPS+OTS+RDS/ADS solution can be used instead of the minicomputer + Oracle DB+OLAP+RAC+EMC storage solution, as shown in figure 11. Overall, the general architecture scheme of moving cloud is shown in figure 12, and the cloud relocation scheme for specific business systems still needs to be analyzed and reasonably selected according to the actual situation.

The above is the detailed explanation of Taobao's technical architecture changes, and the following is the latest series of Ali P8 architects talking about architecture design.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.