Shulou(Shulou.com)06/02 Report--

Introduction: this year is the 18th year of Huawei's storage business. Over the past 18 years, Huawei Storage has witnessed the whole process of the development of China's storage industry from scratch and from weak to strong.

This article recalls in detail the ups and downs of Huawei Storage in the past 18 years: the difficult and tortuous growth process, the ordinary and honorable striver, the important moment to turn things around, and the exciting product technology.

Due to the long length, it is divided into ten sections, which are released in the form of serialization. The specific chapters are as follows:

The first self-proof, a lifelong multiple-choice question, teenagers learn to be in awe of data regression in order to better start. "there is an important task for you." users will always be the best teachers. Jane, the secret of undiminished color, the most artisan future journey like the stars of the sea

Let's officially review the struggle of Huawei Storage in order to open up the mountains and forests.

This is a rather inspirational growth story. From a small team of three or five people to a team size of thousands today; from a pre-research machine to a fully-distributed storage product line; from years of net investment in the initial stage to the world's fastest-growing all-flash memory for eight consecutive quarters; from obscurity in the past to a regular customer of Gartner Magic four Quadrant leaders today.

It took Huawei 18 years to do this.

In the course of 18 years of ups and downs, Huawei has struggled from toddlers, touching the stones of the river bank, to teenagers who walk the world with swords and brave the world, but they have also made a lot of gains. This year is Huawei's storage for 18 years, which is like a bar mitzvah. It not only pays tribute to the youth that one has struggled in the past, but also means that he will shoulder greater responsibilities in the future.

Now let's review the history and feel the temperature of the strivers in the storage industry.

The first self-proof

In the field of technology, the process of any business from 0 to 1 is bound to be extremely difficult, and Huawei Storage is no exception.

Huawei's entry into storage was accidental. After the bursting of the Internet technology bubble in 2001, it spread to almost all technology companies, and Huawei was not immune.

At that time, Huawei thought that only doing communications products would face great risks in the future and needed to actively look for new industrial opportunities. As a result, Huawei's commercial network department was quietly set up, and storage is precisely one of the new businesses. "there were dozens of new businesses at that time, and storage is one of the three businesses that are still alive today." Zhang Guobin, a senior expert in Huawei Storage, said.

Zhang Guobin joined Huawei after graduating from Tsinghua University in 2002 and happened to be assigned to the storage research and development team, which opened a relationship with storage for nearly 20 years and witnessed the whole process of Huawei storage from scratch, from catching up to gradually surpassing.

"the company officially invested in the storage business in 2002. at that time, the storage industry was indeed a sunrise industry with great potential. however, the company was cautious at the beginning, when the team added up to about a dozen people," Zhang Guobin recalled. "the Huawei storage team first needed to prove whether it was able to do the storage thing."

At that time, the global storage industry was in a stage of rapid development. EMC, one of the four knights of the Internet in North America, was famous for helping Morgan Stanley recover all its business 24 hours a day during the 9 / 11 incident, and the global storage industry officially began the golden era. EMC completed a series of acquisitions of Data General, Legato, Documentum, Avamar and VMware in just a few years, quickly completed the role transformation from a high-end storage manufacturer to a storage giant, and started a 15-year storage hegemony career.

However, who would have thought that Huawei Storage, which is in its infancy at this time, will surpass EMC in the Chinese market more than a decade later.

There was no one, no gun, but the pre-developed products had to be developed within a year, which was the difficult situation faced by Huawei's storage team at that time. To this end, Huawei Storage first thought of the idea is to cooperate with university scientific research institutions. As a result, Huawei Storage worked with Professor Xie Changsheng of Huazhong University of Science and Technology (later renamed Huazhong University of Science and Technology) to build a pre-researched storage system based on standard servers, commercial FC cards, open source RAID code and Cache/SCSI components.

"after a year of hard work, the storage prototype has finally been successfully developed, which will run smoothly in the FC environment." Zhang Guobin said, "later, it was accepted by the management, and Huawei Storage also officially launched research and development in 2003."

With the power of open source + innovation, Huawei Storage has proved for the first time its ability to survive. However, there are more unexpected difficulties waiting for Huawei.

A life-long multiple choice question

There is never a shortcut to success in this world, because there are no shortcuts, especially in the field of science and technology.

Huawei Storage, which officially opened its research and development, is immediately faced with a multiple-choice question: open source or innovation.

To this end, there has been a heated discussion within Huawei's storage team. Adopting the open source route means you can accelerate the speed of successful product research and development, which is a "shortcut" for Huawei, which is just involved in the storage field, but open source also generally exists situations such as relatively simple code and functions, low efficiency, and more Bug; while taking the innovation route from the very beginning means that you need to maintain continuous investment, which is a test of time and cost.

"in the long run, choosing open source will bring more risks in the future." Zhang Guobin explained, "after full argumentation and balance, the team finally unanimously decided that it would rather take several years to lay a solid foundation for customers and themselves." This decision has a decisive impact on Huawei Storage's subsequent R & D system building and platform-level R & D strategy, and it is also the key for Huawei Storage to achieve a sustainable breakthrough in the future.

However, over the course of a year, the R & D team found that the road was far from as easy as expected, and it felt like "entering the storage as deep as the sea".

"after working for a year, I found that something was wrong, and storage was too much investment in research and development." Zhang Guobin said, "when the price of enterprise hard drives is very expensive, coupled with high labor costs." In fact, the technical threshold of enterprise storage is very high. As the final home of enterprise data, enterprise storage products have high requirements for reliability and data consistency, which directly determines the need for high investment in storage research and development.

Faced with such high costs, Huawei also began to hesitate. Do you want to stick to it or not? If you stick to it, how long will it take to make a competitive product in the market? After making the product, how big is the market window left for Huawei?

In 2004, Huawei thought it was necessary to grit its teeth, but it needed flexible methods to understand market changes and user needs more quickly, so as to accelerate the research and development of storage products. To this end, Huawei decided to start to cooperate with leading foreign storage manufacturers, on the one hand to meet the internal business needs, on the other hand, through mature products to understand the actual needs of users.

After that, operating system memory management, message communication, various storage protocols, Cache, RAID software, cluster management, configuration operation and maintenance. The team realized all the functions one by one. After completing the single-controller product, Huawei storage research and development team began to realize the research and development of dual-control storage system.

At that time, from single-controller storage to dual-controller storage means that the research and development capability has an essential leap, and there are very high requirements for storage architecture design, software development and so on. Once there is a problem with the single controller, it will lead to the failure of the entire storage, which will undoubtedly have a fatal impact on the user's business; while the dual controller is equivalent to buying an insurance for the storage, one controller fails, and the other controller immediately follows to ensure that the customer's business will not be interrupted.

In 2005, Huawei spent more than three years basically completing the research and development of dual-control storage system, which also means that Huawei storage research and development has achieved a phased victory.

At this time, a good opportunity appeared in front of Huawei Storage, does it have the courage to accept it?

A teenager who wandered around alone

In 2006, Huawei officially launched F800 storage. At this point, Huawei began to consider entering the storage industry on a large scale. At this time, Huawei is faced with a choice: the first is to do it all by itself, whether it is R & D, marketing, channels, and services; the second is to choose to work with partners.

This time, Huawei chose the latter.

Huawei had its first contact with Symantec in 2005, when the storage and security giant Symantec offered an olive branch. After continuous contacts and negotiations, Huawei and Symantec officially launched the joint venture Huawei Symantec (hereinafter referred to as: Huasai) in 2006: Huawei personnel and research and development, Symantec funded, focusing on storage and security business.

The adoption of an independent joint venture was in the best interests of both sides at that time. Huawei received sufficient funds to continuously increase its investment in storage research and development to create more competitive products in the market. Symantec avoided the risk of starting from scratch.

After nearly two years of preparation, Huasai was officially established in 2008, with a storage team of about 200 people. Huawei Storage at this time is like a teenager wandering around alone, with lofty dreams, but also perturbed.

2008 is a landmark year for the storage industry, the storage market is giving birth to great changes, high-end storage innovative architecture, deduplication and compression technology, flash memory technology, cluster file technology. A large number of innovative start-ups have emerged in almost all storage segments, and the innovation of the storage industry is in full blossom.

"the joint venture gives Huawei Storage the opportunity to grow independently." Zhang Guobin said so.

Huasai is like a running teenager, separated from the Huawei system, operating efficiency, decision-making efficiency as quickly as start-ups, actively participate in storage innovation. Huasai developed the first SSD in 2009, and it was only in 2008 that SSD was first used in high-end storage.

Not only that, Huasai, as an independent joint venture company, began to go out and bring in, recruiting a large number of talents around the world, thus laying the foundation for Huawei's global R & D system. By constantly seeing the world and introducing storage giants, Huawei's vision and ideas of the overall talent system of storage have been greatly expanded.

"Symantec did not inject a single line of code into Huasai. Huasai did not use any of Symantec's technology, but the R & D team benefited a lot from communicating with Symantec storage experts." Zhang Guobin clarified.

Not only in R & D system, R & D ideas by leaps and bounds, independent operation of Huasai means that there is no market and sales resources available for large companies, we must establish our own professional sales and marketing team, and bravely go out to meet customers and study orders. Zhang Guobin recalled that the independent operation in those years did give a lot of training to the sales and marketing team, and the head customers of the government, finance, operators and large enterprises all had to be gnawed off one by one. "in 2009, Huasai storage products were used in the image archiving of large state-owned banks, and began to make breakthroughs in various industries."

However, the road to growth has never been plain sailing. It is often said that growth is actually a kind of unforgettable transformation. Huasai, a running storage teenager, also has an unforgettable moment of falling.

Learn to revere data.

In 2009, as Huasai products were used by users in many industries, some problems began to follow: the sound of the alarm resonated with the sound of rotating disk, resulting in batch failures of the hard disk; air pollution and sulfides in the environment corroded the hard disk, causing the hard drive to slow down; the controller short-circuited; and even the legendary cosmic ray caused memory Bit bit flip caused by data inconsistency also appeared.

This has led Huasai to re-examine storage. "to be honest, these problems have really taught Huawei Storage to respect storage and awe of data," says Mr Zhang. " There has never been any casual success in this world, and the reason why competitors in the market have been able to take the lead in the market is inseparable from them over the years. Huawei storage is still a little immature at this time. I seem a little unprepared for these problems.

Starting from the place where it fell, Huasai learned a lesson from it. Under the leadership of Fan Ruiqi, then president of Huasai's storage business, the R & D team did one thing in 2009, in addition to the R & D work on hand. Make all the underlying storage protocols and hardware components clear and clear.

After that, the R & D team nibbled down all the thick English agreement, and then re-carried out anti-vibration, anti-corrosion design, and joint design with hard disk manufacturers to seriously explore each kind of hard disk fault parameters, and carry out joint software and hardware processing, the firmware (Firmware) of millions of lines of hard disk code must be verified in the laboratory for a long time before it can be upgraded to users.

Since then, Huawei Storage has established an immovable principle: storage reliability first, data must not be lost, and storage products can only be listed without data loss.

"to this day, I still remember clearly the days when people get together to do root cause analysis every weekend, asking questions about every problem that arises, until it is finally solved." Zhang Guobin recalls.

In addition, Huasai, a storage teenager, has benefited a lot from the exchange and learning with world-class talents. At that time, many of the Huasai R & D team graduated from famous universities and had accumulated several years of experience in the field of storage, so they were full of confidence in themselves. However, with the arrival of world-class talents, Huawei is not only injected with a variety of leading experience, skills and methods in the storage field, but also has an important impact on the Huasai storage research and development team at the ideological level.

For example, in the very common delay control in the storage field, delay is a key indicator in the storage field. The fluctuations and anomalies of delay may have a great impact on the performance of the storage system, which in turn will directly affect the performance of the business. How to accurately control the delay can be said to be a craft. The first-class storage tycoon's focus on the precise control of delay is like the spirit of ingenuity, which awakens the Huasai R & D team and understands that there is no end to the polishing of storage technology. need to have the spirit of ingenuity to improve the technology.

In the independent operation in the past few years, Huasai Storage has fallen a lot, suffered a lot, and obtained a lot of returns. The whole company is in good operation and has made profits twice in a row; the construction of a talent team has been fruitful, and the R & D system has initially taken shape; product research and development has blossomed one after another, and new products such as the second generation of mid-range systems and SSD hard drives have brightened the industry.

At this time, Huasai chose to return to the Huawei family. what impact will this choice have on the storage of products, research and development, and sales systems that have already been built?

The return is for a better start.

On November 4, 2011, Huawei and Symantec announced that they had reached an agreement for Huawei to acquire a 49 per cent stake in Huawei Symantec for $530 million. Since then, the return of Huasai has become a foregone conclusion.

If planned at a price of $530 million, Huasai was valued at more than $1 billion that year, making it a veritable storage unicorn. Huasai has also used its own experience to prove the fact that these Huawei storage people are capable of surviving and even making a career in the market.

Will the return of this moment repeat the mediocrity of startups acquired into big companies in the past, or will it add to the strength of the entire storage business and take it to a new level? This is not only the concern of the outside world, but also what the storage team is eager to know.

"after the official return, the company immediately invested heavily in storage, obviously feeling that the company wanted to expand and strengthen the storage business." An employee who has experienced the whole process of Huawei's storage departure, independence and return sighed, "if we remain independent, it is impossible to be completely liberalized in R & D out of revenue, profitability and other considerations. especially in the field of basic technology."

Since then, Huawei Storage has officially entered a global and top-level construction stage in its R & D system and R & D talents, laying a solid foundation for future OceanStor storage, Dorado series all-flash memory, Pacific series mass storage and other blockbuster products.

From a domestic point of view, Huawei Storage has already established sound and powerful R & D centers in Chengdu, Beijing and Shenzhen. A number of research institutes in the field of storage have also been established worldwide. After several years of building, the research and development team of Huawei Storage has been comparable to any of the world's top storage companies.

Another biggest change in R & D after the return is: platform. With the gradual establishment and improvement of the global R & D system, Huawei storage R & D platform began to step on the agenda. Zhang Guobin revealed: "in the past, during the Huasai period, the company really wanted to develop a platform." but after all, Huasai is still small in scale, and it takes a long time for the platform to work. " However, after returning to Huawei, the conditions required for Huawei's storage R & D platform strategy are very perfect, and Huawei Storage has transferred most of its R & D efforts to build the platform.

Xiang Fei, head of Huawei's data storage products department, said: "Storage platform is actually a gradual process, which requires the continuous accumulation of storage key technical capabilities, software and hardware research and development capabilities, and project management capabilities, and then through a component-based thinking, so that complex storage systems are made up of various components, and the composition of components between different storage systems will change greatly."

There is no doubt that the platform strategy is to gradually complete the componentization of various technologies and functions, and to build the storage system based on the idea of decoupling, so as to reuse various technologies and components to the maximum. The most direct benefits of this platform strategy are: greatly improved R & D efficiency, interoperability between components, faster iteration of product updates, and faster response to the flexible needs of the market.

Huawei Storage is based on the platform strategy since the first day of its establishment, which is very different from the platform through various mergers and acquisitions. Although it takes a long time to accumulate, the R & D layout and division of labor are gradually improved with the passage of time. With the continuous accumulation and precipitation of core key technologies, the trouble of various products and functions in the future, compatibility and even future planning will be saved, and the flywheel effect will become increasingly prominent.

Another significant change is that the R & D results in Huawei's entire R & D system can be used for storage, such as Huawei 2012 laboratory research and development, operating system technology, and so on. Xiang Fei said bluntly that storage is an extremely complex product, which involves many fields such as computing, network, media, and so on. Considering the future development trend of the data center, it is far from enough to stand on the perspective of the storage R & D team itself. We need to make full use of different R & D teams within Huawei to cooperate to promote the continuous development of storage with a broader perspective and development cooperation.

Since then, Huawei storage technology innovation + user demand of the two-round innovation mechanism began to give full play to its power. In the future, Huawei storage products have been ranked high again and again in various tests, and their groundbreaking innovations are like sesame blossoms, which is in sharp contrast to other relatively "silent" competitors.

But building a platform is a painful and long-term process. Before that, Huawei Storage also encountered a more challenging thing: how to make its product line rich and strong, and achieve breakthroughs in high-end storage and distributed file storage.

The mountain of high-end storage must be leaped.

In the storage industry, high-end storage is like the largest pearl, which directly represents the status of a storage manufacturer in the industry. If soldiers who don't want to be generals are not good soldiers, then manufacturers who don't want to be high-end storage must not be ambitious. This is why Huawei is tirelessly involved in the research and development of high-end storage.

As we all know, the high reliability, high performance and high consistency of high-end storage make its technical threshold extremely high. Since 1990, only EMC and HDS (later renamed Hitachi Vantara) have the absolute strength in the high-end storage market. It is not easy to break through the market competition barriers.

"it was very difficult to make a decision at that time, and there was a lot of controversy and discussion." Zhang Guobin said, "after all, once a decision is made, the input cost will be very huge, and failure is not allowed." In fact, if Huawei Storage wants to further break through the market, it is necessary to conquer high-end storage. "without high-end storage products, medium-and low-end products may not sell well in the long run."

At that time, the trend of cloud computing was just beginning to emerge, and some storage manufacturers had already seen the disadvantages of high-end storage, although they could perfectly meet the needs of traditional business scenarios, but the disadvantages of weak scalability and large investment began to magnify gradually in the era of cloud computing. Zhang Guobin revealed: "at that time, several manufacturers did see the disadvantages of traditional high-end storage, and everyone saw the future direction."

To this end, Huawei began to invest in the research and development of high-end storage based on its own technology foundation and the trend of the industry. At that time, Zhang Guobin led more than 100 people in the mountains for several months, specializing in the new architecture design of Huawei's high-end storage. The whole process is like practicing unparalleled martial arts, day after day hard work can not feel the degree of progress of martial arts, when the governor through the second pulse, pierced that layer of window paper, like a cloud to see the sun, suddenly enlightened.

After continuous exploration and efforts, Huawei Storage finally officially launched the first generation of high-end storage products based on SmartMatrix architecture in 2013. This is the first time that Chinese storage manufacturers have launched high-end storage products, which is of great significance and value.

It not only gives Huawei storage people great confidence, but also drives Huawei people to take the initiative to think about future storage changes, laying a solid foundation for a series of groundbreaking innovations such as Dorado series of high-end all-flash memory in the future. Secondly, it provides a sharp tool for Huawei Storage to open its customers in various industries, which greatly improves the depth and breadth of Huawei Storage's market expansion.

"I have an important task for you."

"Ji Fei, I have an important task for you! lead the team to get the distributed file storage out."

After receiving the notice from the leader, Xiang Fei could not help feeling a little perturbed in his heart. He, who has been working on the development of communication software, is going to become a monk halfway to storage this time.

At this time, Huawei Storage has just returned, and the product line is mainly SAN storage. Before that, Symantec's file storage products were mainly sold during Huasai. Now back, to expand and strengthen the storage business, Huawei's own distributed file storage products are essential. But at that time, the development of distributed file storage products was indeed a major test, so that some voices inside Huawei thought it was difficult to develop.

After all, in the field of distributed file storage, IBM, NetApp and EMC (the acquisition of distributed storage star company Isilon) are well-deserved leaders in terms of product technology and marketing. Huawei needs to work harder than ordinary people to pry into the cake of distributed file storage.

"in Huawei, you dare to try anything. It doesn't matter if you don't understand distributed storage. I'll just regard it as a 'second venture'. Just study hard and stick to it!" Xiang Fei finally set his heart to overcome the difficulty of distributed storage.

So Xiang Fei began to be busy recruiting people and adjusting the structure of the R & D team. "the most difficult thing is people. I spend a lot of time interviewing all kinds of talents around the world and finding the right backbone talents in order to lead the team forward." Xiang Fei said, "many people in the whole team moved me very much. for example, many people in the design department were experts in telecommunications software, and they were all scattered among the new teams. this time, everyone started all over again and all went back to coding posts. Basically, they all developed distributed file systems while learning."

It is not difficult to make distributed file storage, but it is not easy to make market competitive distributed file storage. Because the benchmark in the market is high enough, like Isilon's fully distributed symmetrical scale-out architecture, metadata and data are distributed on all nodes, which is of great technical difficulty. Huawei's distributed storage also uses a fully distributed symmetrical scale-out architecture, so what is its quality?

The fully symmetric distributed architecture is very challenging because all metadata is dynamically distributed on all nodes, and metadata needs to be available in all cases. In the research and development process of Huawei distributed storage, the biggest difficulty is the problems related to metadata. "at that time, the team experts checked out all kinds of problems day and night to ensure that the product could be launched smoothly."

In 2014, after more than two years of efforts, Huawei officially launched distributed file storage OceanStor 9000 with fully symmetrical distributed architecture, making it one of the few storage companies in the industry to adopt this architecture. Moreover, OceanStor 9000 sets a new benchmark for the industry in terms of performance, scalability, stable low latency and automatic layering.

Huawei OceanStor 9000

Over the next few years, Huawei OceanStor 9000 began its wonderful performance in the market. First, in the domestic market, OceanStor 9000 entered into media capital, scientific research, oil and energy exploration and other fields, with excellent products to capture a large number of customers. Not only that, OceanStor 9000 also showed extraordinary combat effectiveness in overseas markets, and quickly gained recognition in Asia-Pacific, Europe, the Middle East and other regional markets.

For a while, EMC, the storage boss, found that Isilon had one of its strongest competitors. So much so that EMC later released a video on the official website to "disapprove" of OceanStor 9000, which is probably the first time that Chinese storage manufacturers have enjoyed the high "treatment" of storage giants.

Users will always be the best teachers.

Users are always the best teachers, because they can give you the most straightforward insight into the real application needs of the business, their own gaps and even the strengths and weaknesses of competitors.

In 2014, Zhang Guobin went to Europe and became a member of Huawei's storage market, shuttling through various exhibitions and speeches every day. I heard that a well-known German telecom company is inviting bids to purchase a batch of storage products. Zhang Guobin decided to contact and visit the company's architects with his colleagues in an effort to take down the big customer.

After the meeting, the architect of the company was very practical and did not wait for Zhang Guobin to eloquently introduce the various advantages of Huawei storage products and come up with a tender form. Huawei Storage is required to list the performance of the products, the ecological docking of various tools, the reliability, and so on. "when I met such rigorous and professional users for the first time, I realized that we are still a little far away from the users." There is still a lot of room for improvement. "

In 2017, Huawei officially launched the OceanStor storage Dorado series V3 products. When Zhang Guobin saw the telecom company architect again, he was already confident and confident, explaining the storage features, performance, reliability, ease of use, tools and functions of V3 products one by one, and after successfully moving customers with excellent professionalism, Huawei storage products successfully passed the POC test, and the customer also purchased several sets of high-end Huawei storage products at one time.

In 2015, a world-class machinery manufacturing company in Sweden began to increase the application of autopilot in its products. Because machine learning and deep learning require continuous training of data, and the data are hundreds of billions of small pictures and videos, traditional distributed storage is usually large files and large bandwidth, so it is difficult to adapt to AI training scenarios like autopilot, the company's architect complained in a conversation with Huawei's storage team. Due to the needs of different scenarios, multiple sets of different storage to run different applications, and began to produce a data chimney.

In this regard, the company's architects discussed with Huawei Storage whether it is possible to support the data lake through a set of distributed storage, in addition to self-driving, but also for applications such as car networking, traditional CAD and CAE. As a result, Huawei Storage decided to jointly innovate with the customer to develop a set of distributed storage to support all application integration.

Next, Huawei Storage works closely with the customer to discuss with the customer how to design each function and parameters, and to test, deploy and apply in each scenario. The Swedish company directly made Huawei understand the challenges and needs of the new business scenario. "in the face of new application scenarios, Huawei storage must go fast, and it needs to be done with users, not behind closed doors."

After returning to Huawei, Huawei Storage has made more and more joint innovation with top customers in various industries at home and abroad, involving many fields such as finance, telecommunications, energy, media capital, and so on. It has a very in-depth understanding of the storage needs and pain points of head customers' data centers, and has laid a solid foundation for product innovation and product power improvement in the future.

The secret of simplicity without reducing

Since 2015, leading customers in many industries at home and abroad, including finance, operators, energy and other industries, have mentioned distribution in the process of communicating with Huawei storage, and users are eager to deploy distributed storage in the core of the main data center. so that core applications can also enjoy the benefits of distributed storage.

"for example, a telecom company in the Netherlands, which had more data than 300PB at that time, was eager to transition to a distributed cloud architecture." Zhang Guobin said, "they hope that distributed can host core businesses such as databases and virtual machines, as well as new applications such as big data and AI."

The needs of these customers have directly contributed to Huawei's creation of a new generation of intelligent distributed storage-OceanStor distributed storage. In 2016, Huawei Storage quickly brought together the best architects in both centralized and distributed areas to solve the problem of how to integrate distributed storage and centralized storage into one product.

"at this time, the strategic advantage of Huawei's storage platform appears. Most of the strength of Huawei's research and development is on the platform products, which can fully mobilize resources to tackle key problems." Zhang Guobin revealed, "this is very difficult for other companies to achieve, because according to their model, the cost and time invested are extremely large."

Also after more than two years of research and development, Huawei Storage officially released a new generation of intelligent distributed storage-OceanStor distributed storage in 2019. Huawei's storage R & D teams in Beijing, Shanghai, Shenzhen, Chengdu and other places worked closely together across regions to design a new storage architecture, developed tens of millions of lines of code, and successively carried out nearly 10 versions of iterations. the result of repeated polishing and optimization.

Huawei OceanStor distributed storage products not only achieve the first distributed storage performance in the world, with high-end single-node performance of 168000 IOPS and latency of less than 1 millisecond, ranking first in industry standard SPC-1 tests, but also support block, file, object and HDFS protocols simultaneously in a set of storage for the first time, and take the lead in integrating artificial intelligence technology into storage lifecycle management.

Behind the fact that OceanStor distributed storage provides a set of simple but powerful storage for user data centers, it is actually the result of repeated polishing and optimization by the R & D team. For example, in order to solve the "invisible bottleneck" of performance fluctuations, Huawei Storage R & D team used operating system scheduling experts and network experts in the global R & D system to completely resolve performance fluctuations and achieve stable performance of 140000 IOPS for non-ARM nodes and 168000 IOPS for ARM nodes.

There are three difficulties in distributed storage, one is how to solve the data inconsistency, because the data is scattered on each node, it is easy to appear data inconsistency, especially in the case of high-performance, large concurrent data read and write, how to maintain data consistency. This requires Huawei storage to be extremely accurate in the concurrency mechanism. To this end, Huawei Storage R & D team has developed a new logging mechanism and tools to track and view IO exceptions in massive data, and to verify data inconsistencies in thousands of nodes in different scenarios.

The other is how to balance distributed storage function and performance, such as deduplication function has a great impact on storage performance. To this end, the Huawei storage R & D team has specially developed a weighting algorithm that allows the re-deletion function to be adaptively adjusted, which subtracts the complexity of constant manual configuration for users in the past, showing it extremely simple and easy to use.

The last part is how to prevent data loss. In general, the application scenarios carried by distributed storage are diversified and complex, which directly promotes the increase of potential factors of data loss. In view of this situation, OceanStor distributed storage holds more than 30 scenarios in a reliability check space of only 64 bytes. The "implantation" of each prevention program needs to be accurate to each bit, which can be called as accurate as a scalpel.

It's time for high-end storage to change!

This is a true user story.

A world-class logistics company has 4000 servers running on a set of high-end storage. According to the traditional high-end storage architecture, if you upgrade the system software, in order to prevent the front-end multipath from running abnormally, you must check the multipath of more than 4000 servers before upgrading. Every time customers upgrade, they are on the verge of collapse as if they are facing the end of the world.

The company's situation really reflects the embarrassment of many users: there are more and more applications, businesses are becoming more flexible, the application environment is changing rapidly, and high-end storage needs to be able to carry more business in the data center. The situation of hosting only a few businesses in the past is gone forever.

Now, it's time for high-end storage to change. Some people say that the technology industry does not respect the past, only respect innovation, you do not innovate, you have to die. At this time, Huawei has completed the accumulation of all-flash technology and high-end storage architecture, and it is time to create a future-oriented product that meets the extreme needs of users' data centers and rejuvenates high-end storage.

As we all know, flash memory has a revolutionary impact on the storage industry, and all-flash storage is considered to be the trend of the times. The traditional high-end storage is precisely based on the products of the traditional hard disk era, even if some manufacturers add flash memory to the high-end storage, it is only an optimized compromise choice.

All-flash memory, this is a brand new track. The new track means that you have the opportunity to catch up with your competitors, especially in the field of high-end all-flash memory, which will be a comprehensive reshaping of software and hardware architecture and function stack. Huawei already has a deep accumulation in the field of flash memory.

Second, Huawei's experience in SmartMatrix high-end storage architecture has promoted a further breakthrough in the architecture of the next generation of high-end storage. Huawei Storage began to integrate Scale-Up and Scale-Out, using modular scale-out, full decoupling and full sharing of all key components, and incorporated a number of intelligent technologies.

Release of OceanStor Dorado V6 new generation all-flash products in 2019

In July 2019, Huawei's new generation of OceanStor Dorado V6 series products, which lasted for nearly three years, were officially unveiled: with the highest 20 million IOPS extreme performance in the industry, the lowest stable 0.1ms delay in the industry, the extreme case of bad controller 8 and bad 7 still working, intelligent operation and maintenance based on AI algorithm throughout the lifecycle, and the realization of fault zero awareness, business zero impact and upgrade zero impact, really ensuring that user business is permanently online.

If Huawei Storage has been following in the past, then the stunning appearance of OceanStor Dorado V6 marks the beginning of Huawei Storage. It is an explosion of technology accumulation and innovation of Huawei Storage for many years, and it shows that Huawei Storage has embarked on a different and groundbreaking path.

A touch of red and blue is the most popular among craftsmen.

Storage product is the final destination of data, in addition to performance and efficiency, data availability and persistence, product reliability and other requirements are extremely strict. The development of storage is sometimes like creating a complex work of art, which requires technical research and development personnel to constantly carve and polish to pursue a perfect state.

"Storage requires a high culture of ingenuity for engineers. Because if the code is not careful, the whole box may collapse, with disastrous consequences for users." So says Huang Keji, chief architect of Huawei Storage.

After long-term accumulation, Huawei storage software has reached the scale of tens of millions of lines of code, and the storage research and development team still has high quality requirements for each line of code. For example, in the field of deduplication and compression, as the most effective technology to reduce storage costs and improve storage efficiency, re-deletion compression is considered to be an important symbol to measure the level of storage products, but once there is a problem with the re-delete compression function, resulting in data damage or loss, the consequences to users are extremely serious. Therefore, Huawei for re-deletion compression test is extremely strict, strictly prevent Bug omission, constantly polish the code, so as to be foolproof.

At the hardware level, Huawei Storage has a special reliability team and professional reliability laboratory to test a variety of hardware components, looking for possible loopholes and constantly improving in reliability.

"there are a lot of low-key and talented people on the front line of Huawei storage research and development." Huang Keji introduced.

For example, there is a technical tycoon who is obsessed with code. He wrote the converged storage file system, the replication core module, the all-flash architecture core module, and the deduplication module. This code god has written from enterprise to distributed core modules in Huawei history. " His way of expression is the code, and we all say that he is a floor-sweeping monk, and there is a lot of light hidden beneath Puhua's pragmatic appearance. "

"in addition to focus, Huawei attaches great importance to the full-stack ability of R & D talents. What is full-stack capability is full-stack skills and full-stack business." Xiang Fei introduced. In terms of full stack of skills, Huawei requires that developers must understand testing, testing must understand product design and back-end maintenance, and let R & D personnel pull through these skills through job rotation, so as to become more comprehensive; for example, within the product architecture, Huawei will let employees master as much technology as possible by letting employees develop in different subsystems. In addition, Huawei will also let R & D go to the front end to support sales and back-end support services on a regular basis.

In the case of storage architects, for example, there is an unwritten rule for storage in Huawei: architects also need to write code. Some architects, from RAID 2.0 features to Huawei OceanStor storage Dorado series V3 and V6 products, do their own architecture, write code, and maintain this state for a long time. So much so that Huawei storage insiders joked: "Storage architects who can't write code are not good storage presales."

A touch of red and blue color is the most ingenious-it is more appropriate to explain the ingenuity of Huawei storage.

The journey of the future is like the sea of stars

Now, the storage industry is at the next crossroads of change.

The continuous maturity and application of artificial intelligence technology is profoundly affecting the storage industry. Intelligent storage integrated with artificial intelligence technology is considered to be an important trend. it hopes to essentially solve the complexity of storage operation and operation and maintenance in the past, and make storage more simple.

"Intelligence will become the basic capability of storage." Zhou Yuefeng, president of Huawei's data storage and machine vision product line, said: "the future storage first needs Storage for AI, which can help users make better use of intelligent technology to meet the needs of training and applications such as AI. The second is AI in Storage, which integrates AI technology into the whole life cycle of storage, making storage management, performance, efficiency and maintenance better."

Huawei has been at the forefront of the industry in the exploration of smart storage. Huawei OceanStor storage Dorado V6 product is the industry's first AI-enhanced high-end storage system, and in the re-deletion compression algorithm, intelligent storage operation and maintenance and other aspects of the extensive use of machine learning, through the AI+ intelligent algorithm to achieve intelligent self-tuning of the storage system, so that storage can be used better and better.

However, intelligent storage still has a long way to go. At this stage, intelligent storage mainly solves the optimization work of operation and maintenance, such as fault location and business management, as well as some intelligent Cache algorithms and re-deletion compression algorithms based on AI, which make some professional storage operations automatic. In the future, storage will be similar to autopilot, which will intelligently select the corresponding storage functions and technologies according to different load conditions and business scenarios.

In addition, when data has become the most critical factor of production in the digital economy, storage as an important part of the data infrastructure, there are still many challenges. To help enterprises realize that data can be saved, moved and used well, all storage manufacturers, including Huawei Storage, need to continue to explore.

"our journey is like the sea of stars, and from then on we only care about both wind and rain." Huawei Storage has been on its way.

Tail note: four serial articles of Huawei Storage bar mitzvah: our journey is like the sea of stars has come to an end. Although it is a long text of ten thousand words, it is impossible to show the history of Huawei's storage over the past 18 years. With this article, I would like to pay tribute to the practitioners of China's storage industry. I hope you will not forget your original ideals and aspirations and forge ahead!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.