Shulou(Shulou.com)06/01 Report--

AnyShare Family 7 is how to solve the problem of backing up a large number of small files. In view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a simpler and easier way.

During this period of time, I spent a lot of time studying the backup of a large number of small files, and found that various schemes have great limitations.

I thought it was a world-class problem, but after watching the AnyShare Family 7 unveiling meeting on July 6, it dawned on me that the original solution was so simple, but it was really practical, and it did help AnyShare Family 7 users solve big problems.

AnyShare Family 7 is a new productivity platform for integrating, managing, and insight into unstructured data. It is actually an intelligent content cloud platform.

AnyShare Family 7 consists of five functional modules, namely, integration of business applications, content application development, document management, teamwork and data insight.

This new version of AnyShare Family 7 has many features and performance improvements over AnyShare Family 6. For example, the time it takes for intelligent search to build indexes is 5 times shorter than that of AnyShare Family 6.

In terms of overall architecture, AnyShare Family 7, like OpenText, adopts a modern micro-service architecture, which is more flexible and adaptable.

But these are not the focus of my attention, I am still concerned about the backup of a large number of small files, because I have been confused about these problems for many years.

He Hongfu, president of Aisu, also said at the AnyShare Family 7 unveiling meeting that massive unstructured data brings problems in the management of massive and small files.

For example, love to count their own, there are 103TB unstructured data, a total of 20.7 million files, the average file size is 5.21MB. In fact, 5MB is no longer a small file, but this is an average, and it is estimated that there are at least a few million files smaller than 1MB.

In order to save these unstructured data, AnyShare uses Ceph-based object storage (or third-party object storage) at the bottom. Because of the flat structure of object storage, it is more suitable to save a large number of files.

However, if you want to backup and restore the object storage in a conventional way, the backup and recovery speed will drop sharply after the file is smaller than 1MB. This is true of your own backup software, as is the backup software of Commvault, the market leader.

That is, the average size of backup 100TB is 1MB, and the backup and recovery time is about half a month. Such a speed is definitely unable to meet the RPO/RTO requirements of enterprises.

However, AnyShare Family 7 adopts a new backup idea, and the same data can increase the speed of backup and recovery to only about 5 days.

And the key thing is that not only the speed is increased by 3 to 4 times, but also the performance is stable and there is no jitter. In other words, the speed of backup and recovery has nothing to do with file size when it comes to 10MB.

It can also be seen from the test that the backup recovery of AnyShare Family 7 has no effect on small files. The backup speed of files below 10MB is the same, and the speed is stable above 250MB/s.

Why can love numbers be done so quickly? The main reason is that there is no standard S3 protocol to back up small files in object storage like other backup manufacturers.

As we all know, object storage basically has the technology of merging small files. In other words, small files are merged into large objects and saved in the object store. The merging of small files has two functions, one is to improve the processing efficiency of file reading and writing, and the other is to improve the storage space utilization.

The corresponding relationship between small files and large objects, object storage generally uses a database to store these metadata. When you use the standard S3 interface to access the merged small file, through the metadata stored in the database, you can easily find the corresponding large object and the corresponding offset, and read the small file.

Although object storage merges small files, the backup software will not feel the merged large objects if accessed through S3. Therefore, backup software backup and recovery still have to deal with the original small files, but can not directly back up the merged large objects, because the backup software does not know the corresponding relationship between small files and large objects.

Aishu started out as a backup software, so I think we should adopt a new way of thinking to solve this problem. That is, there is no need for object storage to merge small files, but AnyShare Family 7 adds an object storage (OSS) gateway in front of the object storage, and the OSS gateway carries out the work of merging small files. For example, all files smaller than 10MB are merged at the OSS gateway into large objects above 128MB, and then saved in the object store. Of course, the OSS gateway needs to have a database to hold these correspondence.

When AnyBackup Family 7 backs up AnyShare Family 7, AnyBackup Family 7 is aware of the existence of OSS gateways and backs up only the merged large objects and, of course, the corresponding metadata. When restoring, in addition to restoring the merged large objects, it also recovers its corresponding metadata. In this way, for AnyBackup backup software, there is no sense of the existence of small files, this is what we have seen above, small files below 10MB, regardless of file size, backup recovery performance is the same.

Wonderful, really wonderful. Aisu makes full use of the cooperation advantages of the two R & D teams of AnyBackup and AnyShare to perfectly solve the problem of backup and recovery of massive small files in AnyShare Family 7.

You may be worried about what to do if small files need to be modified. Because it is very troublesome to modify a small file in a large object. However, AnyShare as a content management platform, this is very rare. Because content management is basically a document that the enterprise has completed. Of course, it doesn't matter if changes are needed, because the content management platform itself has multi-version management capabilities, and the modified files are saved as new versions, so that the stability of large objects that have been archived will not be destroyed.

In order to allay everyone's concerns, Aisu also announced a high-performance backup recovery guarantee plan at this AnyShare Family 7 unveiling meeting.

In other words, regardless of the number and size of files, the backup and recovery speed of AnyShare Family 7 is no less than that of 100MB/s, and by the end of next year, if a third-party backup software backs up a large number of small files faster than Aisu AnyBackup Family 7, then customers will get a permanent license for AnyShare Family 7 free of charge.

It seems that Aishu is very confident, promising not only the absolute speed of backup recovery, but also relative competition. Aishu believes that within a year and a half from now on, its unique dedicated backup performance should be unsurpassed.

Although this is a dedicated backup solution for AnyShare Family 7, it does help AnyShare Family 7 to have this huge differentiation advantage over other content management platforms. Because not all content management vendors have their own backup software, and even if they do, they may not have thought of this idea.

The universal problem of backing up a large number of small files has not been solved, because this solution is only aimed at AnyShare Family 7. However, for users using AnyShare Family 7, this is sufficient. AnyShare is not only an intelligent content cloud platform, but also has its own backup function, so you no longer have to worry about the data protection problems caused by the increasing number of small files.

This is the answer to the question about how AnyShare Family 7 solves the problem of backing up a large number of small files. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.