Shulou(Shulou.com)06/02 Report--

The storage of container cloud platform is not only the persistent storage of data, but also the storage requirements of container cloud platform and image storage, and even the storage requirements of cloud middleware such as Kafka.

The huge impact and challenges brought by the financial business of Internet companies to the development of the financial industry forces the financial industry to consider the development of Internet financial services and mobile services, which puts forward new requirements and challenges to the technological platform and technical capabilities. Internet financial services and mobile services are required to respond quickly to business needs, even at the level of hours and minutes, which requires agile response, agile development, continuous integration, continuous deployment and continuous improvement of the business. to achieve automated operation and maintenance, fault tolerance, exception repair, elastic scaling, grayscale publishing and other capabilities, environmental consistency requirements, infrastructure service platform and other requirements need to be met. To survive in the fierce competition. The development trend of the business of financial companies is like sailing against the current. If you do not advance, you will fall behind. You have to adopt new technologies and platforms.

The container cloud platform provides the capabilities of environment consistency, support for auto scaling, grayscale publishing and blue-green deployment, and also fits with the integrated methodology of DevOps development, operation and maintenance, and micro-service architecture to achieve continuous integration, continuous deployment, continuous monitoring, continuous feedback, and continuous improvement of closed loop. In addition, the deployment and operation and maintenance of big data, AI and other applications also need to consider building a common infrastructure service platform, which is the right time for container cloud.

With the maturity of container technology and the actual landing of container cloud platform, more and more enterprises have built or tried to build container cloud platform. How to manage and use the basic resources such as CPU, memory, storage and network is one of the key capabilities of the container cloud platform. As the infrastructure resources of the container cloud platform, storage ensures the security of the container cloud data, so that the data can be safely and persistently stored, queried and not lost. However, the use of container technology and container cloud platform for storage is different from the original. Persistent data storage is not supported in container internal storage, so you need to consider mounting external storage to the container to ensure the security of data in container migration, extinction, restart and other activities. Data is the core asset of an enterprise, and it has become more and more valuable in the fierce market competition. Therefore, when building a container cloud platform, we must consider the storage of the container, such as: storage mode, storage media, storage products, storage data, performance requirements and so on. In addition, the storage of container cloud platform is not only the persistent storage of data, but also the storage requirements of container cloud platform and image storage, and even the storage requirements of cloud middleware such as Kafka.

I. Container storage mode

Storage resource is one of the infrastructure resources of container cloud, which ensures the normal operation and data persistence of container cloud platform. The methods of container cloud storage include container internal storage, container host disk storage, container remote volume storage and so on.

1. Internal storage mode of the container

The internal storage of the container is the read-write layer of the container, but when the container is destroyed, the read-write layer is also destroyed, and the data in the read-write layer will be lost, just like the data in memory. When the application process is closed, the data in memory will be emptied. The internal storage mode of the container can only be used as a temporary storage mode when the container is running.

2. Local disk storage mode of container host

Container host local disk storage uses local storage volumes, which are maps to native disks. The local volume cannot create a snapshot. The advantage is that there is no need for independent storage, the creation operation is simple, and the reading and writing efficiency is high. The disadvantage is that container drift cannot be implemented between nodes.

3. Container remote volume storage method

Remote volumes include NFS and distributed remote storage volumes. The creation of NFS needs to increase the information of devices, read and write permissions and device directories; distributed storage volumes include Ceph, GlusterFS, ScaleIO, Portworx and so on.

Second, the importance of container cloud storage

Storage is one of the basic resources provided by the container cloud platform. To use the container cloud platform safely and stably, you need to do some research and planning in advance before adopting and implementing the container cloud platform, because:

1. Data is an important asset of the enterprise, to ensure that the data is not lost, data integrity, in order to better carry out business.

two。 Container and virtual machine or physical machine technology implementation focus on different, containers focus on stateless applications, to support stateful applications, data storage must be considered and planned in advance based on business requirements.

3. Container cloud is the basic platform, which involves platform components, mirrors, applications, middleware and other aspects, each of which may have different storage requirements. In order to achieve ideal performance and results, we need to comprehensively consider every aspect, storage and so on as infrastructure resources, is an essential part.

4. It is important to understand that containers are used to host applications. The application of data at all levels has potential value, and capturing and processing, storing and analyzing these data is a step to obtain value. Therefore, the persistence of application data is one of the important basic capabilities of container cloud platform to support business applications. Build a good foundation in order to better serve business applications.

Everything in advance is established, and if not, it is wasted. It illustrates the importance of planning. Temporary cramming will always have a more or less impact on business. Container cloud is the infrastructure platform, storage is the basic resource of container cloud, and infrastructure resources determine the ability of upper layer application services. At present, IaaS layer can not directly provide all the storage resources needed for container cloud platform, and there are technical difficulties in docking with IaaS. Without complete data, it is difficult to respond to business requirements quickly. It is difficult to quickly surpass or follow competitors, so it is difficult to implement cloud business applications supported by big data and AI technologies.

In the final analysis, data, or data! It is the core asset, which needs to pay attention to its safe preservation and backup, its storage mode, media, storage performance and so on.

3. Several aspects to be considered in container cloud storage

When using a container cloud platform, the use of storage probably needs to be considered in the following aspects: storage required by the container cloud platform, persistent storage of application data, image storage, and storage required for middleware deployment. The most important thing is the storage and persistent data storage of the container cloud platform itself. The amount of data stored in persistence may be large in some companies and small in others. It is even more important to plan when there is a large amount of storage demand. An important aspect of data storage is that it supports container migration, which is migrated with the container. Different business applications have different requirements for storage, so there is no clear standard here, which needs to be determined according to the actual situation. The more important the data store, the higher the security requirements.

1. Storage of the container cloud platform itself: the container cloud platform itself has many components, such as ETCD, which stores the status and configuration information of the container cloud platform, which has high requirements for storage performance. In the event of an accident in etcd, the container cloud platform may be paralyzed, so etcd storage requires high stability, security, performance and other aspects. You may also need to consider the cluster deployment of etcd.

two。 Persistent storage of application data: including host local storage volume and remote storage volume. The local storage volume is the mapping of the local disk. The local volume capacity and read and write performance are determined by the disk specification and performance, and the local volume cannot create a snapshot. The advantage is that it does not need independent storage, the creation operation is simple, and the read and write efficiency is high; the disadvantage is that container drift cannot be realized between nodes. Remote volumes include NFS and distribution is the remote storage volume and is usually the first choice for container cloud platforms.

3. Image storage: the image repository is used to store the space needed for application images and middleware images.

4. Storage required for middleware deployment: storage requirements for running some middleware. Different middleware may have different requirements for storage.

Fourth, persistent storage requirements

As users, we pay more attention to data persistent storage. Data is the core asset, and important data must be persisted and backed up, while the internal storage of containers is difficult to achieve persistence and local storage is difficult to migrate. So it is more likely to use remote storage volumes to meet persistent storage needs. This is because:

1. From a business point of view, application runtime exceptions, state, data, and so on are stored for recovery or retry execution.

two。 Log data is an important basis for business tracking, query, statistics, analysis and so on. All operations should be based on persistent storage of data.

3. The auto scaling feature of the container is very suitable for the expansion and scaling needs of microservices. Logs, runtime data, results, etc., migrate with the container and do not disappear with the destruction of the container.

4.Kafka, mysql, etc., need persistent storage support to deploy and save data.

5. The accumulation of the amount of data will bring about qualitative changes.

6. Big data business, AI business and other deployment requirements based on container cloud.

Big data and artificial intelligence all need data to support, so the data storage capacity of this piece will be larger and larger. For example, daily log data may have dozens of G, hundreds of G, these data after cleaning processing analysis, may bring new value growth points, but also bring more and more new business innovation. However, with the accumulation of the amount of data, it will also put pressure on the storage. Therefore, different data may need to consider different storage schemes. This may involve data governance issues. Remote volumes include NFS, NAS, SAN, Ceph, GusterFS and other products to choose from, and different products are selected for different needs.

Mirrors have relatively simple requirements for storage, and mirrors usually need to be constantly updated. However, mirroring needs to consider the image storage of mirror libraries in different environments. In general, image repositories can be deployed without containment, which is relatively convenient to manage host storage resources.

Middleware storage is more because when middleware is deployed, different middleware may have different requirements for storage. The most typical examples, such as Kafka, require multiple disks, file systems using XFS, and the noatime option when Mount file systems are used to achieve better performance. This requires determining what kind of storage to choose based on different components. However, it is not particularly convenient to implement it on the container cloud. Consider non-container deployment.

Fifth, the idea of containerless storage

At present, when it comes to container cloud storage, everyone will think of Ceph or GlusterFS. When we tested the PoC, we also tested it with the crowd. Everyone said it was such a match, so we just matched it. But when you think about it later, the purpose of our mount storage volume is for data persistence and stateful applications. However, according to the original concept of cloud, if you do not move the cloud rigidly for the sake of cloud, you may not need too much storage resources. In addition, the data can be collected to the data center or data processing terminal through real-time data flow, so that the data can run and flow, instead of just storing it, and fully develop the value of the data. Sleeping and static data can not bring value, the running data is valuable, the higher the real-time data, the more valuable the data. This is also the value of big data analysis on the construction of big data platform.

From this point of view, we can make the data run without having to store it all on disk, or we can use this idea on the container cloud platform to throw the data out of the container and then further process or store it. This may make the use of storage resources much easier. For example, application service logs can be collected directly from the container from log files or standard output, and then thrown into message middleware such as ES or Kafka. This function is provided by the container cloud platform, so users do not need to care about the details of the implementation. More importantly, the data can be further processed in real time through ES or Kafka, and some problems of persistent storage of data on the container platform can be avoided.

This is just a personal idea, and it may be more important to consider the persistent storage of data on the container cloud platform. There is a lot of discussion about the corresponding products and technologies, so I won't repeat them here.

VI. Selection of container cloud storage scheme

When using container cloud to choose storage, it needs to be determined based on the actual business requirements. We deeply understand that the container cloud platform is used to host applications and serve enterprise business applications. This is what we have repeatedly emphasized when choosing container cloud products. We use container cloud not for containers, but for business applications, for business. Different businesses have different requirements for storage. There are relatively high security requirements, but also can tolerate data loss and so on. Different business requirements, the storage scheme, storage products will be different, because the products are different, the capital price paid is not the same.

When choosing products, you also need to consider the enterprise's own technical strength. Open source products have the advantages of open source products, and commercial products also have the advantages of commercial products. For financial companies like us, we need to consider more about the security of data. We will be very cautious when using open source products, and post-service is a very important factor. There has to be a company to cover it up, even if you spend more money to buy peace of mind.

Also consider whether you have the ability to choose a product or technology. Open source products require high technical capabilities, so if you choose open source products, it is best to have corresponding technical personnel reserve. Open source products are advanced in technology, and the version iteration changes relatively fast, so we need to have the corresponding manpower to control it. If this can not be done, it is appropriate to choose commercial products, although the cost is high, but it has been verified by many enterprises. Especially for traditional enterprises, the technical strength is still difficult to meet the requirements, unable to control the products, it is best not to easily use open source products.

You may also need to consider container cloud products and architecture solutions, as well as whether there is technical support for the IaaS layer, and the technologies adopted by the IaaS layer.

Storage, Container Cloud and big data

We consider that storage is not only needed for the container cloud, but also for the big data platform. Storage can be considered as a separate piece and can not be tied to the container cloud project. Integrated container cloud platform, big data platform construction, unified to consider the construction of storage resources, may be better. Our ultimate goal is to provide storage resources for the container cloud platform.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.