Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about the common use cases and architecture of SRM. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

Streams Replication Manager (SRM) is an enterprise replication solution that enables fault-tolerant, scalable, and robust cross-cluster Kafka topic replication. SRM provides the ability to dynamically change the configuration and keep Topic properties synchronized between high-performance clusters. SRM also provides custom extensions to facilitate installation, management, and monitoring, making SRM a complete replication solution built for mission-critical workloads. This article focuses on the main use cases of SRM and the implementation architecture of use cases.

Main use case

Learn about the main use cases of SRM.

Apache Kafka has become an important part of the enterprise data pipeline and is used to track clickstream event data, collect logs, collect metrics, and become the enterprise data bus in micro-service-based architectures. Kafka supports internal replication to support data availability in the cluster. However, as Kafka-based applications become critical, enterprises require data availability and persistence across clusters and site failures.

Data replication across clusters and sites is critical for the following use cases:

Disaster recovery

A common enterprise use case for cross-cluster replication is to ensure business continuity in the presence of cluster-or data center-wide disruptions.

Summary and analysis

Aggregate data from multiple streaming pipelines that may come from multiple data centers to run batch analysis jobs to provide an overall view of the entire enterprise.

Data deployment after analysis

This is in contrast to the aggregation use case, where data generated by an analysis application in a cluster (such as an aggregation cluster) may be broadcast to multiple clusters and may be propagated across data centers for end-user use.

Isolation

For performance or security reasons, data needs to be replicated between different environments to isolate access. In many deployments, the ingestion cluster is isolated from the consumption cluster.

Geographical proximity

In geographically distributed access modes that require low latency, replication is used to move data closer to the access location.

Cloud Migration

As more and more enterprises have on-premises and cloud services, Kafka replication can be used to migrate data to public or private clouds and back.

Law and Compliance

Much like isolation use cases, policy-driven replication is used to restrict the data accessible in the cluster to meet legal and regulatory compliance requirements.

Highly available Kafka architecture

The high availability deployment of Kafka must be able to survive a complete outage of a single cluster while continuing to process events without data loss. With SRM, you can implement a high-availability Apache Kafka deployment that follows an active / standby or active / active model.

Active and standby architecture

In the active / standby scenario, you will set up two Kafka clusters and configure SRM to replicate themes in both directions between the two clusters. The VIP or load balancer instructs your producer to extract messages into the active cluster from which consumer groups are being read.

Figure 1. Active / standby architecture standard operation

In the event of a disaster, the VIP or load balancer directs the producer to the standby cluster. You can easily migrate your consumer group to start reading data from the alternate cluster, or if the final consumer delay is acceptable for your use case, just wait until the primary cluster is restored.

When the main cluster shuts down, your producer can still extract it. After the primary cluster is restored, SRM automatically handles the synchronization of the two clusters, making the failure recovery seamless.

Figure 2. Active / standby architecture cluster failure

When an existing disaster recovery site with an established policy is available, it makes sense to implement an active / standby architecture, and your goals include not losing reception during a disaster and backing up at the disaster recovery site.

Active / active architecture

In an active / active scenario, the producer load can be balanced to the primary or secondary cluster. SRM is configured to replicate themes in both directions between the two clusters. What makes this architecture an active / active architecture is that you now have the fact that consumers can read from both clusters at the same time, basically like a consumer group across clusters. In the event of a disaster, the VIP or load balancer directs the producer to the secondary cluster, and the secondary cluster consumer group can still process messages. When the primary cluster shuts down, your producers can still receive and your consumers can still process messages. This will result in zero downtime and automatic failover once the primary cluster comes back online in the event of a disaster.

Figure 1. Active / active architecture

Replication across data center

Example of SRM's cross-data center replication architecture.

Some applications need to have local high availability not only in one data center or one availability zone, but also throughout the data center. You can use SRM to set up replication between Kafka clusters in different data centers, making messages available to consumers in each data center.

If the primary data center fails, the load balancer directs your producer to the local or nearest data center. SRM is configured to replicate themes across all data centers. If you are using more than two datacenters, configure SRM to create "replication circles" to ensure that a single datacenter failure (for example, us-north in the following example) does not stop replication between the remaining clusters.

Figure 1. Replicate the architecture across data center

Cluster migration architecture

Local-to-cloud migration and Kafka version upgrade

Local-to-cloud and Kafka version upgrade sample schemas for SRM.

If you have a local Apache Kafka cluster that you want to migrate to the cloud, you must migrate not only consumers and producers, but also topics and their messages to the new cloud-based cluster.

After replication is set up through SRM, you only need to point the consumer to the new agent to start processing messages from the cloud cluster. This approach ensures that historical data retained in the local Kafka cluster is migrated to the cloud cluster, so that you can replay messages directly from the cloud without having to return to the local cluster.

Figure 1. Local cluster migration. When SRM replicates messages, producers and consumers are using the local cluster.

After migrating clusters, producers, and consumers to the cloud, you can use SRM to change the direction of replication and use the local cluster as a DR cluster.

Producers and consumers have migrated to cloud clusters, and local clusters are used for disaster recovery.

Figure 2. Cluster migration to cloud

If the Kafka cluster must be upgraded to a newer version and cannot be upgraded locally, the new cluster can be configured using the same migration method, and all existing topics and messages can be copied using SRM before migrating producers and consumers. Interact with the new cluster.

Summary and analysis

Example of an aggregation analysis architecture for SRM.

SRM can be used to aggregate data from multiple streaming pipelines that may come from multiple data centers to run batch analysis jobs to provide an overall view of the entire enterprise.

Figure 1. Analysis summary

These are the common use cases and architectures of SRM shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.