Shulou(Shulou.com)06/01 Report--

ORACLE RAC was not designed to be active across datacenters. It was designed to achieve Load Balancer and high availability for multiple hosts with shared storage in a datacenter. But because its architecture does have the potential to achieve Load Balancer and high availability across data centers, several storage vendors have extended its usage environment and come up with solutions across data centers. ORACLE takes this by default, but recommends that all solutions be carefully tested before being put into customer production.

For RAC, the biggest bottleneck for a cross-data center solution is the interconnect between nodes because of its high latency and bandwidth requirements. Generally speaking, the local interconnect transmission delay is between 1 and 2ms, and the local IO delay is between 8 and 15 ms. These two delays have a significant impact on performance, and if you use a dual data center scenario, they both severely impact performance as the room distance increases. Moreover, due to the low delay base of interconnect (1~2ms), the delay caused by the distance between the equipment room has a greater impact on the entire interconnect: think about if the transmission delay of 2ms is caused by the extension of the distance, the delay growth for interconnect is 100%~200%, and for IO is only 15%~25%. Of course, with the heavy use of SSD in storage, the impact of distance on IO is also increasing.

To visually demonstrate the impact of transmission distance on IO and interconnect delays, Figures 1 and 2 show HP test results as a reference:

figure I

Figure 1 shows how IO latency is affected by distance. This test result is obtained when Buffer-to-Buffer Credits(BBC) is enabled. The BBC feature allows a large number of unanswered packets to be stored in the buffer while continuing to send packets. In the case of large data traffic, the farther the distance, the greater the role of BBC.

If BBC is turned on at a distance of 100km, IO latency increases by about 43% compared to local; if BBC is not turned on, IO latency increases by about 120~140%. Another manufacturer's test showed that at a distance of 20km, not turning on the BBC would result in a 20 - 24% drop in traffic.

Figure 2 tests RAC with one or two interconnects under high load and low load, respectively, to examine the effect of distance on interconnects.

figure II

Figure 2: Two findings from this test:

1. Two links can reduce latency by about 50% compared to one link under high load conditions

2. 100km can result in a delay increase of approximately 1 ms.

Figures 1 and 2 show the effect of distance on the link, while Figures 3 and 4 below show the effect of distance on overall RAC performance.

Since Buffer-to-Buffer Credits(BBC) function has a great impact on transmission performance during long-distance transmission, it should be emphasized that Figure 3 shows the test results obtained by two manufacturers when BBC function is enabled. For comparison, Figure 4 shows the test results without BBC enabled.

As can be seen from Figure 3 and Figure 4, when BBC is turned on, the performance of the solutions of the two test vendors is quite good. But if you don't turn on BBC, performance drops sharply over distance. Considering that the performance of dual-node RAC with good configuration in the same computer room is about 30~60% higher than that of single node, if the performance drops by more than 20% due to remote computer room RAC cluster, it is necessary to carefully consider whether to use RAC scheme.

There are two other things to note:

1. The test results given by each manufacturer are often the best data measured in the case of extreme optimization, and the optimization degree of the actual customer site is often significantly lower than the manufacturer's test environment.

2. Manufacturers often give only the best test results for themselves. For example, the test distance range given by the two manufacturers in Figure 3 is different, and the reason may be that beyond this range, the performance will have a large decline.

Based on the above tests, ORACLE recommends considering whether to adopt RAC active-active scheme based on the distance of cables connected to the machine room:

1. If the distance is less than 50km, double active RAC can be considered.

2. If the distance is greater than 50km and less than 100km, consider using double-active RAC. To use it requires very careful testing.

3. If the distance is greater than 100km, it is not recommended to use dual-active RAC. RAC one node can be considered as a highly reliable cluster ①.

RAC one node is a variant of RAC, the effect is somewhat similar to the traditional HP MC/SG + oracle scheme, because only one node will be running at the same time, there will be no large amount of data running on the interconnect.

If you decide to use RAC across data centers, consider carefully the following configuration recommendations:

1. The interconnect and IO links use an unshared, end-to-end cable direct connection, known in English as "Dark Fibre."

2. It is strongly recommended to turn on BBC functionality on the transmission path.

3. Configure 3 voting disks or voting files in ORACLE clusterware. Each of the two data centers is equipped with a voting disk, and a voting file based on NFS or ISCSI is configured in the third computer room to improve the reliability of RAC system.

The first two suggestions are easy to understand from the previous test results. Let's make a detailed explanation of the third suggestion:

If the voting file based on the third server room is not configured, when the link between the two data servers is broken, the hosts on both sides can only access the local storage without knowing the status of the other party. At this point, since there is no third-party arbitration, both RAC hosts will exit the cluster, resulting in business interruption. Because if not, it will lead to data chaos, and the consequences will be more serious.

Remote voting file configuration considerations:

In general, Oracle clusterware accesses the Voting file once per second by reading and writing less than 1 kilobyte of data. The response to each write request IO should be returned within 200 seconds (default, long disk timeout) or 27 seconds (configurable, short disk timeout). For this reason, Oracle recommends that the write IO of voting fiel should return within 14 (27/2) seconds, with a transmission bandwidth of at least 128 kbps.

Arbitral Competition Problem for Storage Active and RAC Clusters

For HP XP7, because of the use of virtual disk array technology, you only need to configure voting disk/file on the virtual disk array to avoid competition. Because RAC nodes that do not access voting disks on the virtual disk array are unlikely to be arbitrated alive by RAC clusterware. In this case, no RAC configuration is required for remote voting files.

For HP 3par, a quasi-storage active-active solution using ALUA protocol, because RAC nodes only use one physical array at the same time, the result is similar to XP7, as long as the voting disks are configured as peer persistence volumes, arbitration conflicts can be avoided. In this case, no RAC configuration is required for remote voting files.

For other storage active-active solution providers that do not use virtual disk array technology, especially those that do local read-write optimization, this is a very serious consideration. Because most of this storage active-active provider arbitration is implemented using a virtual machine at a third location, it is recommended that this virtual machine be as physically close to the RAC's third Voting file as possible to reduce the likelihood of conflicting arbitration results due to physical differences.

l Some storage vendors provide manual adjustments to the arbitration algorithm to ensure that the storage arbitration results are the same as RAC. There are no details, so it is inconvenient to comment, but oracle officials are opposed to this.

Bibliography:

《Oracle RAC and Oracle RAC One Node on Extended Distance (Stretched)Clusters》

《Using standard NFS to support a third voting file for extended cluster configurations - OracleClusterware 11g Release 2》

《Oracle Clusterware Administration and Deployment Guide》

《HP 3Par Remote Copy Software User's guide》

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.