Shulou(Shulou.com)06/02 Report--

Environment: RHEL5.8 RAC 11.2.0.3.0

1: view ORC and voting disk information:

In 11g Release 2 your voting disk data is automatically backed up in the OCR whenever there is a configuration change.

So when restoring, you can restore the backup OCR, which is different from 10g. You don't need to back up voting disk, just back up OCR.

2: use the cluvfy tool to check OCR integrity

[grid@rac1] $cluvfy comp ocr-n all

Verifying OCR integrity

Checking OCR integrity...

Checking the absence of a non-clustered configuration...

All nodes free of non-clustered, local-only configurations

ASM Running check passed. ASM is running on all specified nodes

Checking OCR config file "/ etc/oracle/ocr.loc"...

OCR config file "/ etc/oracle/ocr.loc" check successful

Disk group for ocr location "+ CRSDATA" available on all the nodes

NOTE:

This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.

OCR integrity check passed

Verification of OCR integrity was successful.

3: use ocrcheck to check the integrity of OCR content

[grid@rac1 ~] $ocrcheck

Status of Oracle Cluster Registry is as follows:

Version: 3

Total space (kbytes): 262120

Used space (kbytes): 3016

Available space (kbytes): 259104

ID: 1236405787

Device/File Name: + CRSDATA

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user-if a root user executes ocrcheck, Logical corruption check succeeded will be displayed

4: detect the information of voting disk

[grid@rac1 ~] $crsctl query css votedisk

# # STATE File Universal Id File Name Disk group

1. ONLINE 2b1bd0c122584f5abf72033b2b2d26bd (/ dev/asm-b_crs) [CRSDATA]

2. ONLINE 2bc03776cdd94f5cbfb9165c473fdb0e (/ dev/asm-c_crs) [CRSDATA]

3. ONLINE 3b43c39513a64f2dbf7083a9510ada89 (/ dev/asm-d_crs) [CRSDATA]

Located 3 voting disk (s).

As can be seen from the above, both OCR and voting disk are located in the + CRSDATA disk group. Note that the + CRSDATA disk group also has the boot parameter file of ASM. ASM startup is based on the kfdhdb.spfile of the disk head pointing to the UA NUMBER of this disk on ASM to read the spfile file.

5: manually back up a copy of OCR information:

[root@rac1 grid] # ocrconfig-export / tmp/ocr_20130717.dmp

[root@rac1 grid] # ll / tmp/ocr_20130717.dmp-h

-rw- 1 root root 102K Jul 17 14:45 / tmp/ocr_20130717.dmp

6: view OCR automatic backup information

[grid@rac1 ~] $ocrconfig-showbackup

Rac1 2013-07-16 15:45:24 / u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr

Rac2 2013-07-16 08:13:38 / u01/app/11.2.0.3/grid/cdata/ad-cluster/backup01.ocr

Rac2 2013-07-16 04:14:09 / u01/app/11.2.0.3/grid/cdata/ad-cluster/backup02.ocr

Rac2 2013-07-16 00:14:38 / u01/app/11.2.0.3/grid/cdata/ad-cluster/day.ocr

Rac2 2013-07-07 04:40:11 / u01/app/11.2.0.3/grid/cdata/ad-cluster/week.ocr

PROT-25: Manual backups for the Oracle Cluster Registry are not available

7: save an ASM parameter file. If you don't save it in advance, you can get a copy from $CRS_HOME/dbs/init.ora. The details of the following startup parameters

[grid@rac1 dbs] $sqlplus / as sysasm

SQL > create pfile='/tmp/asm_pfile_130717.txt' from spfile

File created.

8: destroy the disk group + CRSDATA that holds OCR information

[root@rac1 dev] # dd if=/dev/zero of=/dev/asm-b_crs bs=1024 count=1000

[root@rac1 dev] # dd if=/dev/zero of=/dev/asm-c_crs bs=1024 count=1000

9: after destroying disks b and c, the test passed and no error was reported. Stop crs in rac1 and rac2

[root@rac1 dev] # crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.crsd' on' rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

.

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 dev] # crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.crsd' on' rac1'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'

.

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac1 dev] # ps-ef | grep ora_

Root 16189 32265 0 16:26 pts/0 00:00:00 grep ora_

[root@rac1 dev] # ps-ef | grep asm_

Root 16195 32265 0 16:26 pts/0 00:00:00 grep asm_

10: start CRS again and report an error

[root@rac1 dev] # crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@rac1] # tail-50f / u01/app/11.2.0.3/grid/log/rac1/alertrac1.log

[cssd (16559)] CRS-1637:Unable to locate configured voting file with ID 2b1bd0c1-22584f5aMurbf72033bMur2b2d26bd; details at (: CSSNM00020:) in / u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log

2013-07-17 16 purl 2815.947

[cssd (16559)] CRS-1637:Unable to locate configured voting file with ID 2bc03776 CSSNM00020 cdd94f5c murbfb9165cMuz 473fdb0e; details at (: CSSNM00020:) in / u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log

2013-07-17 16 purl 2815.947

[cssd (16559)] CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (: CSSNM00021:) in / u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log

2013-07-17 16 purl 2815.948

[cssd (16559)] CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (: CSSSC00012:) in / u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log

2013-07-17 16 purl 287 073

[cssd (16559)] CRS-1603:CSSD on node rac1 shutdown by user.

Ocrcheck detection error:

[root@rac1 dev] # ocrcheck

PROT-602: Failed to retrieve data from the cluster registry

PROC-26: Error while accessing the physical storage

11: forcibly shut down CRS:

[root@rac1 dev] # crsctl stop crs-f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

[root@rac1 dev] # crsctl stop crs-f

CRS-2797: Shutdown is already in progress for 'rac1', waiting for it to complete

CRS-2797: Shutdown is already in progress for 'rac1', waiting for it to complete

CRS-4133: Oracle High Availability Services has been stopped.

12: start rac1 in exclusive mode

[root@rac1 dev] # crsctl start crs-excl-nocrs

CRS-4123: Oracle High Availability Services has been started.

CRS-2672: Attempting to start 'ora.mdnsd' on' rac1'

CRS-2676: Start of 'ora.mdnsd' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on' rac1'

CRS-2676: Start of 'ora.gpnpd' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on' rac1'

CRS-2672: Attempting to start 'ora.gipcd' on' rac1'

CRS-2676: Start of 'ora.cssdmonitor' on' rac1' succeeded

CRS-2676: Start of 'ora.gipcd' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on' rac1'

CRS-2672: Attempting to start 'ora.diskmon' on' rac1'

CRS-2676: Start of 'ora.diskmon' on' rac1' succeeded

CRS-2676: Start of 'ora.cssd' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.drivers.acfs' on' rac1'

CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on' rac1'

CRS-2672: Attempting to start 'ora.ctssd' on' rac1'

CRS-2681: Clean of 'ora.cluster_interconnect.haip' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on' rac1'

CRS-2676: Start of 'ora.ctssd' on' rac1' succeeded

CRS-2676: Start of 'ora.drivers.acfs' on' rac1' succeeded

CRS-2676: Start of 'ora.cluster_interconnect.haip' on' rac1' succeeded

CRS-2672: Attempting to start 'ora.asm' on' rac1'

CRS-2676: Start of 'ora.asm' on' rac1' succeeded

12: create CRSVOTEDISK disk groups and spfile

[grid@rac1 ~] $asmcmd

ASMCMD > ls

Empty

[grid@rac1 ~] $sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Wed Jul 17 16:58:18 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL > show parameter spfile

NAME TYPEVALUE

-

Spfile string

SQL > create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/ dev/asm-d_crs'

2 attribute 'compatible.asm'='11.2.0.0.0',' compatible.rdbms'='11.2.0.0.0'

Create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/ dev/asm-d_crs'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15033: disk'/ dev/asm-d_crs' belongs to diskgroup "CRSDATA"-the error is reported here because asm-d_crs did not clear the disk header information

Clear asm-d_crs disk head information

[root@rac1 dev] # dd if=/dev/zero of=/dev/asm-d_crs bs=1024 count=1000

SQL > create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'

2 attribute 'compatible.asm'='11.2.0.0.0',' compatible.rdbms'='11.2.0.0.0'

Diskgroup created.

SQL > create spfile='+CRSVOTEDISK' from pfile='/tmp/asm_pfile_130717.txt'

File created.

SQL > quit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit Production

With the Real Application Clusters and Automatic Storage Management options

[grid@rac1 ~] $asmcmd

ASMCMD > ls

CRSVOTEDISK/

ASMCMD > ls CRSVOTEDISK

Ad-cluster/

ASMCMD > ls CRSVOTEDISK/ad-cluster/

ASMPARAMETERFILE/

ASMCMD > ls CRSVOTEDISK/ad-cluster/ASMPARAMETERFILE

REGISTRY.253.821034567

13:Restore OCR from backup:

Change the original disk group + CRSDATA to the newly created disk group + CRSVOTEDISK

[root@rac1 dev] # vim / etc/oracle/ocr.loc

Ocrconfig_loc=+CRSVOTEDISK

Local_only=FALSE

[root@rac1 dev] # ocrconfig-restore / u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr

You can see that an OCRFILE folder has been added

ASMCMD > ls CRSVOTEDISK/ad-cluster

ASMPARAMETERFILE/

OCRFILE/

ASMCMD > ls CRSVOTEDISK/ad-cluster/OCRFILE-l

Type Redund Striped Time Sys Name

OCRFILE MIRROR COARSE JUL 17 17:00:00 Y REGISTRY.255.821036449

ASMCMD > ls CRSVOTEDISK/ad-cluster/ASMPARAMETERFILE-l

Type Redund Striped Time Sys Name

ASMPARAMETERFILE MIRROR COARSE JUL 17 17:00:00 Y REGISTRY.253.821034567

Detection successful

[root@rac1 dev] # ocrcheck

Status of Oracle Cluster Registry is as follows:

Version: 3

Total space (kbytes): 262120

Used space (kbytes): 3016

Available space (kbytes): 259104

ID: 1236405787

Device/File Name: + CRSVOTEDISK

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

14:Restore the Voting Disk:

[root@rac1 dev] # crsctl query css votedisk

# # STATE File Universal Id File Name Disk group

1. OFFLINE 2b1bd0c122584f5abf72033b2b2d26bd () []

2. OFFLINE 2bc03776cdd94f5cbfb9165c473fdb0e () []

3. ONLINE 3b43c39513a64f2dbf7083a9510ada89 (/ dev/asm-d_crs) [CRSDATA]

Located 3 voting disk (s).

[root@rac1 dev] # crsctl replace votedisk + CRSVOTEDISK

CRS-4602: Failed 27 to add voting file 5818c2c531394f45bff13c5a7532c8d4.

CRS-4602: Failed 27 to add voting file 1ce0436528624faabf7d4a1dd8dc978a.

CRS-4602: Failed 27 to add voting file 09def2b244af4f42bf13679a8aa0ff73.

Failure 27 with Cluster Synchronization Services while deleting voting disk.

Failure 27 with Cluster Synchronization Services while deleting voting disk.

Failure 27 with Cluster Synchronization Services while deleting voting disk.

Failed to replace voting disk group with + CRSVOTEDISK.

CRS-4000: Command Replace failed, or completed with errors.

The error here is caused by the fact that asm-d_crs did not clear the disk head information at the beginning.

= = failed to restore voting disk here, let's start again and try to recover again =

The following should be noted when restoring:

After crsctl start crs-excl-nocrs starts, close ASM immediately. Do not create create diskgroup CRSVOTEDISK disk group immediately, and then start ASM with parameters.

Otherwise, the following error may be reported when creating a disk group:

For example: (just take a look at the following operation until: start to resume the operation again)

[grid@rac1 ~] $sqlplus / as sysasm

SQL > create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'

2 attribute 'compatible.asm'='11.2.0.0.0',' compatible.rdbms'='11.2.0.0.0'

Create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'

*

ERROR at line 1:

ORA-15018: diskgroup cannot be created

ORA-15031: disk specification'/ dev/asm-d_crs' matches no disks

ORA-15014: path'/ dev/asm-d_crs' is not in the discovery set

ORA-15031: disk specification'/ dev/asm-c_crs' matches no disks

ORA-15014: path'/ dev/asm-c_crs' is not in the discovery set

ORA-15031: disk specification'/ dev/asm-b_crs' matches no disks

ORA-15014: path'/ dev/asm-b_crs' is not in the discovery set-- the device cannot be found here should also be the same as the following situation, no scan path specified

SQL > col PATH for A50

SQL > select group_number, disk_number, mount_status, header_status, path from v$asm_disk

No rows selected

It indicates that the disk is not identified, and why there is no disk here is now understood, because there is no setting in the parameters.

SQL > show parameter asm

NAME TYPE VALUE

-

Asm_diskgroups stringDATA-the default parameter file is empty at startup, and normally does not display the file that saves the name of the OCR disk group

Asm_diskstring string/dev/asm*-the default parameter file is empty at startup, and the scanning path is not specified

Asm_power_limit integer1

Asm_preferred_read_failure_groups string

So to be on the safe side, you should close ASM immediately after crsctl start crs-excl-nocrs starts. Do not create a create diskgroup CRSVOTEDISK disk group immediately, and then start ASM with parameters.

SQL > startup pfile='/tmp/asm_pfile_130717.txt'

[grid@rac1 ~] $cat / tmp/asm_pfile_130717.txt

+ ASM1.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value

+ ASM1.asm_diskgroups='DATA'#Manual Mount

+ ASM2.asm_diskgroups='DATA'#Manual Mount

* .asm_diskstring='/dev/asm*'

* .asm_power_limit=1

* .diagnostic_dest='/u01/app/grid'

* .instance_type='asm'

* .large_pool_size=12M

* .remote_login_passwordfile='EXCLUSIVE'

Here again check the v$asm_disk to query the disk, you can also successfully create a disk group. Because the ASM was not closed immediately and the modified parameter file was used, the disk group was not found when the disk group was created, which took a long time.

Let's start the resume operation again:

Restart after shutting down crs

[root@rac1 dev] # crsctl stop crs

[root@rac1 dev] # crsctl start crs-excl-nocrs

[root@rac1 dev] # crsctl query css votedisk

Located 0 voting disk (s).

Close ASM on rac1, start ASM with parameter file, create CRS disk group, create spfile

[grid@rac1 ~] $sqlplus / as sysasm

SQL > shutdown immediateASM diskgroups dismountedASM instance shutdownSQL > startup pfile='/tmp/asm_pfile_130717.txt';ASM instance started

SQL > col path for A50

SQL > set linesize 130

SQL > select group_number, disk_number, mount_status, header_status, path from v$asm_disk

GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU PATH

0 0 CLOSED MEMBER / dev/asm-e_data

0 3 CLOSED CANDIDATE / dev/asm-b_crs

0 2 CLOSED CANDIDATE / dev/asm-c_crs

0 1 CLOSED CANDIDATE / dev/asm-d_crs

SQL > create diskgroup CRSVOTEDISK normal redundancy disk'/ dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'

2 attribute 'compatible.asm'='11.2.0.0.0',' compatible.rdbms'='11.2.0.0.0'

Diskgroup created.

SQL > create spfile='+CRSVOTEDISK 'from pfile='/tmp/asm_pfile_130717.txt'

File created.

SQL > quit

Restore crs

[root@rac1 dev] # ocrconfig-restore / u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr

Restore voting disk

[root@rac1 dev] # crsctl replace votedisk + CRSVOTEDISK

Successful addition of voting disk 1b00b0ec4e504f7fbf1f8d20fbbfaa4b.

Successful addition of voting disk 5a3b646433124fdcbf23c3c290de7fe3.

Successful addition of voting disk 5d27d80b96d74f09bf1756be6dee387f.

Successfully replaced voting disk group with + CRSVOTEDISK.

CRS-4266: Voting file (s) successfully replaced

Detection

[root@rac1 ~] # ocrcheck

Status of Oracle Cluster Registry is as follows:

Version: 3

Total space (kbytes): 262120

Used space (kbytes): 3016

Available space (kbytes): 259104

ID: 1236405787

Device/File Name: + CRSVOTEDISK

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@rac1 ~] # crsctl query css votedisk

# # STATE File Universal Id File Name Disk group

1. ONLINE 1b00b0ec4e504f7fbf1f8d20fbbfaa4b (/ dev/asm-b_crs) [CRSVOTEDISK]

2. ONLINE 5a3b646433124fdcbf23c3c290de7fe3 (/ dev/asm-c_crs) [CRSVOTEDISK]

3. ONLINE 5d27d80b96d74f09bf1756be6dee387f (/ dev/asm-d_crs) [CRSVOTEDISK]

Located 3 voting disk (s).

Stop crs to start normally:

[root@rac1 ~] # crsctl stop crs

[root@rac1 ~] # crsctl start crs

At this point, the recovery of crs and voting disk has been completed, but be careful to modify the ocrconfig_loc=+CRSVOTEDISK in / etc/oracle/ocr.loc on the rac2, otherwise you will start the error message:

[/ u01/app/11.2.0.3/grid/bin/oraagent.bin (19510)] CRS-5019:All OCR locations are on ASM disk groups [CRSDATA], and none of these disk groups are mounted. Details are at "(: CLSN00100:)" in "/ u01/app/11.2.0.3/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

2013-07-18 00 1033.678

[/ u01/app/11.2.0.3/grid/bin/oraagent.bin (19510)] CRS-5019:All OCR locations are on ASM disk groups [CRSDATA], and none of these disk groups are mounted. Details are at "(: CLSN00100:)" in "/ u01/app/11.2.0.3/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".

2013-07-18 00 11 03.614

[root@rac2 ~] # crsctl start crs

[root@rac1 ~] # crs_stat-t

Name Type Target State Host

Ora.CRSDATA.dg ora....up.type ONLINE OFFLINE

Ora.DATA.dg ora....up.type ONLINE ONLINE rac1

Ora....ER.lsnr ora....er.type ONLINE ONLINE rac1

Ora....N1.lsnr ora....er.type ONLINE ONLINE rac1

Ora.asm ora.asm.type ONLINE ONLINE rac1

Ora.chris.db ora....se.type ONLINE ONLINE rac1

Ora.cvu ora.cvu.type ONLINE ONLINE rac1

Ora.gsd ora.gsd.type OFFLINE OFFLINE

Ora....network ora....rk.type ONLINE ONLINE rac1

Ora.oc4j ora.oc4j.type ONLINE ONLINE rac1

Ora.ons ora.ons.type ONLINE ONLINE rac1

Ora....SM1.asm application ONLINE ONLINE rac1

Ora....C1.lsnr application ONLINE ONLINE rac1

Ora.rac1.gsd application OFFLINE OFFLINE

Ora.rac1.ons application ONLINE ONLINE rac1

Ora.rac1.vip ora....t1.type ONLINE ONLINE rac1

Ora....SM2.asm application ONLINE ONLINE rac2

Ora....C2.lsnr application ONLINE ONLINE rac2

Ora.rac2.gsd application OFFLINE OFFLINE

Ora.rac2.ons application ONLINE ONLINE rac2

Ora.rac2.vip ora....t1.type ONLINE ONLINE rac2

Ora....ry.acfs ora....fs.type ONLINE ONLINE rac1

Ora.scan1.vip ora....ip.type ONLINE ONLINE rac1

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.