Shulou(Shulou.com)06/01 Report--

Scenario description:

Do OCR backup and restore experiment, OCR has 4 automatic backups. After replacing the OCR disk from + DATA with + OCR2 (/ dev/raw/raw4), manually back up the OCR using ocrconfig-manualbackup after completion, and then dd / dev/raw/raw4. Shut down the cluster, start the cluster, and find that the cluster cannot be started.

Problem analysis (assuming you don't know what the problem is, analyze it first):

1. Check the cluster service and find that the CRS and CSS services failed to start normally.

Crsctl check crs

2. Check CRS and CSS logs and find that OCR disk is abnormal.

3. Restore OCR (in fact, it is the process of using root.sh to rebuild OCR. After reconstruction, you need to re-register related resources such as listener/database, etc.)

Clear cluster configuration information for all nodes: root user executes $GRID_HOME/crs/install/rootcrs.pl

Node 1

[root@node1 install] #. / rootcrs.pl

Using configuration parameter file:. / crsconfig_params

User ignored Prerequisites during installation

Installing Trace File Analyzer

Configure Oracle Grid Infrastructure for a Cluster... Succeeded

Node 2

[root@node2 install] #. / rootcrs.pl

Using configuration parameter file:. / crsconfig_params

User ignored Prerequisites during installation

Installing Trace File Analyzer

Configure Oracle Grid Infrastructure for a Cluster... Succeeded

Clear cluster information for all nodes

Node 1

[root@node1 install] #. / rootcrs.pl-deconfig-force

Using configuration parameter file:. / crsconfig_params

PRCR-1119: Failed to look up CRS resources of ora.cluster_vip_net1.type type

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1070: Failed to check if resource ora.gsd is registered

Cannot communicate with crsd

PRCR-1070: Failed to check if resource ora.ons is registered

Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'

CRS-2673: Attempting to stop 'ora.mdnsd' on' node1'

CRS-2673: Attempting to stop 'ora.crf' on' node1'

CRS-2673: Attempting to stop 'ora.ctssd' on' node1'

CRS-2673: Attempting to stop 'ora.evmd' on' node1'

CRS-2673: Attempting to stop 'ora.asm' on' node1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on' node1'

CRS-2677: Stop of 'ora.evmd' on' node1' succeeded

CRS-2677: Stop of 'ora.crf' on' node1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on' node1' succeeded

CRS-2677: Stop of 'ora.ctssd' on' node1' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on' node1' succeeded

CRS-2677: Stop of 'ora.asm' on' node1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on' node1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' node1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' node1'

CRS-2677: Stop of 'ora.cssd' on' node1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on' node1'

CRS-2677: Stop of 'ora.gipcd' on' node1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on' node1'

CRS-2677: Stop of 'ora.gpnpd' on' node1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

Removing Trace File Analyzer

Successfully deconfigured Oracle clusterware stack on this node

Node 2

[root@node2 install] #. / rootcrs.pl-deconfig-force-lastnode

Using configuration parameter file:. / crsconfig_params

CRS-5702: Resource 'ora.cssd' is already running on' node2'

CRS-4000: Command Start failed, or completed with errors.

CSS startup failed with return code 1

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1119: Failed to look up CRS resources of ora.cluster_vip_net1.type type

PRCR-1068: Failed to query resources

Cannot communicate with crsd

PRCR-1070: Failed to check if resource ora.gsd is registered

Cannot communicate with crsd

PRCR-1070: Failed to check if resource ora.ons is registered

Cannot communicate with crsd

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Delete failed, or completed with errors.

CRS-2673: Attempting to stop 'ora.ctssd' on' node2'

CRS-2673: Attempting to stop 'ora.evmd' on' node2'

CRS-2673: Attempting to stop 'ora.asm' on' node2'

CRS-2677: Stop of 'ora.evmd' on' node2' succeeded

CRS-2677: Stop of 'ora.ctssd' on' node2' succeeded

CRS-2677: Stop of 'ora.asm' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on' node2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' node2'

CRS-2677: Stop of 'ora.cssd' on' node2' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on' node2'

CRS-2676: Start of 'ora.cssdmonitor' on' node2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on' node2'

CRS-2672: Attempting to start 'ora.diskmon' on' node2'

CRS-2676: Start of 'ora.diskmon' on' node2' succeeded

CRS-2676: Start of 'ora.cssd' on' node2' succeeded

CRS-4611: Successful deletion of voting disk + DATA.

ASM de-configuration trace file location: / tmp/asmcadc_clean2016-10-31, 02-02-22-PM.log

ASM Clean Configuration START

ASM Clean Configuration END

ASM with SID + ASM1 deleted successfully. Check / tmp/asmcadc_clean2016-10-31, 02-02-22-PM.log for details.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'

CRS-2673: Attempting to stop 'ora.ctssd' on' node2'

CRS-2673: Attempting to stop 'ora.asm' on' node2'

CRS-2673: Attempting to stop 'ora.mdnsd' on' node2'

CRS-2677: Stop of 'ora.mdnsd' on' node2' succeeded

CRS-2677: Stop of 'ora.ctssd' on' node2' succeeded

CRS-2677: Stop of 'ora.asm' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on' node2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' node2'

CRS-2677: Stop of 'ora.cssd' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.crf' on' node2'

CRS-2677: Stop of 'ora.crf' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on' node2'

CRS-2677: Stop of 'ora.gipcd' on' node2' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on' node2'

CRS-2677: Stop of 'ora.gpnpd' on' node2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

Removing Trace File Analyzer

Successfully deconfigured Oracle clusterware stack on this node

Rebuilding OCR and OLR, using the root.sh script to complete the reconstruction, is actually installing the script executed in RAC. The default location is $GRID_HOME.

Node 1

[root@node1 grid] #. / root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= / u01/11.2.0/grid

Enter the full pathname of the local bin directory: [/ usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the / etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: / u01/11.2.0/grid/crs/install/crsconfig_params

User ignored Prerequisites during installation

Installing Trace File Analyzer

OLR initialization-successful

Adding Clusterware entries to upstart

CRS-2672: Attempting to start 'ora.mdnsd' on' node1'

CRS-2676: Start of 'ora.mdnsd' on' node1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on' node1'

CRS-2676: Start of 'ora.gpnpd' on' node1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on' node1'

CRS-2672: Attempting to start 'ora.gipcd' on' node1'

CRS-2676: Start of 'ora.cssdmonitor' on' node1' succeeded

CRS-2676: Start of 'ora.gipcd' on' node1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on' node1'

CRS-2672: Attempting to start 'ora.diskmon' on' node1'

CRS-2676: Start of 'ora.diskmon' on' node1' succeeded

CRS-2676: Start of 'ora.cssd' on' node1' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

Clscfg:-install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp' root'..

Operation successful.

Successful addition of voting disk 4331dad495c14f71bfdb6d4f1a82d2f9.

Successfully replaced voting disk group with + DATA.

CRS-4266: Voting file (s) successfully replaced

STATE File Universal Id File Name Disk groupONLINE 4331dad495c14f71bfdb6d4f1a82d2f9 (/ dev/raw/raw1) [DATA]

Located 1 voting disk (s).

CRS-2672: Attempting to start 'ora.asm' on' node1'

CRS-2676: Start of 'ora.asm' on' node1' succeeded

CRS-2672: Attempting to start 'ora.DATA.dg' on' node1'

CRS-2676: Start of 'ora.DATA.dg' on' node1' succeeded

Preparing packages for installation...

Cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster... Succeeded

Node 2

[root@node2 grid] #. / root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= / u01/11.2.0/grid

Enter the full pathname of the local bin directory: [/ usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the / etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: / u01/11.2.0/grid/crs/install/crsconfig_params

User ignored Prerequisites during installation

Installing Trace File Analyzer

OLR initialization-successful

Adding Clusterware entries to upstart

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Preparing packages for installation...

Cvuqdisk-1.0.9-1

Configure Oracle Grid Infrastructure for a Cluster... Succeeded

Check resource information

Node 1

[root@node1 grid] # crs_stat-t

Name Type Target State Hostora.DATA.dg ora....up.type ONLINE ONLINE node1

Ora....N1.lsnr ora....er.type ONLINE ONLINE node1

Ora.asm ora.asm.type ONLINE ONLINE node1

Ora.cvu ora.cvu.type ONLINE ONLINE node1

Ora.gsd ora.gsd.type OFFLINE OFFLINE

Ora....network ora....rk.type ONLINE ONLINE node1

Ora....SM1.asm application ONLINE ONLINE node1

Ora.node1.gsd application OFFLINE OFFLINE

Ora.node1.ons application ONLINE ONLINE node1

Ora.node1.vip ora....t1.type ONLINE ONLINE node1

Ora....SM2.asm application ONLINE ONLINE node2

Ora.node2.gsd application OFFLINE OFFLINE

Ora.node2.ons application ONLINE ONLINE node2

Ora.node2.vip ora....t1.type ONLINE ONLINE node2

Ora.oc4j ora.oc4j.type ONLINE ONLINE node1

Ora.ons ora.ons.type ONLINE ONLINE node1

Ora....ry.acfs ora....fs.type ONLINE ONLINE node1

Ora.scan1.vip ora....ip.type ONLINE ONLINE node1

[root@node1 grid] # crsctl stat res-tNAME TARGET STATE SERVER STATE_DETAILSLocal Resourcesora.DATA.dg

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.asm

ONLINE ONLINE node1 Started

ONLINE ONLINE node2 Started

Ora.gsd

OFFLINE OFFLINE node1

OFFLINE OFFLINE node2

Ora.net1.network

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.ons

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.registry.acfs

ONLINE ONLINE node1

ONLINE ONLINE node2Cluster Resources

Ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE node1

Ora.cvu

1 ONLINE ONLINE node1

Ora.node1.vip

1 ONLINE ONLINE node1

Ora.node2.vip

1 ONLINE ONLINE node2

Ora.oc4j

1 ONLINE ONLINE node1

Ora.scan1.vip

1 ONLINE ONLINE node1

Node 2

[root@node2 grid] # crs_stat-t

Name Type Target State Hostora.DATA.dg ora....up.type ONLINE ONLINE node1

Ora....N1.lsnr ora....er.type ONLINE ONLINE node1

Ora.asm ora.asm.type ONLINE ONLINE node1

Ora.cvu ora.cvu.type ONLINE ONLINE node1

Ora.gsd ora.gsd.type OFFLINE OFFLINE

Ora....network ora....rk.type ONLINE ONLINE node1

Ora....SM1.asm application ONLINE ONLINE node1

Ora.node1.gsd application OFFLINE OFFLINE

Ora.node1.ons application ONLINE ONLINE node1

Ora.node1.vip ora....t1.type ONLINE ONLINE node1

Ora....SM2.asm application ONLINE ONLINE node2

Ora.node2.gsd application OFFLINE OFFLINE

Ora.node2.ons application ONLINE ONLINE node2

Ora.node2.vip ora....t1.type ONLINE ONLINE node2

Ora.oc4j ora.oc4j.type ONLINE ONLINE node1

Ora.ons ora.ons.type ONLINE ONLINE node1

Ora....ry.acfs ora....fs.type ONLINE ONLINE node1

Ora.scan1.vip ora....ip.type ONLINE ONLINE node1

[root@node2 grid] # crsctl stat res-tNAME TARGET STATE SERVER STATE_DETAILSLocal Resourcesora.DATA.dg

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.asm

ONLINE ONLINE node1 Started

ONLINE ONLINE node2 Started

Ora.gsd

OFFLINE OFFLINE node1

OFFLINE OFFLINE node2

Ora.net1.network

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.ons

ONLINE ONLINE node1

ONLINE ONLINE node2

Ora.registry.acfs

ONLINE ONLINE node1

ONLINE ONLINE node2Cluster Resources

Ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE node1

Ora.cvu

1 ONLINE ONLINE node1

Ora.node1.vip

1 ONLINE ONLINE node1

Ora.node2.vip

1 ONLINE ONLINE node2

Ora.oc4j

1 ONLINE ONLINE node1

Ora.scan1.vip

1 ONLINE ONLINE node1

View the disk group information, and mount it manually if it is not mounted:

SQL > select name,state from v$asm_diskgroup

4. Add resources (monitoring, database, instance, etc.)

Add snooping

[grid@node1 ~] $srvctl add listener-l listener

View monitoring

[grid@node1 ~] $srvctl config listener

Add db and instance

[oracle@node1] $srvctl add database-h

[oracle@node1] $srvctl add database-d orcl-o / u01/app/oracle/product/11.2.0/db_1-c RAC

[oracle@node1] $srvctl add instance-h

[oracle@node1] $srvctl add instance-d orcl-I orcl1-n node1

[oracle@node1] $srvctl add instance-d orcl-I orcl2-n node2

[oracle@node1] $srvctl config database-d orcl

5. Restart the cluster after adding resources

[root@node1 grid] # crsctl stop cluster-all

[root@node1 grid] # crsctl start cluster-all

After the addition is complete, there may be a problem that the database cannot be started automatically. Try to execute the following statement:

[oracle@node1] $srvctl enable database-d orcl

[oracle@node1] $srvctl enable instance-d orcl-I orcl1

[oracle@node1] $srvctl enable instance-d orcl-I orcl2

[oracle@node1] $srvctl start database-d orcl

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.