Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to fix RAC node failure in kingsql". The explanation in this article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to fix RAC node failure in kingsql".

The rac2 and rac3 cluster services cannot be started after the virtual machine installed a long time ago is booted today

[grid@rac2 cssd] $crs_stat-t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac3 cssd] $crs_stat-t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@rac1] $srvctl status database-d kingsql

Instance kingsql1 is running on node rac1

Instance kingsql2 is not running on node rac2

Instance kingsql3 is not running on node rac3

A problem was found in rac2's cssd log

2016-04-19 1700 node 47.205: [CSSD] [3666859776] clssnmvDHBValidateNCopy: node 1, rac1, has a disk HB, but no network HB, DHB has rcfg 356447081, wrtcnt, 294277, LATS 4294697710, lastSeqNo 294274, uniqueness 1461034467, timestamp 1461056437Ember 21174804

2016-04-19 1700 node 47.205: [CSSD] [3666859776] clssnmvDHBValidateNCopy: node 3, rac3, has a disk HB, but no network HB, DHB has rcfg 356447080, wrtcnt, 41563, LATS 4294697710, lastSeqNo 41560, uniqueness 1463643843, timestamp 1463643914 18290394

[grid@rac1 ~] $crs_stop-all

CRS-2500: Cannot stop resource 'ora.gsd' as it is not running

Attempting to stop `ora.oc4j` on member `rac1`

Attempting to stop `ora.CRS.dg` on member `rac1`

Attempting to stop `ora.DATA.dg` on member `rac1`

Attempting to stop `ora.kingsql.db` on member `rac1`

Attempting to stop `ora.ons` on member `rac1`

CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server' rac1'

Stop of `ora.ons` on member `rac1` succeeded.

Attempting to stop `ora.cvu` on member `rac1`

Attempting to stop `ora.rac2.vip` on member `rac1`

Attempting to stop `ora.rac3.vip` on member `rac1`

Attempting to stop `ora.LISTENER_ SCAN1.lsnr` on member `rac1`

Stop of `ora.rac3.vip` on member `rac1` succeeded.

Stop of `ora.rac2.vip` on member `rac1` succeeded.

Stop of `ora.LISTENER_ SCAN1.lsnr` on member `rac1` succeeded.

Attempting to stop `ora.scan1.vip` on member `rac1`

Stop of `ora.scan1.vip` on member `rac1` succeeded.

Stop of `ora.oc4j` on member `rac1` succeeded.

Stop of `ora.cvu` on member `rac1` succeeded.

Attempting to stop `ora.net1.network` on member `rac1`

Stop of `ora.net1.network` on member `rac1` succeeded.

Stop of `ora.kingsql.db` on member `rac1` succeeded.

Stop of `ora.DATA.dg` on member `rac1` succeeded.

Stop of `ora.CRS.dg` on member `rac1` succeeded.

Attempting to stop `ora.asm` on member `rac1`

Stop of `ora.asm` on member `rac1` succeeded.

Attempting to stop `ora.asm` on member `rac1`

-- stuck here, so execute crsctl stop crs

CRS-0184: Cannot communicate with the CRS daemon.

[root@rac1 ~] # / u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'

CRS-2673: Attempting to stop 'ora.mdnsd' on' rac1'

CRS-2673: Attempting to stop 'ora.ctssd' on' rac1'

CRS-2673: Attempting to stop 'ora.evmd' on' rac1'

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on' rac1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' rac1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on' rac1' succeeded

CRS-2677: Stop of 'ora.evmd' on' rac1' succeeded

CRS-2677: Stop of 'ora.ctssd' on' rac1' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' rac1'

CRS-2677: Stop of 'ora.cssd' on' rac1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on' rac1'

CRS-2677: Stop of 'ora.crf' on' rac1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on' rac1'

CRS-2677: Stop of 'ora.gipcd' on' rac1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on' rac1'

CRS-2677: Stop of 'ora.gpnpd' on' rac1' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed

CRS-4133: Oracle High Availability Services has been stopped.

[root@rac2 ~] # / u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2796: The command may not proceed when Cluster Ready Services is not running

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

[root@rac3 ~] # / u01/app/11.2.3/grid/bin/crsctl stop crs

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac3'

CRS-2673: Attempting to stop 'ora.crsd' on' rac3'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac3'

CRS-2673: Attempting to stop 'ora.CRS.dg' on' rac3'

CRS-2673: Attempting to stop 'ora.DATA.dg' on' rac3'

CRS-2677: Stop of 'ora.DATA.dg' on' rac3' succeeded

CRS-2677: Stop of 'ora.CRS.dg' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.asm' on' rac3'

CRS-2677: Stop of 'ora.asm' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.ons' on' rac3'

CRS-2677: Stop of 'ora.ons' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on' rac3'

CRS-2677: Stop of 'ora.net1.network' on' rac3' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac3' has completed

CRS-2677: Stop of 'ora.crsd' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.ctssd' on' rac3'

CRS-2673: Attempting to stop 'ora.evmd' on' rac3'

CRS-2673: Attempting to stop 'ora.asm' on' rac3'

CRS-2673: Attempting to stop 'ora.mdnsd' on' rac3'

CRS-2677: Stop of 'ora.evmd' on' rac3' succeeded

CRS-2677: Stop of 'ora.mdnsd' on' rac3' succeeded

CRS-2677: Stop of 'ora.ctssd' on' rac3' succeeded

CRS-2677: Stop of 'ora.asm' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on' rac3'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' rac3'

CRS-2677: Stop of 'ora.cssd' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.crf' on' rac3'

CRS-2677: Stop of 'ora.crf' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on' rac3'

CRS-2677: Stop of 'ora.gipcd' on' rac3' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on' rac3'

CRS-2677: Stop of 'ora.gpnpd' on' rac3' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac3' has completed

CRS-4133: Oracle High Availability Services has been stopped.

Start

[root@rac1 ~] # / u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[root@rac2 ~] # / u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

-- there is still a problem

[root@rac3 ~] # / u01/app/11.2.3/grid/bin/crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

[grid@rac1 ~] $crs_start-all

[grid@rac1] $crs_stat-t

Name Type Target State Host

Ora.CRS.dg ora....up.type ONLINE ONLINE rac1

Ora.DATA.dg ora....up.type ONLINE ONLINE rac1

Ora....ER.lsnr ora....er.type ONLINE ONLINE rac1

Ora....N1.lsnr ora....er.type ONLINE ONLINE rac3

Ora.asm ora.asm.type ONLINE ONLINE rac1

Ora.cvu ora.cvu.type ONLINE ONLINE rac3

Ora.gsd ora.gsd.type OFFLINE OFFLINE

Ora.kingsql.db ora....se.type ONLINE ONLINE rac3

Ora....network ora....rk.type ONLINE ONLINE rac1

Ora.oc4j ora.oc4j.type ONLINE ONLINE rac3

Ora.ons ora.ons.type ONLINE ONLINE rac1

Ora....SM1.asm application ONLINE ONLINE rac1

Ora....C1.lsnr application ONLINE ONLINE rac1

Ora.rac1.gsd application OFFLINE OFFLINE

Ora.rac1.ons application ONLINE ONLINE rac1

Ora.rac1.vip ora....t1.type ONLINE ONLINE rac1

Ora.rac2.vip ora....t1.type ONLINE ONLINE rac1

Ora....SM3.asm application ONLINE ONLINE rac3

Ora....C3.lsnr application ONLINE ONLINE rac3

Ora.rac3.gsd application OFFLINE OFFLINE

Ora.rac3.ons application ONLINE ONLINE rac3

Ora.rac3.vip ora....t1.type ONLINE ONLINE rac3

Ora.scan1.vip ora....ip.type ONLINE ONLINE rac3

Restart the rac2 operating system

[root@rac2 ~] # reboot

[root@rac2 ~] #

Broadcast message from root@rac2

(/ dev/pts/0) at 17:40.

The system is going down for reboot NOW!

[grid@rac1] $crs_stat-t

Name Type Target State Host

Ora.CRS.dg ora....up.type ONLINE ONLINE rac1

Ora.DATA.dg ora....up.type ONLINE ONLINE rac1

Ora....ER.lsnr ora....er.type ONLINE ONLINE rac1

Ora....N1.lsnr ora....er.type ONLINE ONLINE rac3

Ora.asm ora.asm.type ONLINE ONLINE rac1

Ora.cvu ora.cvu.type ONLINE ONLINE rac3

Ora.gsd ora.gsd.type OFFLINE OFFLINE

Ora.kingsql.db ora....se.type ONLINE ONLINE rac3

Ora....network ora....rk.type ONLINE ONLINE rac1

Ora.oc4j ora.oc4j.type ONLINE ONLINE rac3

Ora.ons ora.ons.type ONLINE ONLINE rac1

Ora....SM1.asm application ONLINE ONLINE rac1

Ora....C1.lsnr application ONLINE ONLINE rac1

Ora.rac1.gsd application OFFLINE OFFLINE

Ora.rac1.ons application ONLINE ONLINE rac1

Ora.rac1.vip ora....t1.type ONLINE ONLINE rac1

Ora....SM2.asm application ONLINE ONLINE rac2

Ora....C2.lsnr application ONLINE ONLINE rac2

Ora.rac2.gsd application OFFLINE OFFLINE

Ora.rac2.ons application ONLINE ONLINE rac2

Ora.rac2.vip ora....t1.type ONLINE ONLINE rac2

Ora....SM3.asm application ONLINE ONLINE rac3

Ora....C3.lsnr application ONLINE ONLINE rac3

Ora.rac3.gsd application OFFLINE OFFLINE

Ora.rac3.ons application ONLINE ONLINE rac3

Ora.rac3.vip ora....t1.type ONLINE ONLINE rac3

Ora.scan1.vip ora....ip.type ONLINE ONLINE rac3

[grid@rac1] $crsctl status res-t

NAME TARGET STATE SERVER STATE_DETAILS

Local Resources

Ora.CRS.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ONLINE ONLINE rac3

Ora.DATA.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ONLINE ONLINE rac3

Ora.LISTENER.lsnr

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ONLINE ONLINE rac3

Ora.asm

ONLINE ONLINE rac1 Started

ONLINE ONLINE rac2 Started

ONLINE ONLINE rac3 Started

Ora.gsd

OFFLINE OFFLINE rac1

OFFLINE OFFLINE rac2

OFFLINE OFFLINE rac3

Ora.net1.network

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ONLINE ONLINE rac3

Ora.ons

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ONLINE ONLINE rac3

Cluster Resources

Ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE rac3

Ora.cvu

1 ONLINE ONLINE rac3

Ora.kingsql.db

1 ONLINE ONLINE rac3 Open

2 ONLINE ONLINE rac2 Open

3 ONLINE ONLINE rac1 Open

Ora.oc4j

1 ONLINE ONLINE rac3

Ora.rac1.vip

1 ONLINE ONLINE rac1

Ora.rac2.vip

1 ONLINE ONLINE rac2

Ora.rac3.vip

1 ONLINE ONLINE rac3

Ora.scan1.vip

1 ONLINE ONLINE rac3

Thank you for reading, the above is the content of "how to repair RAC node failure in kingsql". After the study of this article, I believe you have a deeper understanding of how to repair RAC node fault in kingsql, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.