Shulou(Shulou.com)06/01 Report--

Suse11SP3 system platform Crsctl executes shutdown Times error handling

Crsctl executive shutdown Times error.

Jason1:/u01/app/11.2.0/grid/bin #. / crsctl stop crs

CRS-2791: Starting shutdown of Oracle High AvailabilityServices-managed resources on 'jason1'

CRS-2673: Attempting to stop 'ora.crsd' on' jason1'

CRS-2790: Starting shutdown of Cluster Ready Services-managedresources on 'jason1'

CRS-2673: Attempting to stop 'ora.OCR.dg' on' jason1'

CRS-2673: Attempting to stop 'ora.registry.acfs' on' jason1'

CRS-2673: Attempting to stop 'ora.oracle.db' on' jason1'

CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on'jason1'

CRS-2673: Attempting to stop 'ora.cvu' on' jason1'

CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on' jason1'

CRS-2673: Attempting to stop 'ora.oc4j' on' jason1'

CRS-2677: Stop of 'ora.LISTENER.lsnr' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.jason1.vip' on' jason1'

CRS-2677: Stop of 'ora.cvu' on' jason1' succeeded

CRS-2672: Attempting to start 'ora.cvu' on' jason2'

CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on' jason1'succeeded

CRS-2673: Attempting to stop 'ora.scan1.vip' on' jason1'

CRS-2677: Stop of 'ora.jason1.vip' on' jason1' succeeded

CRS-2672: Attempting to start 'ora.jason1.vip' on' jason2'

CRS-2676: Start of 'ora.cvu' on' jason2' succeeded

CRS-2677: Stop of 'ora.scan1.vip' on' jason1' succeeded

CRS-2672: Attempting to start 'ora.scan1.vip' on' jason2'

CRS-2677: Stop of 'ora.oracle.db' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.DATA.dg' on' jason1'

CRS-2673: Attempting to stop 'ora.FRA.dg' on' jason1'

CRS-2677: Stop of 'ora.registry.acfs' on' jason1' succeeded

CRS-2676: Start of 'ora.jason1.vip' on' jason2' succeeded

CRS-2676: Start of 'ora.scan1.vip' on' jason2' succeeded

CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on'jason2'

CRS-2677: Stop of 'ora.OCR.dg' on' jason1' succeeded

CRS-2677: Stop of 'ora.FRA.dg' on' jason1' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.asm' on' jason1'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on' jason2'succeeded

CRS-2677: Stop of 'ora.asm' on' jason1' succeeded

CRS-2677: Stop of 'ora.oc4j' on' jason1' succeeded

CRS-2672: Attempting to start 'ora.oc4j' on' jason2'

CRS-2676: Start of 'ora.oc4j' on' jason2' succeeded

CRS-2673: Attempting to stop 'ora.ons' on' jason1'

CRS-2677: Stop of 'ora.ons' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.net1.network' on' jason1'

CRS-2677: Stop of 'ora.net1.network' on' jason1' succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resourceson 'jason1' has completed

CRS-2677: Stop of 'ora.crsd' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.crf' on' jason1'

CRS-2673: Attempting to stop 'ora.ctssd' on' jason1'

CRS-2673: Attempting to stop 'ora.evmd' on' jason1'

CRS-2673: Attempting to stop 'ora.asm' on' jason1'

CRS-2673: Attempting to stop 'ora.mdnsd' on' jason1'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on' jason1'

CRS-2677: Stop of 'ora.crf' on' jason1' succeeded

CRS-2677: Stop of 'ora.evmd' on' jason1' succeeded

CRS-2677: Stop of 'ora.mdnsd' on' jason1' succeeded

CRS-2677: Stop of 'ora.ctssd' on' jason1' succeeded

CRS-5022: Stop of resource "ora.drivers.acfs" failed:current state is "UNKNOWN"

CRS-2675: Stop of 'ora.drivers.acfs' on' jason1' failed

CRS-2677: Stop of 'ora.asm' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on'jason1'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on' jason1'succeeded

CRS-2673: Attempting to stop 'ora.cssd' on' jason1'

CRS-2677: Stop of 'ora.cssd' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on' jason1'

CRS-2677: Stop of 'ora.gipcd' on' jason1' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on' jason1'

CRS-2677: Stop of 'ora.gpnpd' on' jason1' succeeded

CRS-2799: Failed toshut down resource 'ora.drivers.acfs' on' jason1'

CRS-2795: Shutdownof Oracle High Availability Services-managed resources on 'jason1' has failed

CRS-4687: Shutdowncommand has completed with errors.

CRS-4000: CommandStop failed, or completed with errors.

Jason1:/u01/app/11.2.0/grid/bin #

Check the crs log to find out if any of the following logs exist each time you shut down

CRS-10001:07-Jan-16 23:23 ACFS-9118: oracleacfs.ko driver in use-cannot unload

Prove that every time the resource ora.drivers.acfs is closed, another program is using the module to cause the shutdown to fail. View the content of MOS as follows:

SLES: 11.2.0.3 "crsctl stop crs" Fails to Stop ora.drivers.acfs With CRS-2675 (document ID 1417294.1)

In this Document

APPLIES TO:

Oracle Database-Enterprise Edition-Version 11.2.0.3 and later

SUSE\ UnitedLinux x86-64

SYMPTOMS

11.2.0.3 Grid Infrastructure on Suse Linux, "crsctl stop crs" or "crsctl stop crs-f" fails:

"crsctl stop crs" or "crsctl stop crs-f" output

CRS-2675: Stop of 'ora.drivers.acfs' on' racnode1' failed

..

CRS-2799: Failed to shut down resource 'ora.drivers.acfs' on' racnode1'

CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'racnode1' has failed

CRS-4687: Shutdown command has completed with errors.

CRS-4000: Command Stop failed, or completed with errors.

$GRID_HOME/log//alert.log

[client (4936)] CRS-10001:01-Feb-12 12:35 ACFS-9290: Waiting for ASM to shutdown.

[client (5098)] CRS-10001:01-Feb-12 12:35 ACFS-9118: oracleacfs.ko driver in use-cannot unload.

[client (5117)] CRS-10001:01-Feb-12 12:35 ACFS-9118: oracleacfs.ko driver in use-cannot unload.

/ log/racnode1/agent/ohasd/orarootagent_root/orarootagent_root.log "

"$GRID_HOME/bin/acfsload stop" output

..

ACFS-9119: oracleacfs driver failed to unload.

$GRID_HOME/log//agent/ohasd/orarootagent_root/orarootagent_root.log

2012-02-1509RESOURCE_ 43.219: [AGFW] [2097063696] {0VOV36062} Agent received the message: RESOURCE_ stop [ora.drivers.acfs 1 1] ID 4099

2012-02-1509 ora.drivers.acfs 44 ora.drivers.acfs 43.219: [AGFW] [2097063696] {0V 0V 36062} ora.drivers.acfs 1 1

2012-02-1509 state changed from 44ONLINE to 43.219: [AGFW] [2097063696] {0state changed from 036062} state changed from: ONLINE to: STOPPING

2012-02-1509 CLSN00108 44 clsn_agent::stop 43.220: [ora.drivers.acfs] [2094962448] {0CLSN00108 036062} [stop] (: CLSN00108:)

2012-02-1509 Arg Value 4415. 220: [ora.drivers.acfs] [2094962448] {0Arg Value 06062} [stop] Arg Value = stop

2012-02-1509 getOracleHomeAttrib 4415. 220: [ora.drivers.acfs] [2094962448] {0 getOracleHomeAttrib: oracle_home = / ocw/grid

2012-02-1509 Utils::getCrsHome crsHome 43.220: [ora.drivers.acfs] [2094962448] {0Utils::getCrsHome crsHome / ocw/grid} [stop]

..

2012-02-1509 Adding Environment Variables 43.220: [ora.drivers.acfs] [2094962448] {0Adding Environment Variables 0662} [stop] 43.220

2012-02-1509 Adding Environment Variables 44 IS_HASD_AGENT= 43.220: [ora.drivers.acfs] [2094962448] {0Adding Environment Variables 0662} [stop] 36062} [Adding Environment Variables _ _ IS_HASD_AGENT=

2012-02-1509Utils:execCmd action 44flags 43.220: [ora.drivers.acfs] [2094962448] {0flags 036062} [stop] flags = 6 ohome = (null) cmdname = acfsload.

2012-02-1509 getOracleHomeAttrib 4415. 220: [ora.drivers.acfs] [2094962448] {0 getOracleHomeAttrib: oracle_home = / ocw/grid

2012-02-1509 ACFS-9290 44ACFS-9290 43.321: [ora.drivers.acfs] [2094962448] {0VOVERV 36062} [stop] (: CLSN00010:) ACFS-9290: Waiting for ASM to shutdown.

2012-02-1509 CLSN00010 43.321: [ora.drivers.acfs] [2094962448] {0VOVOR 36062} [stop] (: CLSN00010:)

..

2012-02-1509ACFS-9118 44ACFS-9118 53.433: [ora.drivers.acfs] [2094962448] {0VOVOR 36062} [stop] (: CLSN00010:) ACFS-9118: oracleacfs.ko driver in use-cannot unload.

2012-02-1509 ora.drivers.acfs 44displacement 53.433: [CLSN00010] [2094962448] {0lv 01v 36062} [stop] (: CLSN00010:)

2012-02-1509 execCmd ret 44 execCmd ret 53.433: [ora.drivers.acfs] [2094962448] {0 execCmd ret 0662} [execCmd ret] 53.433

2012-02-1509 CLSN00108 44 clsn_agent::stop 53.434: [ora.drivers.acfs] [2094962448] {0lv 0vir 36062} [stop] (: CLSN00108:) clsn_agent::stop}

2012-02-1509 completed with status 44 completed with status 53.434: [AGFW] [2094962448] {0 stop for resource: ora.drivers.acfs 1 1 completed with status: SUCCESS

CAUSE

The issue was investigated in bug 13726093 and it's closed as OS issue

SOLUTION

Engage OS vendor to disable software that scans ACFS devices.

The temporary workaround is to disable the resource if ACFS is not needed. To disable:

# $GRID_HOME/bin/acfsroot disable

Once it's disabled and node rebooted, it's status will be

Ora.registry.acfs

ONLINE OFFLINE racnode1

Note:

It's known that multipath daemon may open / dev/ofsctl:

Lsof / dev/ofsctl

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

Multipath 5356 root 58r BLK 251,0 0t0 41683748 / dev/ofsctl

Adding ofsctl to multipath blacklist solves the issue.

For findability: rootcrs.pl-unlock; roothas.pl-unlock

REFERENCESBUG:13606669-CRSCTL STOP CLUSTERWARE FAILS TO STOP, ACFS-9118

BUG:13613644-CRSCTL STOP CRS IS NOT STOPPING THE ORA.DRIVERS.ACFS RESOURCE 11.2.0.3

BUG:13726093-CRSCTL STOP CRS FAILS TO STOP ACFS

BUG:13736590-CRS-2799: FAILED TO SHUT DOWN RESOURCE 'ORA.DRIVERS.ACFS'

BUG:13810374-FAILS TO STOP ACFS RESOURCE IN 11.2.0.3

In this example, the multipath-bound hard disk is used as the ASM disk, so the multipath service opens the / dev/ofsctl device, which causes the ora.drivers.acfs to fail. Finally, the whole shutdown command is completed with an error. Set the settings in the multipath configuration file as prompted

Blacklist {

Devnode "^ sda"

Devnode "ofsctl"

}

After setting up, try again to close the command problem resolved.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.