Learning Oracle through cases-- A "bloodshed" caused by misoperation of AIX rac 03/01 Update SLTechnology News&Howtos

Learning Oracle through cases-- A "bloodshed" caused by misoperation of AIX rac

2025-03-01 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >

Shulou(Shulou.com)06/01 Report--

Learning Oracle through cases-- A "bloodshed" caused by misoperation of AIX rac

System environment:

Operating system: AIX 5300-09

Cluster software: CRS 10.2.0.1

Database: Oracle 10.2.0.1

This case is used for VG Concurrent-based shared storage to realize the concurrency of volume groups through HACMP.

Case study:

1. Error phenomena:

1. Oracle users cannot access device files

2. CRS server failed to start

[oracle@aix211 ~] $ls-l / dev

/ dev/__vg10: No permission/dev/audit: No permission/dev/cd0: No permission/dev/clone: No permission/dev/console: No permission/dev/error: No permission

Check the properties of the device file and find that it has been changed to oracle:dba

[oracle@aix211 ~] $ls-ld / dev

Drw-rw---- 6 oracle dba 3584 Sep 16 11:38 / dev

Re-change device file properties

[root@aix211 /] # chown root.system / dev [root@aix211 /] # ls-ld / devdrw-rw---- 6 root system 3584 Sep 16 11:38 / dev [root@aix211 /] # chmod 775 / dev

Oracle users can access device files normally

[root@aix211 /] # su-oracle [oracle@aix211 ~] $ls-l / devtotal 24crwMui control1_1brw-rw-----1 root system 10,0 Aug 29 2013 IPL_rootvgsrwxrwxrwx 1 root system 0 Sep 16 10:22 SRCbrw-rw---- 1 oracle dba 88, 9 Sep 11 12:15 control1_1brw-rw---- 1 oracle dba 88 10 Sep 11 12:15 control2_2brw-rw---- 1 oracle dba 88, 11 Sep 11 12:16 control3_3crw-rw---- 1 root system 88, 0 Sep 11 12:08 datavg

But CRS server still doesn't start properly!

2. Reconfigure CRS:

1. Clean up ocr and vote disk disk information (two nodes)

[root@aix211 /] # dd if=/dev/zero of=/dev/rrac_ocr bs=8192 count=25602560+0 records in2560+0 records out [root@aix211 /] # dd if=/dev/zero of=/dev/rrac_vote bs=8192 count=25602560+0 records in2560+0 records out [root@aix211 /] # ls-l / dev | grep ocrbrw-rw---- 1 oracle dba 88, 1 Sep 11 12:15 rac_ocrcrw-r- 1 root oinstall 88 1 Sep 16 11:05 rrac_ rrac_ [root @ aix211 /] # chown oracle:dba / dev/rrac_ocr

2. Rerun the root.sh script to configure CRS (two nodes)

Node1:

[root@aix211 install] #. / rootdelete.sh

Shutting down Oracle Cluster Ready Services (CRS): Sep 16 11 Sep 48 ERR | ERR | failed to connect to daemon, errno (2) Stopping resources.Error while stopping resources. Possible cause: CRSD is down.Stopping CSSD.Unable to communicate with the CSS daemon.Shutdown has begun. The daemons should exit soon.Checking to see if Oracle CRS stack is down...Oracle CRS stack is not running.Oracle CRS stack is down now.Removing script for Oracle Cluster Ready servicesUpdating ocr file for downgradeCleaning up SCR settings in'/ etc/oracle/scls_scr'

[root@aix211 install] # / u01/crs_1/root.sh

WARNING: directory'/ u01' is not owned by rootChecking to see if Oracle CRS stack is already configuredChecking to see if any 9i GSD is upSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory'/ u01' is not owned by rootclscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node: node 1: aix211 aix211-priv aix211node 2: aix212 aix212-priv aix212clscfg: Arguments check out successfully.NO KEYS WERE WRITTEN. Supply-force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600seconds.CSS is active on these nodes. Aix211CSS is inactive on these nodes. Aix212Local node checking complete.Run root.sh on remaining nodes to start CRS daemons.

Node2:

[root@aix212 install] #. / rootdelete.sh

[root@aix212@ /] # / u01/crs_1/root.sh

WARNING: directory'/ u01' is not owned by rootChecking to see if Oracle CRS stack is already configuredSetting the permissions on OCR backup directorySetting up NS directoriesOracle Cluster Registry configuration upgraded successfullyWARNING: directory'/ u01' is not owned by rootclscfg: EXISTING configuration version 3 detected.clscfg: version 3 is 10G Release 2.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node: node 1: aix211 aix211-priv aix211node 2: aix212 aix212-priv aix212clscfg: Arguments check out successfully.NO KEYS WERE WRITTEN. Supply-force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600seconds.CSS is active on these nodes. Aix211 aix212CSS is active on all nodes.Waiting for the Oracle CRSD and EVMD to startOracle CRS stack installed and running under init (1m) Running vipca (silent) for configuring nodeappsThe given interface (s), "en0" is not public. Public interfaces should be used to configure virtual IPs.

Run vipca on node2 to configure vip

@ so far, CRS has been reconfigured successfully!

[root@aix212@ /] # crsctl check crsCSS appears healthyCRS appears healthyEVM appears healthy [root@aix212@ /] # crs_stat-tName Type Target State Host-ora.aix211.gsd application ONLINE ONLINE aix211 ora.aix211.ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212

Re-register Listener and Database

1. Register listener

Through the netca tool, re-reconfigure can complete the registration of listener!

[root@aix212@ /] # crs_stat-tName Type Target State Host-ora....11.lsnr application ONLINE ONLINE aix211 ora.aix211.gsd application ONLINE ONLINE aix211 ora.aix211.ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora....12.lsnr application ONLINE ONLINE aix212 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212

2. Register Database and Instance

[root@aix212@ /] # srvctl add database-h

Usage: srvctl add database-d-o [- m] [- p] [- A / netmask] [- r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [- s] [- n] [- y {AUTOMATIC | MANUAL}]-d Unique name for the database-o ORACLE_HOME for cluster database-m Domain for cluster database-p Server parameter file for cluster database-A Database cluster alias-n Database name (DB_NAME) If different from the unique name given by the-d option-r Role of the database (primary, physical_standby, logical_standby)-s Startup options for the database-y Management policy for the database (automatic, manual)-h Print usage

[root@aix212@ /] # su-oracle

[oracle@aix212@] $srvctl add database-d prod-o $ORACLE_HOME

[oracle@aix212@] $crs_stat-tName Type Target State Host-ora....11.lsnr application ONLINE ONLINE aix211 ora.aix211.gsd application ONLINE ONLINE Aix211 ora.aix211.ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora....12.lsnr application ONLINE ONLINE aix212 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212 ora.prod.db application OFFLINE OFFLINE

[oracle@aix212@] $srvctl add instance-hUsage: srvctl add instance-d-I-n-d Unique name for the database-i Instance name-n Node name-h Print usage [oracle@aix212@ ~] $srvctl add instance-d prod-I prod1-n aix211 [oracle@aix212@ ~] $srvctl add instance-d prod-I prod2-n aix212 [oracle@aix212@ ~] $crs _ stat-tName Type Target State Host-ora....11.lsnr application ONLINE ONLINE aix211 ora.aix211.gsd application ONLINE ONLINE aix211 ora.aix211. Ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora....12.lsnr application ONLINE ONLINE aix212 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212 ora.prod.db application OFFLINE OFFLINE ora....d1.inst application OFFLINE OFFLINE Ora....d2.inst application OFFLINE OFFLINE

Start Database through the crs tool:

[oracle@aix212@] $srvctl start database-d prod

PRKP-1001: Error starting instance prod1 on node aix211

CRS-0184: Cannot communicate with the CRS daemon.

PRKP-1001: Error starting instance prod2 on node aix212

CRS-0184: Cannot communicate with the CRS daemon.

Failed to start Instance. Start it manually through sqlplus:

[oracle@aix212@] $sqlplus'/ as sysdba'SQL*Plus: Release 10.2.0.1.0-Production on Tue Sep 16 12:08:10 2014Copyright (c) 1982, 2005, Oracle. All rights reserved.Connected to an idle instance.SQL > startupORACLE instance started.Total System Global Area 1258291200 bytesFixed Size 2020552 bytesVariable Size 352324408 bytesDatabase Buffers 889192448 bytesRedo Buffers 14753792 bytesDatabase mounted.Database opened. [oracle@aix211 aix211] $sqlplus'/ as sysdba'SQL*Plus: Release 10.2.0.1.0-Production on Tue Sep 16 12:09:37 2014Copyright (c) 1982,2005, Oracle. All rights reserved.Connected to an idle instance.SQL > startupORACLE instance started.Total System Global Area 1258291200 bytesFixed Size 2020552 bytesVariable Size 335547192 bytesDatabase Buffers 905969664 bytesRedo Buffers 14753792 bytesDatabase mounted.Database opened.

View crs startup resource information:

[oracle@aix211 aix211] $crs_stat-tName Type Target State Host-ora....11.lsnr application ONLINE ONLINE aix211 ora.aix211.gsd application ONLINE ONLINE Aix211 ora.aix211.ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora....12.lsnr application ONLINE ONLINE aix212 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212 ora.prod.db application ONLINE ONLINE aix211 ora....d1.inst application ONLINE ONLINE aix211 Ora....d2.inst application ONLINE ONLINE aix212

Then restart Instance through the crs tool:

[oracle@aix211 aix211] $srvctl stop instance-d prod-I prod1

[oracle@aix211 aix211] $crs_stat-tName Type Target State Host-ora....11.lsnr application ONLINE ONLINE aix211 ora.aix211.gsd application ONLINE ONLINE Aix211 ora.aix211.ons application ONLINE ONLINE aix211 ora.aix211.vip application ONLINE ONLINE aix211 ora....12.lsnr application ONLINE ONLINE aix212 ora.aix212.gsd application ONLINE ONLINE aix212 ora.aix212.ons application ONLINE ONLINE aix212 ora.aix212.vip application ONLINE ONLINE aix212 ora.prod.db application ONLINE ONLINE aix211 ora....d1.inst application OFFLINE OFFLINE Ora....d2.inst application ONLINE ONLINE aix212

[oracle@aix211 aix211] $srvctl start instance-d prod-I prod1

@ so far, Database can be started and closed normally through the crs tool. The bloodshed caused by misoperation has been rescued successfully.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.

*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.