Shulou(Shulou.com)06/01 Report--

Received a call informing the company that a maintenance customer database in charge of the company has a fault

Customer feedback, disk read/write error occurred on the application side:

There was also an error logging into the database:

At the same time, users feedback that some services have been interrupted

Log in to the operating system and query the gv$instance view to find out that it is a two-node 11gR2 cluster

Check the database alarm log,ASM alarm log and CRSD log of node 1, and no recent suspicious error message is found.

Log in to node 2 and find that the database service's/oracle directory has reached 100% space consumption. Check the disk file space usage found, listening generated listener.log, XML file takes up a lot of space, so clean it up

From the error log, there was a problem with disk writing and suspected physical bad blocks in the data file. But dbv userid/password file ='+ DATA/database name/datafile/system.345'does not detect bad blocks

After clearing out the oracle installation directory, users reported that business had returned to normal. But there is still some confusion about this error, why disk space problems can lead to data file write errors.

Continue checking:

The following information appears in ASM's alarm log:

Tue Jun 12 17:00:27 2018

Non critical error DIA-48181 caught while writing to trace file "/oracle/app/diag/asm/+asm/+ASM2/trace/+ASM2_ora_5142.trc"

Error message: HPUX-ia64 Error: 28: No space left on device

Additional information: 1

Writing to the above trace file is disabled for now on...

Non critical error ORA-48113 caught while writing to trace file "/oracle/app/diag/asm/+asm/+ASM2/trace/+ASM2_ora_14098.trc"

Error message:

Writing to the above trace file is disabled for now on...

WARNING: ASM communication error: op 0 state 0x0 (15055)

ERROR: direct connection failure with ASM

NOTE: Deferred communication with ASM instance

ASM connectivity loss messages have appeared here.

No wonder disk read and write errors occur

After cleaning the log file, no new information is written into the alarm log.

Check cluster component status:

$ /oracle/grid/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

CRS service has experienced problems

Summary:

It should be inferred that the/oracle directory is full of disk space, causing an error logging into ASM when the database instance reads or writes data files through ASM. At this point, the error shown in the previous section to write to the system table space occurs

Here's another key tip:

ORA-29701: unable to connect to Cluster Sychironization Service

There is already a cluster level process problem. No connection to the data file itself (business returned to normal after clearing space)

At the same time, he gave advice to customers:

Suggestions for improvement:

1. It is recommended to deploy crontab script to regularly clean up the listening logs and log-related xml files in/oracle directory.

2. CRS cluster has experienced problems and has not recovered after a long time. It is recommended to restart CRS cluster service.

by

/oracle/grid/bin/crsctl stop crs

/oracle/grid/bin/crsctl start crs

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.