Shulou(Shulou.com)06/01 Report--

Customer site, telecom-related 24-hour business, a province database two-node alert constantly generate ORA-7445 errors, and throw out Trace dump files, Trc files. There are no 7445 related errors in one-node alert, only core files are generated. There are no obvious alarms in crs, syslog and other logs. Did not cause a restart, did not cause business impact. Just keep reporting errors, because the Oracle directory space is 40G, in order to prevent the database instance from being rammed, the customer still decides to solve the error.

Customer host HP-UX B.11.31 ia64,Oracle Release 10.2.0.5.0.

Node 2 alert:

Thu Nov 24 18:27:38 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_17889.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:27:40 EAT 2016

Trace dumping is performing id= [cdmp _ 20161124182740]

Thu Nov 24 18:28:29 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_18076.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:29:03 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_19496.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:43:22 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_12403.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:44:37 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_14293.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:46:41 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_17336.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:50:20 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_20339.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 18:50:22 EAT 2016

Trace dumping is performing id= [cdmp _ 20161124185022]

Thu Nov 24 18:53:06 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_23424.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:06:40 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_18689.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:38:31 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_6449.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:38:33 EAT 2016

Trace dumping is performing id= [cdmp _ 20161124193833]

Thu Nov 24 19:39:49 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_10964.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:39:53 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_9299.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:41:19 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_12404.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Thu Nov 24 19:42:47 EAT 2016

Errors in file / oracle/admin/essbj/udump/essbj2_ora_13863.trc:

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Node 1 alert:

Fri Nov 25 11:16:05 EAT 2016

Trace dumping is performing id= [cdmp _ 20161125111604]

Fri Nov 25 12:00:05 EAT 2016

Thread 1 advanced to log sequence 34397 (LGWR switch)

Current log# 8 seq# 34397 mem# 0: / vgbj03/oradata/essbj/vgbj03_1_rd81.log

Current log# 8 seq# 34397 mem# 1: / vgbj04/oradata/essbj/vgbj04_1_rd82.log

Fri Nov 25 12:00:51 EAT 2016

Errors in file / oracle/admin/essbj/bdump/essbj1_j000_11783.trc:

ORA-12012: error on auto execute of job 281

ORA-00031: session marked for kill

ORA-06512: at "SYS.KILL_LONG_SQL", line 67

ORA-06512: at line 1

Fri Nov 25 12:17:15 EAT 2016

Trace dumping is performing id= [cdmp _ 20161125121714]

Fri Nov 25 12:37:21 EAT 2016

Errors in file / oracle/admin/essbj/bdump/essbj1_j000_5909.trc:

ORA-12012: error on auto execute of job 281

ORA-00031: session marked for kill

ORA-06512: at "SYS.KILL_LONG_SQL", line 67

ORA-06512: at line 1

Fri Nov 25 12:50:53 EAT 2016

Trace dumping is performing id= [cdmp _ 20161125125053]

Fri Nov 25 13:33:53 EAT 2016

Trace dumping is performing id= [cdmp _ 20161125133352]

The trc file generated in node 2 is viewed and a sql is pointed out.

Ksedmp: internal or fatal error

ORA-07445: exception encountered: core dump [kghsrch () + 128] [SIGSEGV] [Address not mapped to object] [0xFFFFFFFFFFFFFFF0] [] []

Current SQL statement for this session:

SELECT OCCUPY_STAFF_ID, OCCUPY_DEPART_ID FROM TF_R_TEMPOCCUPY WHERE RES_NO =: 1 AND RES_TYPE_CODE =: 2 AND RSRV_TAG1 =: 3

The sql is intended to be executed separately, and the tail-f alert log is checked to see if the 7445 related bug error caused by the SQL problem. Because the SQL has bound variables. We need to parse the value of the variable and then execute the sql statement.

Alter session set nls_date_format = 'yyyy-mm-dd,hh34:mi:ss'

Set linesize 400

Col sql_Id format a20

Col name format a20

Col datatype_string format a14

Col value_string format a20

Select sql_id,name, datatype_string, last_captured,value_string

From v$sql_bind_capture where sql_id='&sql_id' order by LAST_CAPTURED,POSITION

After bringing the variable into the SQL for separate execution, no error occurred. The documents closest to the contents of the trace file found on mos are as follows:

SMON Terminates With ORA-7445 [Kghsrch () + 128] (document ID 1189894.1)

Bug 10036960: ORA-07445 [KGHSRCH] AND ORA-07445 [KGHLKREMF] FOLLOWED BY INSTANCE CRASH.

ORA-07445 [kghsrch] Associated With Session Kills (document ID 787914.1)

Instance Termination with ORA-07445 [kghsrch () + 144], ORA-00600 [kghfrh:ds] (document ID 2128933.1)

However, because the database only reports an error and does not cause the consequence of the instance down, it is excluded from being completely consistent with the mos document bug. And the database version of the system itself is already 10.2.0.5 and the fix version described in the bug is this version.

1. 07445 of the problems encountered by the system are only partially consistent with the bug-related information provided by oracle officially.

two。 The solution it provides is not consistent with the current database environment (for example, it is recommended to upgrade from 10gR2 to 10.2.0.4 or 10.2.0.5, but the current production database itself is 10.2.0.5)

3. The instance crash mentioned by bug does not appear in the database, which does not affect the normal operation of the business.

Therefore, the exception thrown may be related to an unexpected situation in the in-memory structure of the business access object, and because it continues to generate trace files and cdump files, customers are advised to restart the thrown problem instance (instance 2) before continuing to execute the business. Then check to see if there is a corresponding 7445 error. Restart the instance after the customer gives a time window.

Tue Nov 29 21:38:37 EAT 2016

ALTER DATABASE OPEN

Block change tracking file is current.

Picked broadcast on commit scheme to generate SCNs

Tue Nov 29 21:38:38 EAT 2016

Sending CIC to internal enable redo thread

Tue Nov 29 21:38:38 EAT 2016

LGWR: STARTING ARCH PROCESSES

ARC0 started with pid=33, OS id=14097

Tue Nov 29 21:38:38 EAT 2016

ARC0: Archival started

ARC1: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC1 started with pid=34, OS id=14099

Tue Nov 29 21:38:38 EAT 2016

Thread 2 opened at log sequence 32257

Current log# 12 seq# 32257 mem# 0: / vgbj03/oradata/essbj/vgbj03_1_rd121.log

Current log# 12 seq# 32257 mem# 1: / vgbj04/oradata/essbj/vgbj04_1_rd122.log

Tue Nov 29 21:38:38 EAT 2016

ARC0: Becoming the'no FAL' ARCH

ARC0: Becoming the'no SRL' ARCH

Tue Nov 29 21:38:38 EAT 2016

Successful open of redo thread 2

Tue Nov 29 21:38:38 EAT 2016

ARC1: Becoming the heartbeat ARCH

Tue Nov 29 21:38:38 EAT 2016

Starting background process CTWR

CTWR started with pid=35, OS id=14101

Block change tracking service is active.

Tue Nov 29 21:38:38 EAT 2016

SMON: enabling cache recovery

Tue Nov 29 21:38:39 EAT 2016

Successfully onlined Undo Tablespace 4.

Tue Nov 29 21:38:39 EAT 2016

SMON: enabling tx recovery

Tue Nov 29 21:38:39 EAT 2016

Database Characterset is ZHS16GBK

Opening with internal Resource Manager plan

Replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

QMNC started with pid=36, OS id=14127

Tue Nov 29 21:38:43 EAT 2016

Completed: ALTER DATABASE OPEN

Tue Nov 29 21:38:52 EAT 2016

ALTER SYSTEM SET service_names='essbj_db' SCOPE=MEMORY SID='essbj2'

Tue Nov 29 21:48:47 EAT 2016

ALTER SYSTEM SET service_names='essbj_db','essbj' SCOPE=MEMORY SID='essbj2'

Wed Nov 30 00:16:10 EAT 2016

ALTER SYSTEM ARCHIVE LOG

Wed Nov 30 00:16:10 EAT 2016

Thread 2 advanced to log sequence 32258 (LGWR switch)

Current log# 4 seq# 32258 mem# 0: / vgbj04/oradata/essbj/vgbj04_1_rd41.log

Current log# 4 seq# 32258 mem# 1: / vgbj03/oradata/essbj/vgbj03_1_rd42.log

Wed Nov 30 00:16:15 EAT 2016

ALTER SYSTEM ARCHIVE LOG

Wed Nov 30 00:16:17 EAT 2016

Thread 2 advanced to log sequence 32259 (LGWR switch)

Current log# 3 seq# 32259 mem# 0: / vgbj04/oradata/essbj/vgbj04_1_rd31.log

Current log# 3 seq# 32259 mem# 1: / vgbj03/oradata/essbj/vgbj03_1_rd32.log

Wed Nov 30 02:00:27 EAT 2016

Thread 2 advanced to log sequence 32260 (LGWR switch)

Current log# 5 seq# 32260 mem# 0: / vgbj03/oradata/essbj/vgbj03_1_rd51.log

Current log# 5 seq# 32260 mem# 1: / vgbj04/oradata/essbj/vgbj04_1_rd52.log

Wed Nov 30 06:13:11 EAT 2016

Thread 2 advanced to log sequence 32261 (LGWR switch)

Current log# 10 seq# 32261 mem# 0: / vgbj03/oradata/essbj/vgbj03_1_rd101.log

Current log# 10 seq# 32261 mem# 1: / vgbj04/oradata/essbj/vgbj04_1_rd102.log

There are no more related errors.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.