Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to solve the problem that a node crs can not start due to gipc in oracle". The content of the article is simple and clear, and it is easy to learn and understand. Please follow Xiaobian's train of thought to study and learn "how to solve the problem that a node crs can not start in oracle due to gipc".

Two-node RAC, of which one node cluster CRS cannot be started. After analysis, the reason is that the 2-node gipcd process is abnormal, which leads to the failure of normal communication between nodes, and the problem can be recovered after restarting 2-node gipcd.bin. From the point of view of the phenomenon, ora.crsd and ora.evmd cannot be started, and the other components are normal.

1. Inspection and analysis 1.1. Node 1 Cluster alert Log

The manual restart of node 1 cluster log at 13:08 is as follows. The information that cannot be deleted by olsnodes.log always exists in this environment, and the information here can be ignored.

2018-11-26 13 140 0814 29.521:

[client] CRS-0009:log file "/ home/u01/app/grid/11.2.0/product/log/sxmms1/client/olsnodes.log" reopened

2018-11-26 13 140 0814 29.521:

[client] CRS-0019:file rotation terminated. Log file: "/ home/u01/app/grid/11.2.0/product/log/sxmms1/client/olsnodes.log"

2018-11-26 13 08R 42.421:

[ohasd (903)] CRS-2112:The OLR service started on node sxmms1.

2018-11-26 13 08R 42.433:

[ohasd (903)] CRS-1301:Oracle High Availability Service started on node sxmms1.

2018-11-26 13 08R 42.433:

[ohasd] CRS-8017:location: / etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred

2018-11-26 13 0814 45.864:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin] CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2018-11-26 13 08R 51.238:

[gpnpd (1118)] CRS-2328:GPNPD started on node sxmms1.

2018-11-26 13 08V 53.710:

[cssd (1184)] CRS-1713:CSSD daemon is started in clustered mode

2018-11-26 13 08R 55.508:

[ohasd] CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE

2018-11-26 13 08R 55.509:

[ohasd] CRS-2769:Unable to failover resource 'ora.diskmon'.

2018-11-26 13 purl 09 purl 03.406:

[cssd (1184)] CRS-1707:Lease acquisition for node sxmms1 number 1 completed

2018-11-26 13 purl 09 purl 04.658:

[cssd (1184)] CRS-1605:CSSD voting file is online: ORCL:OCR2; details in / home/u01/app/grid/11.2.0/product/log/sxmms1/cssd/ocssd.log.

2018-11-26 13 purl 09 purl 07.670:

[cssd (1184)] CRS-1601:CSSD Reconfiguration complete. Active nodes are sxmms1 sxmms2.

2018-11-26 13 purl 099 purl 09.989:

[ctssd (1269)] CRS-2407:The new Cluster Time Synchronization Service reference node is host sxmms2.

2018-11-26 13 purl 099 purl 09.990:

[ctssd (1269)] CRS-2401:The Cluster Time Synchronization Service started on host sxmms1.

2018-11-26 13 purl 09 purl 11.701:

[ohasd] CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE

2018-11-26 13 purl 09 purl 11.701:

[ohasd] CRS-2769:Unable to failover resource 'ora.diskmon'.

2018-11-26 13 1015 08.710:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (1129)] CRS-5818:Aborted command 'start' for resource' ora.ctssd'. Details at (: CRSAGF00113:) {0:0:2} in / home/u01/app/grid/11.2.0/product/log/sxmms1/agent/ohasd/orarootagent_root/orarootagent_root.log.

2018-11-26 13 10 12. 714:

[ohasd] CRS-2757:Command 'Start' timed out waiting for response from the resource' ora.ctssd'. Details at (: CRSPE00111:) {0:0:2} in / home/u01/app/grid/11.2.0/product/log/sxmms1/ohasd/ohasd.log.

[client (1584)] CRS-10001:26-Nov-18 13:10 ACFS-9391: Checking for existing ADVM/ACFS installation.

[client (1589)] CRS-10001:26-Nov-18 13:10 ACFS-9392: Validating ADVM/ACFS installation files for operating system.

[client (1591)] CRS-10001:26-Nov-18 13:10 ACFS-9393: Verifying ASM Administrator setup.

[client (1594)] CRS-10001:26-Nov-18 13:10 ACFS-9308: Loading installed ADVM/ACFS drivers.

[client (1597)] CRS-10001:26-Nov-18 13:10 ACFS-9154: Loading 'oracleoks.ko' driver.

[client (1625)] CRS-10001:26-Nov-18 13:10 ACFS-9154: Loading 'oracleadvm.ko' driver.

[client (1653)] CRS-10001:26-Nov-18 13:10 ACFS-9154: Loading 'oracleacfs.ko' driver.

[client (1764)] CRS-10001:26-Nov-18 13:10 ACFS-9327: Verifying ADVM/ACFS devices.

[client (1773)] CRS-10001:26-Nov-18 13:10 ACFS-9156: Detecting control device'/ dev/asm/.asm_ctl_spec'.

[client (1777)] CRS-10001:26-Nov-18 13:10 ACFS-9156: Detecting control device'/ dev/ofsctl'.

[client (1782)] CRS-10001:26-Nov-18 13:10 ACFS-9322: completed

2018-11-26 13 140 10 14 067:

[ohasd] CRS-2807:Resource 'ora.asm' failed to start automatically.

2018-11-26 13 140 10 14 067:

[ohasd] CRS-2807:Resource 'ora.crsd' failed to start automatically.

2018-11-26 13 140 10 14 067:

[ohasd] CRS-2807:Resource 'ora.evmd' failed to start automatically.

2018-11-26 13 1114 42.738:

[ohasd] CRS-2765:Resource 'ora.ctssd' has failed on server' sxmms1'.

2018-11-26 13 1115 45.381:

[ctssd (2151)] CRS-2407:The new Cluster Time Synchronization Service reference node is host sxmms2.

2018-11-26 13 1115 45.382:

[ctssd (2151)] CRS-2401:The Cluster Time Synchronization Service started on host sxmms1.

1.2. Node 1 AGENT analysis

The log only intercepts part of the content. from the perspective of the log, almost many components have timed out at startup.

/ home/u01/app/grid/11.2.0/product/log/sxmms1/agent/ohasd/orarootagent_root/orarootagent_root.log

2018-11-26 13 clsdmc_respget return 10 clsdmc_respget return: [ora.ctssd] [2525660928] {0:0:2} [start] clsdmc_respget return: status=0, ecode=0, returnbuf= [0x7f51780ce0c0], buflen=8

2018-11-26 13 Start 10 Start 06.792: [ora.ctssd] [2525660928] {0:0:2} [start] Start: "? with length of 8

2018-11-26 13 translateReturnCodes 10: [ora.ctssd] [2525660928] {0:0:2} [start] translateReturnCodes, return = 0, state detail = Checkcb data [0x7f51780ce0c0]: mode [0xc0] offset [0 ms].

[clsdmc] [2525660928] CLSDMC.C returnbuflen=8, extraDataBuf=C0, returnbuf=7805FCE0

2018-11-26 13 clsdmc_respget return 10 clsdmc_respget return: [ora.ctssd] [2525660928] {0:0:2} [start] clsdmc_respget return: status=0, ecode=0, returnbuf= [0x7f517805fce0], buflen=8

2018-11-26 13 Start 10 Start 07.793: [ora.ctssd] [2525660928] {0:0:2} [start] Start: "? with length of 8

2018-11-26 13 translateReturnCodes 10: [ora.ctssd] [2525660928] {0:0:2} [start] translateReturnCodes, return = 0, state detail = Checkcb data [0x7f517805fce0]: mode [0xc0] offset [0 ms].

2018-11-26 13 CRSAGF00113 10: [AGENT] [2527762176] {0:0:2} {0:0:2} Created alert: (: CRSAGF00113:): Aborting the command: start for resource: ora.ctssd 11

2018-11-26 13 CLSN00110 10: [ora.ctssd] [2527762176] {0:0:2} [start] (: CLSN00110:) clsn_agent::abort {

2018-11-26 13 abort 10 abort 08.710: [ora.ctssd] [2527762176] {0:0:2} [start]

2018-11-26 13 Agent::abort last call info 10 Agent::abort last call info 08.710: [ora.ctssd] [2527762176] {0:0:2} [start] Agent::abort last call info: "Agent::Agent refreshAttr"

2018-11-26 13 abort command 10 abort command 08.710: [ora.ctssd] [2527762176] {0:0:2} [start] abort command: start

2018-11-26 13 tryActionLock 10 tryActionLock 08.710: [ora.ctssd] [2527762176] {0:0:2} [start]

[clsdmc] [2525660928] CLSDMC.C returnbuflen=8, extraDataBuf=C0, returnbuf=780CE0C0

2018-11-26 13 clsdmc_respget return 10: [ora.ctssd] [2525660928] {0:0:2} [start] clsdmc_respget return: status=0, ecode=0, returnbuf= [0x7f51780ce0c0], buflen=8

2018-11-26 13 Start 10: [ora.ctssd] [2525660928] {0:0:2} [start] Start: Extended check return buffer: "? with length of 8

2018-11-26 13 translateReturnCodes 10: [ora.ctssd] [2525660928] {0:0:2} [start] translateReturnCodes, return = 0, state detail = Checkcb data [0x7f51780ce0c0]: mode [0xc0] offset [0 ms].

2018-11-26 13 Start action aborted 10 Start action aborted 08.795: [ora.ctssd] [2525660928] {0:0:2} [start]

2018-11-26 13 clsnUtils::error Exception type=2 string= 10 clsnUtils::error Exception type=2 string= 08.796: [ora.ctssd] [2525660928] {0:0:2} [start]

CRS-5017: The resource action "ora.ctssd start" encountered the following error:

Start action for octssd aborted. For details refer to "(: CLSN00107:)" in "/ home/u01/app/grid/11.2.0/product/log/sxmms1/agent/ohasd/orarootagent_root/orarootagent_root.log".

2018-11-26 13 CRS-5017 10 CRS-5017 08.796: [AGFW] [2525660928] {0:0:2} sending status msg [CRS-5017: The resource action "ora.ctssd start" encountered the following error:

Start action for octssd aborted. For details refer to "(: CLSN00107:)" in "/ home/u01/app/grid/11.2.0/product/log/sxmms1/agent/ohasd/orarootagent_root/orarootagent_root.log".

] for start for resource: ora.ctssd 1 1

2018-11-26 13 CLSN00107 10 clsn_agent::start 08.796: [ora.ctssd] [2525660928] {0:0:2} [start] (: CLSN00107:) clsn_agent::start}

2018-11-26 13 RESOURCE_ 10 ID 08.797: [AGFW] [2523559680] {0:0:2} Agent sending reply for: RESOURCE_ start [ora.ctssd 11] 361

2018-11-26 13 got lock 10 12. 711: [ora.ctssd] [2527762176] {0:0:2} [start] got lock

2018-11-26 13 tryActionLock 10 12. 711: [ora.ctssd] [2527762176] {0:0:2} [start] tryActionLock}

2018-11-26 13 abort 10 12. 711: [ora.ctssd] [2527762176] {0:0:2} [start] abort}

2018-11-26 13 CLSN00110 10 clsn_agent::abort 12.711: [ora.ctssd] [2527762176] {0:0:2} [start] (: CLSN00110:) clsn_agent::abort}

2018-11-26 13 start for resource 10 completed with status 12.711: [AGFW] [2527762176] {0:0:2} Command: start for resource: ora.ctssd 11 completed with status: TIMEDOUT

2018-11-26 13 RESOURCE_ 10 ID 12.712: [AGFW] [2523559680] {0:0:2} Agent sending reply for: RESOURCE_ start [ora.ctssd 11] 361

[clsdmc] [2527762176] CLSDMC.C returnbuflen=8, extraDataBuf=C0, returnbuf=84006B40

1.3. Node 1CRSD log analysis

The CRSD log sees an exception message, but does not specifically point to where the problem caused it to fail to start.

2018-11-26 13 03V 59.877: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 04 29.881: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 04 59.886: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 05VR 29.890: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 05VR 59.895: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 06 29.897: [CRSMAIN] [3661960960] Policy Engine is not initialized yet!

2018-11-26 13 init CSS context succeeded 1155.357: [CRSMAIN] [3963266848] First attempt: init CSS context succeeded.

[clsdmt] [3956815616] Listening to (ADDRESS= (PROTOCOL=ipc) (KEY=sxmms1DBG_CRSD))

2018-11-26 13 connkey 1115 55.361: [clsdmt] [3956815616] PID for the Process [2324]

2018-11-26 13 file for home/ home/u01/app/grid/11.2.0/product host sxmms1 bin crs to 1115 55.361: [clsdmt] [3956815616] Creating PID [2324] file for home/ home/u01/app/grid/11.2.0/product host sxmms1 bin crs to / home/u01/app/grid/11.2.0/product/crs/init/

2018-11-26 13 to the file 1115 55.362: [clsdmt] [3956815616] Writing PID [2324] to the file [/ home/u01/app/grid/11.2.0/product/crs/init/sxmms1.pid]

2018-11-26 13 1115 56.304: [CRSMAIN] [3956815616] Policy Engine is not initialized yet!

2018-11-26 13 1115 56.304: [CRSMAIN] [3963266848] CRS Daemon Starting

2018-11-26 13 allcomp 1115 allcomp 56.305: [CRSD] [3963266848]

2018-11-26 13 default 1115 default 56.305: [CRSD] [3963266848]

2018-11-26 13 COMMCRS 1115 COMMCRS 56.305: [CRSD] [3963266848]

2018-11-26 13 COMMNS 1115 COMMNS 56.305: [CRSD] [3963266848]

2018-11-26 13 CSSCLNT 1115 CSSCLNT 56.305: [CRSD] [3963266848]

2018-11-26 13 GIPCLIB 1115 GIPCLIB 56.305: [CRSD] [3963266848]

2018-11-26 13 GIPCXBAD 1115 GIPCXBAD 56.305: [CRSD] [3963266848]

2018-11-26 13 GIPCLXPT 1115 GIPCLXPT 56.305: [CRSD] [3963266848]

2018-11-26 13 GIPCUNDE 1115 GIPCUNDE 56.305: [CRSD] [3963266848]

2018-11-26 13 GIPC 1115 GIPC 56.305: [CRSD] [3963266848]

1.4. Node 1EVMD log analysis

The evmd log is the information that cannot be started in the morning, but the subsequent contents are the same, so the error message at 09:51 can also be used for analysis. The log shows that CRS cannot be started, gipc connection timed out, and OCR problems are excluded because all other nodes are normal.

2018-11-26 09 error while waiting for connection complete 51V 56.086: [OCRMSG] [2406381344] prom_connect: error while waiting for connection complete [24]

2018-11-26 09 51V 56.086: [CRSOCR] [2406381344] OCR context init failure. Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]

2018-11-26 09 CONN NOT ESTABLISHED 51V 57.088: [OCRMSG] [2406381344] prom_waitconnect: CONN NOT ESTABLISHED

2018-11-26 09 msg 51V 57.088: [OCRMSG] [2406381344] GIPC error [29] gipcretConnectionRefused

2018-11-26 09 error while waiting for connection complete 51V 57.088: [OCRMSG] [2406381344] prom_connect: error while waiting for connection complete [24]

2018-11-26 09 51V 57.088: [CRSOCR] [2406381344] OCR context init failure. Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]

2018-11-26 09 CONN NOT ESTABLISHED 51R 58.089: [OCRMSG] [2406381344] prom_waitconnect: CONN NOT ESTABLISHED

2018-11-26 09 msg 58.089: [OCRMSG] [2406381344] GIPC error [29] msg [gipcretConnectionRefused]

2018-11-26 09 OCRMSG 51R 58.089: [OCRMSG] [2406381344] prom_connect: error while waiting for connection complete [24]

2018-11-26 09 51 OCR context init failure 58.089: [CRSOCR] [2406381344]. Error: PROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]

1.5. Node 1GIPC log analysis

According to the analysis of the gipc log, it is found that a timeout occurred when detecting the connection with the 2 nodes, resulting in that the connection could not be established. Gipc is the connection establishment of the private network in the RAC cluster. When there is a problem in establishing the connection, it needs further analysis.

2018-11-26 13 endp 0000000000001a67 0515 GIPCDCLT: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000001a67

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 28.954: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000348

2018-11-26 13 2946344704 2946344704 gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000000723

2018-11-26 13 found node sxmms1 05 gipcdMonitorCssCheck 29.109: [GIPCDMON] [2818569984]

2018-11-26 13 found node sxmms2 05 gipcdMonitorCssCheck 29.110: [GIPCDMON] [2818569984]

2018-11-26 13 updating timeout node sxmms2 05 gipcdMonitorCssCheck 29.110: [GIPCDMON] [2818569984]

2018-11-26 13 skipping live node 05 sxmms2', time 29.110: [GIPCDMON] [2818569984] gipcdMonitorFailZombieNodes: skipping live node 'sxmms2', time 0 ms, endp 00000000000000, 00000000000007ba

2018-11-26 13 interfaces 0515 Returning NETDATA 30.624: [CLSINET] [2818569984]

2018-11-26 13 eth3',ip='192.168.0.1',mac='a0 05Partition 30.624: [CLSINET] [2818569984] # 0 Interface 'eth3',ip='192.168.0.1',mac='a0-36-9fkashi 5dkashi 5cFukui 54" maskhorse "255.255.255.0" jurisprudential" 192.168.0.0" academic usefulness clusterpieces interconnect'

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05V 31.960: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000032d

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 32.478: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000121

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 32.559: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000001a67

2018-11-26 13 inf 05 rank 33.550: [GIPCDMON] [2818569984] gipcdMonitorSaveInfMetrics: inf [0] eth3-rank 99, avgms 1.111111 [151117 / 11717]

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 33.964: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000348

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05purl 34.114: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000723

2018-11-26 13 found node sxmms1 0515 gipcdMonitorCssCheck 34.115: [GIPCDMON] [2818569984]

2018-11-26 13 found node sxmms2 05purl 34.116: [GIPCDMON] [2818569984] gipcdMonitorCssCheck: found node sxmms2

2018-11-26 13 updating timeout node sxmms2 05purl 34.116: [GIPCDMON] [2818569984] gipcdMonitorCssCheck: updating timeout node sxmms2

2018-11-26 13 skipping live node 05purl 34.116: [GIPCDMON] [2818569984] gipcdMonitorFailZombieNodes: skipping live node 'sxmms2', time 0 ms, endp 00000000000000, 00000000000007ba

2018-11-26 13 interfaces 05 interfaces 35.646: [CLSINET] [2818569984]

2018-11-26 13 eth3',ip='192.168.0.1',mac='a0 05Partition 35.646: [CLSINET] [2818569984] # 0 Interface 'eth3',ip='192.168.0.1',mac='a0-36-9fkashi 5dkashi 5cFukui 5cFukui 54pm maskcare "255.255.255.0" jurisdiction netbook" 192.168.0.0"

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05V 36.971: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000032d

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 37.480: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000121

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 37.565: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000001a67

2018-11-26 13 endp 0515 GIPCDCLT: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000348

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 0515 gipcdClientThread 39.120: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000723

2018-11-26 13 found node sxmms1 0515 gipcdMonitorCssCheck 39.121: [GIPCDMON] [2818569984]

2018-11-26 13 found node sxmms2 0515 gipcdMonitorCssCheck 39.122: [GIPCDMON] [2818569984]

2018-11-26 13 updating timeout node sxmms2 0515 gipcdMonitorCssCheck 39.122: [GIPCDMON] [2818569984]

2018-11-26 13 skipping live node 05 sxmms2', time 39.122: [GIPCDMON] [2818569984] gipcdMonitorFailZombieNodes: skipping live node 'sxmms2', time 0 ms, endp 00000000000000, 00000000000007ba

2018-11-26 13 interfaces 0515 Returning NETDATA 40.626: [CLSINET] [2818569984]

2018-11-26 13 eth3',ip='192.168.0.1',mac='a0 05Partition 40.626: [CLSINET] [2818569984] # 0 Interface 'eth3',ip='192.168.0.1',mac='a0-36-9fkashi 5dkashi 5cFukui 54" maskseed "255.255.255.0" jurisprudential" 192.168.0.0" academic usefulness "clusterfolk interconnect'

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 41.982: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000032d

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 42.482: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000121

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05 endp 0000000000001a67 42.572: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000001a67

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 43.986: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000348

2018-11-26 13 2946344704 gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000000723

2018-11-26 13 found node sxmms1 0515 gipcdMonitorCssCheck 44.128: [GIPCDMON] [2818569984]

2018-11-26 13 found node sxmms2 0515 gipcdMonitorCssCheck 44.128: [GIPCDMON] [2818569984]

2018-11-26 13 updating timeout node sxmms2 0515 gipcdMonitorCssCheck 44.128: [GIPCDMON] [2818569984]

2018-11-26 13 skipping live node 05 sxmms2', time 44.128: [GIPCDMON] [2818569984] gipcdMonitorFailZombieNodes: skipping live node 'sxmms2', time 0 ms, endp 00000000000000, 00000000000007ba

2018-11-26 13 interfaces 0515 Returning NETDATA 45.632: [CLSINET] [2818569984]

2018-11-26 13 eth3',ip='192.168.0.1',mac='a0 05Partition 45.632: [CLSINET] [2818569984] # 0 Interface 'eth3',ip='192.168.0.1',mac='a0-36-9fkashi 5dkashi 5cmuri 5cmuri 54 maskshaw "255.255.255.0"

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05VR 46.992: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000032d

2018-11-26 13 2946344704 2946344704 gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000121

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05 req from local client of type gipcdmsgtypeInterfaceMetrics 47.578: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000001a67

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 05Frev 48.997: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000348

2018-11-26 13 gipcdClientInterfaceMetrics: Received type (gipcdmsgtypeInterfaceMetrics), endp (00000000000348), len (1032), buf (0x7f7ca0254448), inf (ip: 192.168.0.1 Received type 14459, mask: 255.255.255.0, subnet: 192.168.0.0, mac:, ifname:) time (0), retry (0), stamp (0), send (0), recv (0)

2018-11-26 13 enqueue local interface metrics 05 to worklist 48.997: [GIPCDCLT] [2946344704] gipcdClientInterfaceMetrics: enqueue local interface metrics (1) to worklist

2018-11-26 13 13 GIPCDCLT 05VR 49.133: [GIPCDCLT] [2946344704] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000723

2018-11-26 13 gipcdClientInterfaceMetrics: Received type (gipcdmsgtypeInterfaceMetrics), endp (00000000000723), len (1032), buf (0x7f7ca0254448), inf (ip: 192.168.0.1ve29489, mask: 255.255.255.0, subnet: 192.168.0.0, mac:, ifname:) time (0), retry (0), stamp (0), send (0), recv (0)

2018-11-26 13 13 enqueue local interface metrics 05 to worklist 49.133: [GIPCDCLT] [2946344704] gipcdClientInterfaceMetrics: enqueue local interface metrics (1) to worklist

2018-11-26 13 found node sxmms1 0515 gipcdMonitorCssCheck 49.134: [GIPCDMON] [2818569984]

2018-11-26 13 found node sxmms2 0515 gipcdMonitorCssCheck 49.134: [GIPCDMON] [2818569984]

2018-11-26 13 updating timeout node sxmms2 0515 gipcdMonitorCssCheck 49.135: [GIPCDMON] [2818569984]

1.6. Time-out analysis of connection establishment

According to the above analysis, the failure to start some resources is caused by the gipc establishment timeout, and problems in the private network can be divided into several situations.

1. There are problems with network card, network configuration, routing, network switch and so on at the os level.

2. Exception occurs in iptable, selinux, etc.

2. At least one process of the two nodes gipc has an exception.

Through the inspection of the 1-node system log massage, netstat, iptables, selinux, etc., it is confirmed that the network at the system level is normal, and the private network can also be ping-connected, and others can also be judged indirectly by the cluster status. Because the 1-node cssd service is normal, it can basically be confirmed that the connectivity of the private network is normal, while the rank value of the eht2 network card in the 1-node gipc log is 99, and the status of the network card is normal, excluding the network card and gipc of the 1 node. You need to check the 2 nodes for further judgment.

1.7. Node 2 Cluster ALERT Log

According to the ALERT log analysis of node 2 cluster, during the restart of node 1 at 13:08, the gipc, evmd and ctss services of node 2 were also abnormal.

2018-11-26 1300 40.747:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 13 06V 57.926:

[cssd (15973)] CRS-1625:Node sxmms1, number 1, was manually shut down

2018-11-26 13 06V 57.944:

[cssd (15973)] CRS-1601:CSSD Reconfiguration complete. Active nodes are sxmms2.

2018-11-26 13 09R 08.318:

[cssd (15973)] CRS-1601:CSSD Reconfiguration complete. Active nodes are sxmms1 sxmms2.

2018-11-26 13 1015 46.812:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 13 2015 52.862:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 13 31R 02.879:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 13 41 12. 924:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 13 51R 20.976:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 14 01R 33.018:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 14 1115 45.027:

[/ home/u01/app/grid/11.2.0/product/bin/orarootagent.bin (6098)] CRS-5018: (: CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0

2018-11-26 14 2015 10.218:

[ohasd (5572)] CRS-2765:Resource 'ora.gipcd' has failed on server' sxmms2'.

2018-11-26 14 2015 10.287:

[ohasd (5572)] CRS-2765:Resource 'ora.ctssd' has failed on server' sxmms2'.

2018-11-26 14 2015 10.870:

[ohasd (5572)] CRS-2765:Resource 'ora.evmd' has failed on server' sxmms2'.

2018-11-26 14 2015 11.060:

[/ home/u01/app/grid/11.2.0/product/bin/scriptagent.bin (52396)] CRS-5822:Agent'/ home/u01/app/grid/11.2.0/product/bin/scriptagent_grid' disconnected from server. Details at (: CRSAGF00117:) {0:11:97} in / home/u01/app/grid/11.2.0/product/log/sxmms2/agent/crsd/scriptagent_grid/scriptagent_grid.log.

2018-11-26 14 2015 11.060:

1.8. Node 2 GIPC log analysis

According to the analysis of the log, it is found that the rank value has exceeded 0, and the drop information has timed out, while the system log, network card and netstat of the two nodes are not abnormal, so it can be basically confirmed that the connection between the two nodes can not be established due to the abnormal gipc process of the two nodes.

2018-11-26 13 found node sxmms2 07 found node sxmms2 41.526: [GIPCDMON] [4093638400]

2018-11-26 13 skipping live node 07VR 41.526: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 737498576 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 0715 GIPCDCLT: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 endp 0000000011befdfb 42.226: [GIPCDCLT] [4166940416] gipcdClientThread:

2018-11-26 13 interfaces 07 interfaces 42.384: [CLSINET] [4093638400]

2018-11-26 13 eth3',ip='192.168.0.2',mac='a0 07Partition 42.384: [CLSINET] [4093638400] # 0 Interface 'eth3',ip='192.168.0.2',mac='a0-36-9f Mustang 5dPUBE 66Repulet eccentricities massively "255.255.255.0"

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07VR 43.840: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000370

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 44.077: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011bf00f6

2018-11-26 13 inf 07VR 44.888: [GIPCDMON] [4093638400] gipcdMonitorSaveInfMetrics: inf [0] eth3-rank 0, avgms 3000000000.000000 [16 / 0 / 0]

2018-11-26 13 saving 07 saving 44.888: [GIPCDMON] [4093638400] gipcdMonitorSaveInfMetrics: saving: eth3:0

2018-11-26 13 no valid interfaces found to node for 07 no valid interfaces found to node for 45.542: [GIPCHALO] [4091537152] gipchaLowerProcessNode: no valid interfaces found to node for 737502596 ms, node 0x7f00d022b420 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-09e82f35, dstLuid 50a77767-85458162 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [11888: 11888], createTime 725590486, sentRegister 1, localMonitor 1, flags 0x4}

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 45.812: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000000ca1

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 46.529: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befc2f

2018-11-26 13 found node sxmms2 07 found node sxmms2 46.531: [GIPCDMON] [4093638400]

2018-11-26 13 skipping live node 07VR 46.531: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 737503586 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07VR 46.544: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 47.232: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befdfb

2018-11-26 13 Received type 07 Received type: [GIPCDCLT] [4166940416] gipcdClientInterfaceMetrics: Received type (gipcdmsgtypeInterfaceMetrics), endp (0000000011befdfb), len (1032), buf (0x7f00e82872e8), inf (ip: 192.168.0.2 gipcdmsgtypeInterfaceMetrics, mask: 255.255.255.0, subnet: 192.168.0.0, mac:, ifname:) time (0), retry (0), stamp (0), send (0), recv (0)

2018-11-26 13 enqueue local interface metrics 07 enqueue local interface metrics 47.232: [GIPCDCLT] [4166940416] gipcdClientInterfaceMetrics: enqueue local interface metrics (1) to worklist

2018-11-26 13 interfaces 07 interfaces 47.380: [CLSINET] [4093638400] Returning NETDATA: 1

2018-11-26 13 eth3',ip='192.168.0.2',mac='a0 07Parade 47.380: [CLSINET] [4093638400] # 0 Interface 'eth3',ip='192.168.0.2',mac='a0-36-9f Mustang 5dPUBE 66Kui eccentricities masquerade "255.255.255.0" mentality netbooks" 192.168.0.0"

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07Frev 48.846: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000370

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 endp 0000000011bf00f6 49.083: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 endp 0000000000000ca1 50.813: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 51.534: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befc2f

2018-11-26 13 found node sxmms2 07 gipcdMonitorCssCheck 51.536: [GIPCDMON] [4093638400]

2018-11-26 13 skipping live node 07VR 51.536: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 737508586 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 gipchaLowerProcessNode: no valid interfaces found to node for 737508606 ms, node 0x7f00d022b420 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-09e82f35, dstLuid 50a77767-85458162 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [11894: 11894], createTime 725590486, sentRegister 1, localMonitor 1, flags 0x4}

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07VR 51.550: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 52.237: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befdfb

2018-11-26 13 interfaces 07 interfaces 52.386: [CLSINET] [4093638400]

2018-11-26 13 eth3',ip='192.168.0.2',mac='a0 07Partition 52.386: [CLSINET] [4093638400] # 0 Interface 'eth3',ip='192.168.0.2',mac='a0-36-9f Mustang 5dPUBE 66 Mustang eccentricities, maskskills, 255.255.255.0, "192.168.0.0"

2018-11-26 13 4166940416 4166940416 gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000370

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 54.089: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011bf00f6

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 55.815: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000000ca1

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 endp 0000000011befc2f 56.540: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics

2018-11-26 13 found node sxmms2 07 found node sxmms2 56.541: [GIPCDMON] [4093638400]

2018-11-26 13 skipping live node 07VR 56.541: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 737513596 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07VR 56.556: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 req from local client of type gipcdmsgtypeInterfaceMetrics 57.243: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befdfb

2018-11-26 13 interfaces 07 interfaces 57.381: [CLSINET] [4093638400]

2018-11-26 13 eth3',ip='192.168.0.2',mac='a0 07Partition 57.381: [CLSINET] [4093638400] # 0 Interface 'eth3',ip='192.168.0.2',mac='a0-36-9f Mustang 5dPUBE 66Repulet eccentricities massively "255.255.255.0" cognac" 192.168.0.0"

2018-11-26 13 no valid interfaces found to node for 07 no valid interfaces found to node for 57.556: [GIPCHALO] [4091537152] gipchaLowerProcessNode: no valid interfaces found to node for 737514606 ms, node 0x7f00d022b420 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-09e82f35, dstLuid 50a77767-85458162 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [11900: 119 900], createTime 725590486, sentRegister 1, localMonitor 1, flags 0x4}

2018-11-26 13 13 req from local client of type gipcdmsgtypeInterfaceMetrics 0715 856: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000370

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 07 endp 0000000011bf00f6 59.095: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics

2018-11-26 13 13 req from local client of type gipcdmsgtypeInterfaceMetrics 00.816: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000000000ca1

2018-11-26 13 13 req from local client of type gipcdmsgtypeInterfaceMetrics 01.544: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befc2f

2018-11-26 13 GIPCDMON 0815 01.546: [GIPCDMON] [4093638400] gipcdMonitorCssCheck: found node sxmms2

2018-11-26 13 skipping live node 08VR 01.546: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 737518596 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 GIPCDCLT 0815 01.561: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 0815 02.248: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befdfb

2018-11-26 13 interfaces 08 Returning NETDATA 02.401: [CLSINET] [4093638400]

2018-11-26 13 found node sxmms1 09 found node sxmms1 gipcdMonitorCssCheck 41.659: [GIPCDMON] [4093638400]

2018-11-26 13 updating timeout node sxmms1 09 updating timeout node sxmms1 gipcdMonitorCssCheck 41.659: [GIPCDMON] [4093638400]

2018-11-26 13 found node sxmms2 09 found node sxmms2 gipcdMonitorCssCheck 41.659: [GIPCDMON] [4093638400]

2018-11-26 13 skipping live node 09 sxmms1', time 41.659: [GIPCDMON] [4093638400] gipcdMonitorFailZombieNodes: skipping live node 'sxmms1', time 0 ms, endp 00000000000000, 00000000470e5359

2018-11-26 13 gipchaLowerDropMsg: dropping because of sequence timeout, waited 30050, msg 0x7f00dc0a8478 {len 1160, seq 2, type gipchaHdrTypeRecvEstablish (5), lastSeq 0, lastAck 0, minAck 1, flags 0x1, srcLuid bcd67f68-d3658d20, dstLuid 00000000-00000000, msgId 1}, node 0x7f00d02332c0 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-fe7418b0, dstLuid bcd67f68-d3658d20 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [30: 30], createTime 737588686, sentRegister 1, localMonitor 1, flags 0x0}

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 09 endp 0000000011befdfb 42.350: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 0000000011befdfb

2018-11-26 13 interfaces 09 interfaces 42.427: [CLSINET] [4093638400]

2018-11-26 13 eth3',ip='192.168.0.2',mac='a0 09Partition 42.427: [CLSINET] [4093638400] # 0 Interface 'eth3',ip='192.168.0.2',mac='a0-36-9f Mustang 5dPUBE 66 Mustang eccentricities massively "255.255.255.0" cognac" 192.168.0.0"

2018-11-26 13 gipchaLowerDropMsg: dropping because of sequence timeout, waited 30060, msg 0x7f00dc0c71b8 {len 1160, seq 3, type gipchaHdrTypeRecvEstablish (5), lastSeq 0, lastAck 0, minAck 2, flags 0x1, srcLuid bcd67f68-d3658d20, dstLuid 00000000-00000000, msgId 2}, node 0x7f00d02332c0 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-fe7418b0, dstLuid bcd67f68-d3658d20 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [31: 31], createTime 737588686, sentRegister 1, localMonitor 1, flags 0x0}

2018-11-26 13 gipchaLowerDropMsg: dropping because of sequence timeout, waited 30060, msg 0x7f00dc038cc8 {len 1160, seq 3, type gipchaHdrTypeRecvEstablish (5), lastSeq 0, lastAck 0, minAck 2, flags 0x1, srcLuid bcd67f68-d3658d20, dstLuid 00000000-00000000, msgId 2}, node 0x7f00d02332c0 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-fe7418b0, dstLuid bcd67f68-d3658d20 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [31: 31], createTime 737588686, sentRegister 1, localMonitor 1, flags 0x0}

2018-11-26 13 gipchaLowerDropMsg: dropping because of sequence timeout, waited 30060, msg 0x7f00dc04f288 {len 1160, seq 3, type gipchaHdrTypeRecvEstablish (5), lastSeq 0, lastAck 0, minAck 2, flags 0x1, srcLuid bcd67f68-d3658d20, dstLuid 00000000-00000000, msgId 2}, node 0x7f00d02332c0 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-fe7418b0, dstLuid bcd67f68-d3658d20 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [31: 31], createTime 737588686, sentRegister 1, localMonitor 1, flags 0x0}

2018-11-26 13 gipchaLowerProcessNode: no valid interfaces found to node for 737619746 ms, node 0x7f00d02332c0 {host 'sxmms1', haName' gipcd_ha_name', srcLuid 3de74e37-fe7418b0, dstLuid bcd67f68-d3658d20 numInf 1, contigSeq 0, lastAck 0, lastValidAck 0, sendSeq [31: 31], createTime 737588686, sentRegister 1, localMonitor 1, flags 0x4}

2018-11-26 13 req from local client of type gipcdmsgtypeInterfaceMetrics 09VR 42.708: [GIPCDCLT] [4166940416] gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceMetrics, endp 00000000000355

two。 Conclusion

According to the above process analysis, the reason why the 1-node CRS cannot be started is that the 2-node gipc process is abnormal, which leads to the normal connection between the two nodes. Gipc is an important part of RAC. From 11gR2 (11.2.0.2), oracle decided that the private network card should be managed by the cluster itself. The new cluster feature gipc (Grid IPC) is introduced, which exists in the cluster in the form of daemon gipcd.bin. The main functions are as follows:

1. When the cluster is started, the private network card of the cluster is found, and the information of the private network of the cluster is obtained from gpnp profile. And check the found private network interface.

two。 Use the previously discovered private network cards to discover other nodes in the cluster and establish contact with the private network cards of other nodes

3. If the cluster is configured with multiple private network cards, when there is a problem with one or more private network cards of a node, the private network with the problem is offline and other nodes are notified.

After confirming the role of gipcd.bin, the reason why the 1-node CRS cannot be started has been found. The connection of the cluster private network is achieved through this process, but the 2-node gipc process is in an abnormal state, so the 1-node cannot join the cluster after many restarts.

3. Solution

From the above analysis, it is determined that the 1-node CRS caused by the abnormal 2-node gipc process cannot start normally. Although gipc is used for private network connection, its own restart will not cause the cluster exception, so through the manual kill-9 gipcd.bin process, the gipc process will start automatically, and the 1-node will start successfully.

Thank you for your reading, the above is the content of "how to solve the problem that a node crs can not start due to gipc in oracle". After the study of this article, I believe you have a deeper understanding of how to solve the problem that a node crs cannot start due to gipc in oracle, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.