Cluster failed to start due to ASM password file Issue (ORA-01017: invalid username/password)
3 min readNode server2 got evicted from the Cluster due to NIC hardware failure
oracle@server2 /home/oracle
06:19:25 : olsnodes -n
server1 1
server2 2
server3 3
DB Alert Log: /u01/oracle/product/diag/crs/server3/crs/trace/alert.log String containing event: 2022-07-09 04:06:51.055 [OCSSD(62652)]CRS-1612: Network communication with node server2 (2) has been missing for 50% of the timeout interval. If this persists, removal of this node from cluster will occur in 14.490 seconds 2022-07-09 04:12:50.846 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:53.204 [CLSECHO(28043)]ACFS-9391: Checking for existing ADVM/ACFS installation. 2022-07-09 04:12:53.283 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:53.356 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:57.874 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:59.202 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:59.274 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]].
CRS Alert Log: /u01/oracle/product/diag/crs/server2/crs/trace/crsd.trc OCRASM:1209646848: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge 2022-07-09 06:50:33.016 : OCRASM:1209646848: ASM Error Stack : ORA-01017: invalid username/password; logon denied 2022-07-09 06:50:33.016 : OCRASM:1209646848: proprasmo: kgfoCheckMount returned [7] 2022-07-09 06:50:33.016 : OCRASM:1209646848: proprasmo: The ASM instance is down 2022-07-09 06:50:33.017 : OCRRAW:1209646848: proprioo: Failed to open [+GRID_DG/RR-stand/OCRFILE/REGISTRY.255.913557553]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE. 2022-07-09 06:50:33.017 : OCRRAW:1209646848: proprioo: No OCR/OLR devices are usable OCRUTL:1209646848: u_fill_errorbuf: Error Info : [Insufficient quorum to open OCR devices] default:1209646848: u_set_gbl_comp_error: comptype '107' : error '0' 2022-07-09 06:50:33.017 : default:1209646848: clsvactversion:4: Retrieving Active Version from local storage. 2022-07-09 06:50:33.019 : OCRRAW:1209646848: proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
We tried to restart crs on server2 , but it didn't worked. /u01/grid/oracle/product/19/grid/bin/crsctl stop crs /u01/grid/oracle/product/19/grid/bin/crsctl start crs
root@server2 ~ 08:33 AM # /u01/grid/oracle/product/19/grid/bin/crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online
Compared the asm password file on all the nodes
oracle@server1 /u01/grid/oracle/product/19/grid/dbs
09:38:56 +ASM1: ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 old_orapw+ASM_old
-rw-r-----. 1 oracle oinstall 2048 Jun 12 02:40 orapw+ASM1
oracle@server2 /u01/grid/oracle/product/19/grid/dbs
09:39:36 : ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 orapw+ASM
oracle@server3 /u01/grid/oracle/product/19/grid/dbs
09:39:54 : ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 orapw+ASM
SOLUTION : Copied /u01/grid/oracle/product/19/grid/dbs/orapw+ASM from node3 to node 1
oracle@server1 /home/oracle
09:43:16 +ASM1: asmcmd
ASMCMD> pwget --asm
+GRID_DG/orapwasm
ASMCMD> pwcopy --asm -f /u01/grid/oracle/product/19/grid/dbs/orapw+ASM +GRID_DG/orapwasm
copying /u01/grid/oracle/product/19/grid/dbs/orapw+ASM -> +GRID_DG/orapwasm
ASMCMD> exit
Once we copied the ASM password file into respective Diskgroup , CRS Services on node 2 came up automatically.
MOS note:
12c GI: resource ora.storage can't started due to wrong ASM password location information (Doc ID 2264675.1)
