Cluster failed to start due to ASM password file Issue (ORA-01017: invalid username/password)

Node server2 got evicted from the Cluster due to NIC hardware failure

oracle@server2 /home/oracle
06:19:25 : olsnodes -n
server1 1
server2 2
server3 3

DB Alert Log:
/u01/oracle/product/diag/crs/server3/crs/trace/alert.log
String containing event: 2022-07-09 04:06:51.055 [OCSSD(62652)]CRS-1612: Network communication with node server2 (2) has been missing for 50% of the timeout interval. If this persists, removal of this node from cluster will occur in 14.490 seconds                                                                                             2022-07-09 04:12:50.846 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:53.204 [CLSECHO(28043)]ACFS-9391: Checking for existing ADVM/ACFS installation. 2022-07-09 04:12:53.283 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]].
2022-07-09 04:12:53.356 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:57.874 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]].
2022-07-09 04:12:59.202 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]]. 2022-07-09 04:12:59.274 [GIPCD(27120)]CRS-42216: No interfaces are configured on the local node for interface definition bond1(:.)?:30.2.56.0: available interface definitions are [eno1(:.)?:10.159.241.64][bond0(:.)?:170.71.56.0][bond0(:.)?:[fe80:0:0:0:0:0:0:0]][eno1(:.)?:[fe80:0:0:0:0:0:0:0]].

CRS Alert Log:
/u01/oracle/product/diag/crs/server2/crs/trace/crsd.trc
OCRASM:1209646848: SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge
2022-07-09 06:50:33.016 : OCRASM:1209646848: ASM Error Stack : ORA-01017: invalid username/password; logon denied
2022-07-09 06:50:33.016 : OCRASM:1209646848: proprasmo: kgfoCheckMount returned [7]
2022-07-09 06:50:33.016 : OCRASM:1209646848: proprasmo: The ASM instance is down
2022-07-09 06:50:33.017 : OCRRAW:1209646848: proprioo: Failed to open [+GRID_DG/RR-stand/OCRFILE/REGISTRY.255.913557553]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2022-07-09 06:50:33.017 : OCRRAW:1209646848: proprioo: No OCR/OLR devices are usable
OCRUTL:1209646848: u_fill_errorbuf: Error Info : [Insufficient quorum to open OCR devices]
default:1209646848: u_set_gbl_comp_error: comptype '107' : error '0'
2022-07-09 06:50:33.017 : default:1209646848: clsvactversion:4: Retrieving Active Version from local storage.
2022-07-09 06:50:33.019 : OCRRAW:1209646848: proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.

We tried to restart crs on server2 , but it didn't worked.
/u01/grid/oracle/product/19/grid/bin/crsctl stop crs
/u01/grid/oracle/product/19/grid/bin/crsctl start crs

root@server2 ~
08:33 AM # /u01/grid/oracle/product/19/grid/bin/crsctl check crs 
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Compared the asm password file on all the nodes

oracle@server1 /u01/grid/oracle/product/19/grid/dbs
09:38:56 +ASM1: ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 old_orapw+ASM_old
-rw-r-----. 1 oracle oinstall 2048 Jun 12 02:40 orapw+ASM1

oracle@server2 /u01/grid/oracle/product/19/grid/dbs
09:39:36 : ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 orapw+ASM

oracle@server3 /u01/grid/oracle/product/19/grid/dbs
09:39:54 : ls -ltr orapw
-rw-r-----. 1 oracle oinstall 5120 Jul 8 2020 orapw+ASM

SOLUTION :

Copied /u01/grid/oracle/product/19/grid/dbs/orapw+ASM from node3 to node 1

oracle@server1 /home/oracle
09:43:16 +ASM1: asmcmd
ASMCMD> pwget --asm
+GRID_DG/orapwasm
ASMCMD> pwcopy --asm -f /u01/grid/oracle/product/19/grid/dbs/orapw+ASM +GRID_DG/orapwasm
copying /u01/grid/oracle/product/19/grid/dbs/orapw+ASM -> +GRID_DG/orapwasm
ASMCMD> exit

Once we copied the ASM password file into respective Diskgroup , CRS Services on node 2 came up automatically.

MOS note:
12c GI: resource ora.storage can't started due to wrong ASM password location information (Doc ID 2264675.1)