Voting Disk in Oracle RAC

Voting disks manage information about node membership. Each voting disk must be accessible by all nodes in the cluster for nodes to be members of the cluster

1. You can store OCR and voting disks on Oracle Automatic Storage Management (Oracle ASM)

2. If it is stored in ASM, no need to configure manually as the files will be created depending on the redundancy in ASM.

3. We must have odd number of disks.

4. Oracle recommends minimum of 3 and maximum of 5. 

In 10g, Clusterware can supports 32 voting disks but in 11gR2 supports 15 voting disks.

5. Once you configure voting disks on Oracle ASM, you can only make changes to the voting disks' configuration using the crsctl replace votedisk command. 

The number of voting files you can store in a particular Oracle ASM disk group depends upon the redundancy of the disk group.

If voting disks are stored on Oracle ASM with normal or high redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

Using the crsctl replace votedisk command, you can move a given set of voting disks from one Oracle ASM disk group into another

What Information is stored in VOTING DISK/FILE

Static data : Info about the nodes in cluster

Dynamic data: Disk heartbeat logging

It contains the important details of the cluster nodes membership like:

1) Which node is part of the cluster?
2) Which node is leaving the cluster?
3) Which node is joining the cluster?

To identify the status & voting disk location :

$ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group
 1. ONLINE   7c54856e98474f61bf349401e7c9fb95 (/dev/sdb1) [DATA]

If the voting disks are stored in Oracle ASM, then run the following command to migrate the voting disks to the Oracle ASM disk group you specify:

crsctl replace votedisk +asm_disk_group

If you did not store voting disks in Oracle ASM, then run the following command using the File Universal Identifier (FUID) obtained in the previous step:

$ crsctl delete css votedisk FUID

Add a voting disk, as follows:

$ crsctl add css votedisk path_to_voting_disk

Purpose of Voting disk:

Voting disks are used by Clusterware for health check.

As we know now that voting disks is used by CSSD. It contains both network & disk heartbeat from all nodes and if any break in heartbeat will result in eviction of the node from cluster. There are possible scenarios with missing heartbeats.

* Network heart beat is successful, but disk heart beat is missed.
* Disk heart beat is successful, but network heart beat is missed.
* Both heart beats failing.

When a cluster is involved with many nodes, then few more scenarios are possible.

Nodes have a split into N sets of nodes., communicating within the sets, but not with the members in other set.
Just one node going unhealthy. Nodes with quorum (minimum number of nodes to make cluster valid) will maintain active membership of the cluster and other node(s) will be fenced/rebooted.

What is the voting disk used for?

A voting disk is a backup communications mechanism that allows CSS daemons to negotiate which sub-cluster will survive. These voting disks keep a status of who is currently alive and counts votes in case of a cluster reconfiguration.

It works as follows:

a) Ensures that you cannot join the cluster if you cannot access the voting disk(s)

b) Leave the cluster if you cannot communicate with it (to ensure we do not have aberrant nodes)

c) Should multiple sub-clusters form, it will only allow one to continue. It prefers a greater number of nodes, and secondly the node with the lowest incarnation number.

What should the permissions be set to for the voting disk and ocr when doing an Oracle RAC Install?

The Oracle Real Application Clusters install guide is correct. It describes the PRE-INSTALL ownership/permission requirements for ocr and voting disk. 

This step is needed to make sure that the Oracle Clusterware install succeeds. Please don't use those values to determine what the ownership/permmission should be POST INSTALL. 

The root script will change the ownership/permission of ocr and voting disk as part of install. 

The POST INSTALL permissions will end up being : OCR - root:oinstall - 640 Voting Disk - oracle:oinstall - 644

Why should we have ODD number of voting disk ?

If you lose 1/2 or more of all of your voting disks, then nodes get evicted from the cluster, or nodes kick themselves out of the cluster. It doesn’t threaten database corruption.

Alternatively you can use external redundancy which means you are providing redundancy at the storage level using RAID. For this reason when using Oracle for the redundancy of your voting disks, Oracle recommends that customers use 3 or more voting disks in Oracle RAC

Note: For best availability, the 3 voting files should be physically separate disks. It is recommended to use an odd number as 4 disks will not be any more highly available than 3 disks, 1/2 of 3 is 1.5…rounded to 2, 1/2 of 4 is 2, once we lose 2 disks, our cluster will fail with both 4 voting disks or 3 voting disks.

Restoring corrupted voting disks is easy since there isn’t any significant persistent data stored in the voting disk.

Hope it helped !! 🙂