The cluster volume manager is a software module
that virtualizes the physical storage and presents the usable
entities for the applications, such as the database at the operating
system level. It is a storage administrator layer at the host level.
The Veritas cluster volume manager (CVM) uses
objects for storage management. The two types of objects used by
Volume Manager are physical objects and virtual objects.
* Physical objects ? Volume Manager uses two
physical objects: physical disks and partitions. Partitions are
created on the physical disks (on systems that use partitions).
* Virtual objects - Volume Manager creates
virtual objects, called volumes. Each volume records and retrieves
data from one or more physical disks. A volume is a virtual disk
device that appears to applications, databases, and file systems as
a physical disk. Volumes are also composed of other virtual objects
such as sub-disks and plexes. These are used to change volume
configuration features such as concatenation, mirroring, and
striping. Volumes and their virtual components are called virtual
Volume management involves using physical disks
or LUNS to create disk groups. These disks are combined to create
volumes with a layout or configuration appropriate to their purpose,
for example, mirroring or striping etc. The volumes are flexible
objects that can easily be extended or modified. The volume
management operations include:
* Placing the shared disks or LUNS from the
shared storage array into the Volume Manager control. When the LUNS
are placed under the VM control, they are known as VM disks.
* Creating the Disk Groups ? the disk groups
that contain the database application must be available to all nodes
in the cluster. A disk group allows disks, volumes, and file systems
that are relevant to a single database to be arranged into a logical
collection for easy administration.
* Creating volumes with the appropriate layout
(mirror, stripe. etc.) and size from the disk group storage pool.
Volumes are flexible objects that can be extended or modified.
* Making file systems from the volumes and
mounting them for database use. Volumes can also be used for
database applications, in which case they are referred to as raw
It is important to understand that volumes are
the basic storage components that an Oracle database utilizes. The
volumes, as created by the CVM, are accessible by all members or
nodes of the cluster. This is an important distinction in the RAC
environment. A volume can be accessed either directly by the RAC
database as a raw partition, or through mounted file systems such as
a CFS, as shown in Figure 15.16.
Figure 5.16: Veritas Cluster Volume Management
Configure NFS file system
Using NFS mount for the shared storage for the
RAC database system, provides yet another option. In this section,
the usage of NFS type shared storage structures will be covered by
looking at the Filer System supplied by Network Appliance System.
* One Network Appliance F2XX/F7XX/F8XX/F9XX
filer with Data ONTAP? 6.1.3R2 or later
* One Gigabit switch with at least four ports
* One Gigabit NIC in the filer
* One or more disk shelves, based on the disk
Once the hardware connection is made to the
nodes in the Linux Cluster, install Data ONTAP 6.1.3 R2 and give it
a name, such as FDATA1, and an IP Address for the filer storage
Create a volume on the FDATA1 filer for storing
Oracle Database files. This volume will be used to store all the
data files, control files, and log files for the Oracle Database.
Use the at the filer console:
FDATA1> vol create oradata 14
Then Edit the /etc/exports file on FData1 and
add the following entries to that file:
Execute the following command at the filer
FData1> exportfs ?a
Now create mount points and mount volumes.
Update the /etc/fstab file on all the server nodes and add the
FData1:/vol/racdata - /oradb nfs ? Yes rw ,
bg , hard , nointr , rsize=32768,wsize=32768,tcp,noac,vers=3,timeo=600
* FData1 is the name of the filer.
* oradb is the mount point on the cluster
* The mount options that are required for the
Oracle RAC are:
* noac: This mount option disables caching on
the client side.
* tcp: Mount the file system using the tcp
Veritas Cluster Volume Management
The Veritas Volume Management (VxVM) tool manages physical
disks and presents logical disks for application use. Logical
disks are known as volumes. A volume is the disk space that
appears to the database as a physical partition. VxVM operates as
a sub-system between the operating system and the database system.
We need to create one or more shared disk groups besides the
rootdg disk group. rootdg must exist and cannot be shared between
the systems. At least one disk must be in the rootdg disk group.
The shared disk group is accessible for all the nodes in the
The cluster functionality of a VxVM works together with the
cluster manager daemon. The CM informs VxVM of the changes in
cluster membership based on which access is controlled. When a
node joins a cluster it gains access to shared disks. When a node
leaves a cluster it no longer has access to shared disks. A node
joins a cluster when the cluster manager is started on that node.
To the cluster manager, all nodes are the same. All nodes that
join the cluster can potentially access VxVM objects configured
within shared disk groups. However, the cluster functionality of
VxVM requires that one node act as the master node. All other
nodes in the cluster are slave nodes. Any node is capable of being
the master node and it is responsible for coordinating certain
We must run all the commands that configure or reconfigure VxVM
objects on the master node. Tasks that must be initiated from the
master node include setting up shared disk groups, creating and
reconfiguring volumes, and performing snapshot operations. VxVM
determines that the first node to join a cluster performs the
function of master node. If the master node leaves a cluster, one
of the slave nodes is chosen to be the new master.
There are three interfaces to use with the volume manager:
Volume Manager Storage Administrator (VMSA) is a graphical
user interface to Volume Manager.
Command Line Interface (CLI): The command line interface
(CLI) consists of UNIX utilities that are invoked from the
command line to perform Volume Manager and standard UNIX
tasks. You can use the CLI not only to manipulate Volume
Manager objects, but also to perform scripting and debugging
Volume Manager Support Operations (vxdiskadm): The Volume
Manager Support Operations interface, commonly known as
vxdiskadm, is a menu-driven, text-based interface that you can
use for disk and disk group administration functions.
Placing the physical disks into the group creates a disk group
and then individual volumes can be created for the various Oracle
files. Let us look at some examples.
To create a disk group called oradg over 4 LUNs:
# vxdg -s init oradg1 ora01=c1t1d0s2
ora02=c1t2d0s2 ora03=c1t3d0s2 ora04=c1t4d0s2
To create a RAID 0+1 (mirror/stripe) volume across 4 disk LUNs:
# vxassist -g oradg1 make
rac_system_500m 500m layout=stripe stripeunit=256 ncolumn=4
user=oracle group=dba alloc="ora01 ora02 ora03 ora04"