Raw Devices in Linux

This is an excerpt from the bestselling book Oracle Grid & Real Application Clusters.  To get immediate access to the code depot of working RAC scripts, buy it directly from the publisher and save more than 30%.

Shared disks are visible to all nodes in the cluster. RAC database requires the use of raw devices or cluster file system files or ASM resources.

Raw Device binding information is available in the file /etc/sysconfig/rawdevices.

For example, usually they are specified as:

# format: 
<rawdev> <major> <minor>

or

<rawdev> <blockdev>

For example:

/dev/raw/raw1 /dev/emcpowera5
/dev/raw/raw2 /dev/emcpowera6
/dev/raw/raw1 /dev/sda1
/dev/raw/raw2 8 5

In the above, the emcpowera5 is bound with /dev/raw/rwa1 and /dev/raw/raw2 is bound with /dev/emcpowera6. These are SAN (EMC symmetrix) power-devices.

Size of such raw devices can be viewed in the file /proc/partitions. For example,

$ cat /proc/partitions | more

 232     0    8923200 emcpowera 268 0 536 190 0 0 0 0 0 190 140
 232     1          1 emcpowera1 4 0 8 0 0 0 0 0 0 0 0
 232     5    1028097 emcpowera5 18 0 36 20 0 0 0 0 0 20 20
 232     6    1028128 emcpowera6 22 0 44 10 0 0 0 0 0 10 10
 232     7    1028128 emcpowera7 22 0 44 10 0 0 0 0 0 10 0

The third field/column shows the Size of the device in KB. Column(1) represents the major number and column(2) shows the minor number.

The actual size of partitions and the state of raw device bindings can also be determined using the following commands, respectively:

#fdisk ?l

and

#raw ?qa

In some systems, the logical volume manager is available to create the necessary disk partitions. When using raw devices, carefully partition the disks to insure partitions are sized adequately.  The LVM (Logical Volume Manager) is very useful and makes the management of raw devices more flexible.

Use pvcreate to create a physical volume for use by the logical volume manager.

$ pvcreate -d /dev/sda

For a single partition on a multi-partition drive, use the partition designator such as /dev/sda1.

Use vgcreate from a root session to create a volume group for the drive or for the partition that will be used for the raw.

$ vgcreate -l 256 -p 256 -s 128k /dev/sda

The above command allows 256 logical partitions and 256 physical partitions, with a 128K extent size.

Use lvcreate to create the logical volumes inside the volume group. An example script is shown below.

pvcreate -d /dev/sda
vgcreate -l 256 -p 256 -s 128k /dev/pv1 /dev/sda
lvcreate -L 500m /dev/pv1
lvcreate -L 500m /dev/pv1
lvcreate -L 300m /dev/pv1
lvcreate -L 100m /dev/pv1
?.

The above commands create /dev/pv1/lvol1 to lvoln.

Next, bind the volumes to the raw devices. This is accomplished through the /usr/bin/raw command.

vgchange -a y /dev/pv1
/usr/bin/raw /dev/raw/raw1 /dev/pv1/lvol1
/usr/bin/raw /dev/raw/raw2 /dev/pv1/lvol2
/usr/bin/raw /dev/raw/raw3 /dev/pv1/lvol3
/usr/bin/raw /dev/raw/raw4 /dev/pv1/lvol4
/usr/bin/raw /dev/raw/raw5 /dev/pv1/lvol5
/usr/bin/raw /dev/raw/raw6 /dev/pv1/lvol6
/usr/bin/raw /dev/raw/raw7 /dev/pv1/lvol7
/usr/bin/raw /dev/raw/raw8 /dev/pv1/lvol8
/usr/bin/raw /dev/raw/raw9 /dev/pv1/lvol9

Soft links can be created from the raw volumes to make file recognition easy.

Configure OCFS for Linux

In this section, how to create OCFS on the Linux platform will be covered. To verify disks/partitions use the command fdisk:

/sbin/fdisk /dev/sdd

Verify Partition Information

Command (m for help): p

Disk /dev/sdd: 255 heads, 63 sectors, 4443 cylinders

Units = cylinders of 16065 * 512 bytes

Device Boot       Start       End    Blocks   Id  System
/dev/sdd1             1      1530  12289693+  83  Linux
/dev/sdd2          1531      4443  23398672+   5  Extended
/dev/sdd5          1531      1913   3076416   83  Linux
/dev/sdd6          1914      2168   2048256   83  Linux
/dev/sdd7          2169      2806   5124703+  83  Linux

What is shown above is a relatively simple storage scenario. One disk device /dev/sdd, is divided into multiple partitions. Based on the above plan the OCFS mount points can be created as follows:

/dev/sdd1  --> /data/oracle/NYREP/u02       12289693+ (i.e. 12 gb)
/dev/sdd5  --> /data/oracle/NYREP/u01        3076416  (i.e.  3 gb)
/dev/sdd6  --> /data/oracle/NYREP/r01        2048256  (i.e.  2 gb)
/dev/sdd7  --> /data/oracle/NYREP/archive    5124703+ (i.e.  5 gb)