Oracle 11g ASM Fast Rebalance

If there has ever been the need to remove a disk from an ASM disk group or add, respectively re-add a disk, it should be known that the rebalancing which occurs after such an action can be significant and this should not be done unless absolutely needed. Especially in a clustered ASM configuration, the rebalancing can cause the sending of lots of extent map lock and extent map unlock messages between ASM instances.

In order to prevent this, it is possible with Oracle ASM 11g where the ALTER DISKGROUP <dg_name> MOUNT command has been enhanced to mount ASM disk groups in restricted mode. If a diskgroup is mounted in restricted mode by one ASM instance in a cluster, no other instance can mount the ASM disk group and access any files there. This eliminates the overhead of sending locking messages, which improves the performance of the rebalancing operation.

An ASM disk group which is mounted in restricted mode on one node of the cluster can be maintained fully like a disk group which is mounted in non-restricted mode, noting that now the maintenance procedure can be finished much faster. After the maintenance operation is finished, it is necessary to unmount the disk group explicitly and then re-mount it in normal mode.

The below code listing shows the commands:

SYS AS SYSDBA @ +ASM SQL> ALTER DISKGROUP data DISMOUNT;
SYS AS SYSDBA @ +ASM SQL> ALTER DISKGROUP data MOUNT RESTRICTED; 

- - Perform maintenance now

SYS AS SYSDBA @ +ASM SQL> ALTER DISKGROUP data DISMOUNT;
SYS AS SYSDBA @ +ASM SQL> ALTER DISKGROUP data MOUNT;

It is also possible to start up an ASM in restricted session mode as the first ASM instance in a cluster. This causes all disk groups to be mounted by this particular instance in restricted mode for maintenance:

SYS AS SYSDBA @ +ASM SQL> STARTUP RESTRICT

It is not allowed to mount disk groups in restricted mode while the ASM cluster is performing a rolling upgrade operation.

Stretched ASM cluster configuration with ASM Preferred Mirror Read in 11g

Preferred mirror read is a feature which is particularly relevant for extended cluster environments. Such configurations are also called stretched clusters. They are typically used to mirror entire data centers across distances longer than 6 miles. Assume that there are data centers which are located 40 miles distant from each other and the goal is to mirror the two sites. For this, use ASM shared storage for an Oracle cluster which has member nodes with active instances in each site that is to be mirrored. For the ASM storage, define one ASM disk group with multiple failure groups.

In 10g, an instance in an Oracle Real Application Cluster must always use the primary copy of an extent in order to read it, even if the secondary copy is located in the local fail group and the primary copy is on the remote site. This has the negative effect that the primary copy of an extent needs to be read from the remote disks into the other instance and would be shipped through the interconnect to the requesting instance.

This problem is addressed in 11g with the preferred mirror read feature. It is possible now to define preferred read fail groups within an ASM disk group for each instance in a clustered ASM configuration, which allows for a preferred read of the local to the instance copy of an extent no matter if it is the primary or the secondary copy. This configuration is shown in the below graphic.

Figure 6:   Stretched Oracle 10g Cluster with Two ASM Fail Groups

The above graphic shows the situation in a 10g extended cluster configuration with two mirrored sites which both use the same ASM diskgroup with normal redundancy and two fail groups. Each fail group holds either the primary or the secondary copy of every ASM extent.

Figure 7:  Stretched Oracle 11g Cluster with Two ASM Fail Groups and Preferred Mirror Read

The preferred fail group can be configured to use for an instance in a cluster by adjusting the ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter for ASM instances. This parameter is dynamically changeable with need for restart with an ALTER SYSTEM statement. It is not valid for DB instances - only for ASM instances.

% It is important to make sure that the failure groups specified in the           ASM_PREFERRED_FAILURE_GROUPS parameter contain only disks that           are local to the corresponding ASM instance.

There is a new column in v$asm_disk which shows the character Y if a disk belongs to a preferred read fail group for the ASM instance:

SELECT preferred_read FROM v$asm_disk;

The view v$asm_disk_iostat can be queried from both an ASM instance as well as a DB instance for the disk I/O statistics. Queried on a DB instance, it only shows the statistic for this ASM client. If it is queried from an ASM instance, information will be shown for the I/O behavior of each ASM client.

SELECT * FROM v$asm_disk_iostat;

Now look at how to setup preferred mirror read. Assume that there are two sites which need to be mirrored with ASM using normal redundancy with two failure groups and preferred mirror read. Here are the pfiles of the two ASM instances.

init+ASM1.ora on site1:

instance_type='asm'
cluster_database=true
asm_diskgroups='DATA', 'FRA'
+ASM2.instance_number=2
+ASM1.instance_number=1
+ASM1.ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITE1
. . .

init+ASM2.ora on site2:

instance_type='asm'
cluster_database=true
asm_diskgroups='DATA', 'FRA'
+ASM2.instance_number=2
+ASM1.instance_number=1
+ASM2.ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITE2
. . .