This is an excerpt from the bestselling book
Oracle Grid & Real Application Clusters. To get immediate
access to the code depot of working RAC scripts, buy it
directly from the publisher and save more than 30%.
The GCS maintains the status of
the resources. It also keeps an inventory of the access requests for
the data blocks. After the blocks are transferred from one instance
to another to meet requests, the requesting processes need to be
notified that the block is actually available. Therefore, processes
utilize interrupts to inform of the arrival or completion of block
transfers. The GCS uses various interrupts to manage resource
allocation. These interrupts are:
* Blocking Interrupt -
When exclusive access is needed for a requestor, the GCS sends a
blocking interrupt to a process that currently owns the shared
resource, notifying it that a request for an exclusive resource is
* Acquisition Interrupt -
When the requested access (e.g., exclusive) is made available after
releasing an earlier access mode, an acquisition interrupt is sent
to alert the process that has requested the exclusive resource. The
acquisition interrupt helps to notify the requesting process.
* Block Arrival Interrupt
- When a process requests a block from the GCS, the request is
forwarded to the instance holding the block. Then the requested
block is sent to the requesting process, and the process informs the
GCS that it has received the block. This notification is called
block arrival interrupt.
The block requests are granted
for many processes at the same time, but they follow a queuing
mechanism. The GCS maintains two types of queues for resource
If the GCS is unable to grant a
resource request immediately, then the GCS puts it in the convert
queue. The GCS then tracks all waiting requests.
Once a resource is granted to
the requesting process, it is kept in the granted queue. The GCS
tracks resource requests in the granted queue.
Cache Fusion and Recovery
In the RAC system, whenever
there is a node failure, the instance running on the failed node
crashes and becomes unusable. There can be several reasons for such
a failure. In this section, focus will be placed on the changes that
take place in the global cache and how the recovery of the failed
instance is undertaken by one of the surviving instances.
Only the cache resources that
reside on the failed nodes or are mastered by the GCS on the failed
nodes need to be re-built or re-mastered. Rebuilt or re-master does
not mean building a block; the lock ownership is merely changed and
this is explained later with examples.
All resources previously
mastered at the failed instance are redistributed across the
remaining instances. These resources are reconstructed at their new
master instance. All other resources previously mastered at
surviving instances remain unaffected.
The cluster manager first
detects the node and instance failure. It communicates the failure
status to the GCS by way of the LMON process. At this stage, any
surviving instance in the cluster initiates the recovery process.
Remember, instance recovery does
not include restarting the failed instance or recovering
applications that were running on that instance. Also note that,
even after a node failure and instance loss, the redo log file of
the failed instance is still available to the other recovering
instance, since the redo log file is located on the shared cluster
file system or shared raw partition. This is an important feature of
the RAC system.
Because of past images, instance
recovery is performed differently in the RAC implementation. The
SMON process of a surviving instance performs recovery of the failed
instance or thread. However, note that the foreground process
performs recovery in a stand-alone instance.
Recovery Methodology and
Oracle performs the following
steps to recover:
1. In the initial phase of
recovery, GES enqueues are reconfigured and the global resource
directory is frozen. All GCS resource requests and writes are
2. GCS resources are
reconfigured among the surviving instances. One of the surviving
instances becomes the recovering instance. The SMON process of the
recovering instance starts a first pass of the redo log read of the
failed instance's redo thread.
3. Block resources that need to
be recovered are identified and the global resource directory is
reconstructed. Pending requests or writes are cancelled or replayed.
4. Resources identified in the
previous log read phase are defined as recovery resources. Buffer
space for recovery is allocated.
5. Assuming that there are past
images of blocks to be recovered in other caches in the cluster,
source buffers are requested from other instances. The resource
buffers are the starting point of recovery for a particular block.
6. All resources and enqueues
required for subsequent processing have been acquired and the global
resource directory is now unfrozen. Any data blocks that are not in
recovery can now be accessed. At this time, the system is partially
7. The SMON merges the redo
thread order by SCN to ensure that changes are written in an orderly
fashion. This process is important for multiple simultaneous
failures. If multiple instances die simultaneously, neither the PI
buffers nor the current buffers for a data block can be found in any
surviving instance's cache. Then a log merger of the failed
instances is performed.
8. Now the second pass of
recovery begins and redo is applied to data files, releasing the
recovery resources immediately after block recovery, so that more
and more blocks become available as cache recovery proceeds.
9. After all blocks have been
recovered and recovery resources have been released, the system is
available for normal use.
Figure 7.7 shows the basic steps
in the recovery.
Figure 7.7: Online Instance