Question: Can you explain what a RAC private
network does? Why is it important to have the private network
tuned properly in a RAC cluster interconnect?
Answer: The
Cluster Interconnect is also referred to as the
private
network. This is a private
interconnect and the only traffic on this network should be for
cluster operations, including Cache Fusion. There is a misconception
that the private network is used only for the cluster heartbeat, a
pulse sent from one node to another to signify that the node is
alive. A heartbeat is a lightweight operation on a network so why
devote a network switch only usable for the private network?
This
incorrect line of thinking leads to some Oracle RAC deployments that
combine the public and private networks on the same switch.
Other RAC deployments may
separate the public and private networks, but put the private
network on a switch that is used for other network purposes. Each of
these configuration options is a mistake that many people later
regret when Oracle RAC performance suffers. The private network
needs to remain private, solely and exclusively for the use of the
Cluster Interconnect.
Chapter 2
has already shown how the private network is used for so much more
than just a cluster heartbeat, and it should be clear that the
private network is used for any cross-instance communications. When
two sessions on different instances want to modify the same block,
Global Enqueue Services 'talk? across the Cluster Interconnect.
Global cache transfers of data blocks will dominate the
private network traffic. Examine these metrics from an AWR report.
Global
Cache Load Profile
~~~~~~~~~~~~~~~~~~~~~~~~~
Per Second
Per Transaction
---------------
---------------
Global Cache blocks received:
1,279.68
79.62
Global Cache blocks served:
637.30
39.65
GCS/GES messages received:
4,367.95
271.76
GCS/GES messages sent:
7,440.72
462.94
DBWR Fusion writes:
21.15
1.32
Estd
Interconnect traffic (KB)
17,642.23
The metrics
above were taken from one instance of a three-node RAC database over
a one-hour period of time. The metrics above show an average of
1,916.98 global cache blocks sent and received per second. The
database block size is 8 Kbytes, which means there are 15,335.84
kilobytes of traffic each second on the private network just to
perform global cache transfers. The total private network traffic is
17,642.23 kilobytes is for all cross-instance traffic including the
global cache transfers.
The total
Cluster Interconnect traffic for this one-hour snapshot is about
17.6 megabytes. These metrics prove that the Cluster Interconnect is
used for so much more than just heartbeat traffic. These metrics
should also reinforce the notion that the private network needs to
remain private, as it can be busy. It is not uncommon to see some
Oracle RAC deployments that have many gigabytes per hour of traffic
on the Cluster Interconnect.
On the
Oracle RAC nodes, it is very easy to determine which interfaces
support the public and private networks for the cluster. The Oracle
Interface Configuration Tool (oifcfg)
can provide that information.
[root@host02 oracle]#
cd /u01/app/crs12.1.0.1/bin
[root@host02 bin]#
./oifcfg
getif
eth0
192.168.56.0 global
public
eth1
192.168.10.0 global
cluster_interconnect
From the
output above, we can see that device
eth0 is for the public
network and eth1 is for
the Cluster Interconnect. It is a requirement that the device names
be the same on all nodes so if we execute the same commands on
another node, the output should be the same. If another node has a
different interface, cluster communications will not work correctly.
It is also a requirement that the public and private interfaces be
on different subnets.
|
|
 |
|
Learn RAC Tuning
Internals!
This is an excerpt from the landmark book
Oracle RAC Performance tuning,
a book that provides real world advice for resolving
the most difficult RAC performance and tuning issues.
Buy it
for 30% off directly from the publisher.
|
|
| |
|
Burleson is the American Team
Note:
This Oracle
documentation was created as a support and Oracle training reference for use by our
DBA performance tuning consulting professionals.
Feel free to ask questions on our
Oracle forum.
Verify
experience!
Anyone
considering using the services of an Oracle support expert should
independently investigate their credentials and experience, and not rely on
advertisements and self-proclaimed expertise. All legitimate Oracle experts
publish
their Oracle
qualifications.
Errata?
Oracle technology is changing and we
strive to update our BC Oracle support information. If you find an error
or have a suggestion for improving our content, we would appreciate your
feedback. Just
e-mail:
 and include the URL for the page.
Copyright © 1996 - 2020
All rights reserved by
Burleson
Oracle ®
is the registered trademark of Oracle Corporation.
|
|
