Call now: 252-767-6166  
Oracle Training Oracle Support Development Oracle Apps

 E-mail Us
 Oracle Articles
New Oracle Articles

 Oracle Training
 Oracle Tips

 Oracle Forum
 Class Catalog

 Remote DBA
 Oracle Tuning
 Emergency 911
 RAC Support
 Apps Support
 Oracle Support

 SQL Tuning

 Oracle UNIX
 Oracle Linux
 Remote s
 Remote plans
 Application Server

 Oracle Forms
 Oracle Portal
 App Upgrades
 SQL Server
 Oracle Concepts
 Software Support

 Remote S


 Consulting Staff
 Consulting Prices
 Help Wanted!


 Oracle Posters
 Oracle Books

 Oracle Scripts

Don Burleson Blog 









Cluster Interconnect

Oracle RAC Cluster Tips by Burleson Consulting

This is an excerpt from the bestselling book Oracle Grid & Real Application Clusters.  To get immediate access to the code depot of working RAC scripts, buy it directly from the publisher and save more than 30%.

The cluster interconnect is a private network which is used to transfer cluster traffic, Oracle Resource directory information, and blocks for satisfying queries. There are two components involved in setting up and configuring the cluster interconnect, the physical hardware and the interconnect software. The physical connection between the nodes may be through a simple crossover cable or via a switch or hub.

There is a wide variety of technology. Fast Ethernet and Gigabit Ethernet connectivity provide high bandwidth and are widely used. However, such general purpose interconnect technology may not offer acceptable latency. To that end, special interconnect products like Memory Channel for HP Tru64 (TruCuster), Hyper Fabric Interconnect for HP-UX clusters, and SCI adapters for SunPlex Clusters should be preferred wherever possible.

In addition, there is Infiniband interconnect architecture for Linux based systems, which is being provided by vendors such as Mellanox and TopSpin. TopSpin Clustering kit provides Infiniband switch multiple ports, Fiber Channel Gateway for SAN storage or Ethernet gateway for NAS storage, 10 Gbps Infiniband Host Channel Bus Adapters.

In another example, SunPlex Cluster systems offers support for Remote Shared Memory technology (RSM), which enhances the performance of distributed applications by allowing for fast messaging between instances of a distributed application. RSM improves bandwidth and reduces the latency many-fold, allowing a SunPlex/Oracle RAC deployment to deliver superior performance.

The RSM API provides mechanisms upon which user applications, such as Oracle Real Application Clusters, can build low-latency, high bandwidth messaging to directly access interconnect hardware, bypassing the operating system. Utilizing this feature requires SCI-PCI interconnect in the Sun Cluster 3.0 environment. The following Table 6.1 shows the protocols used and the throughput achieved.





Memory Channel



100 MB/sec

Fast Ethernet



10 MB/sec

Gigabit Ethernet



UDP 100 MB/sec

Hyper Fabric



400 MB/sec




70 MB/sec

Table 6.1: Interconnect Types

If more than two nodes are interconnected in a cluster, they must be linked with a switch. Wherever possible, redundant interconnects must be provided by using dual switches. The following commands show which interconnect is used for UDP or TCP:

sqlplus> connect / as sysdba

oradebug setmypid

oradebug ipc


The corresponding trace can be found in the user_dump_dest directory, and contains the following information in the last couple of lines:

SKGXPCTX: 0x32911a8 ctx

admno 0x12f7150d admport:


    info for network 0

  socket no 9 IP UDP 43307


  info for network 1

  socket no 0 IP UDP 0


context timestamp 0x1ca5

  no ports

When other protocols such as HMP or RDG are used, the trace file will not reveal the IP address.

Another method of finding the protocol is to look into the alert log file of the Oracle instance. During startup, a message is posted showing the IPC details. For example,

Wed Oct 30 05:28:55 2002

cluster interconnect IPC version: Oracle UDP/IP with Sun RSM disabled

IPC Vendor 1 proto 2 Version 1.0

Message Size

The message size is platform dependent. In Oracle9i Release 2 (, message size was limited to 32K. Before, the Operating System UDP settings are overwritten and a 32k buffer is used.

Oracle9i allows larger message sizes, depending on the platform. To increase throughput on an interconnect, the UDP kernel parameters must be adjusted.

Oracle opens the UDP receive ports with a receive buffer size of 128k, but it also honors O/S settings and can even adjust the size based on the maximum block size seen, up to the O/S maximum. The operating system maximum for the UDP receive buffer size is:

Linux:                              256k

Solaris / Tru64 5.1:       1MB

HP:                                 2GB

When implementing the Sun Cluster with the clusterware provided by Veritas, the LMX message size is up to 64K, with a default of 32K.


This is an excerpt from the bestselling book Oracle Grid & Real Application Clusters, Rampant TechPress, by Mike Ault and Madhu Tumma.

You can buy it direct from the publisher for 30%-off and get instant access to the code depot of Oracle tuning scripts.


Oracle Training at Sea
oracle dba poster

Follow us on Twitter 
Oracle performance tuning software 
Oracle Linux poster


Burleson is the American Team

Note: This Oracle documentation was created as a support and Oracle training reference for use by our DBA performance tuning consulting professionals.  Feel free to ask questions on our Oracle forum.

Verify experience! Anyone considering using the services of an Oracle support expert should independently investigate their credentials and experience, and not rely on advertisements and self-proclaimed expertise. All legitimate Oracle experts publish their Oracle qualifications.

Errata?  Oracle technology is changing and we strive to update our BC Oracle support information.  If you find an error or have a suggestion for improving our content, we would appreciate your feedback.  Just  e-mail:  

and include the URL for the page.


Burleson Consulting

The Oracle of Database Support

Oracle Performance Tuning

Remote DBA Services


Copyright © 1996 -  2017

All rights reserved by Burleson

Oracle ® is the registered trademark of Oracle Corporation.

Remote Emergency Support provided by Conversational