Call now: 252-767-6166  
Oracle Training Oracle Support Development Oracle Apps

 
 Home
 E-mail Us
 Oracle Articles
New Oracle Articles


 Oracle Training
 Oracle Tips

 Oracle Forum
 Class Catalog


 Remote DBA
 Oracle Tuning
 Emergency 911
 RAC Support
 Apps Support
 Analysis
 Design
 Implementation
 Oracle Support


 SQL Tuning
 Security

 Oracle UNIX
 Oracle Linux
 Monitoring
 Remote s
upport
 Remote plans
 Remote
services
 Application Server

 Applications
 Oracle Forms
 Oracle Portal
 App Upgrades
 SQL Server
 Oracle Concepts
 Software Support

 Remote S
upport  
 Development  

 Implementation


 Consulting Staff
 Consulting Prices
 Help Wanted!

 


 Oracle Posters
 Oracle Books

 Oracle Scripts
 Ion
 Excel-DB  

Don Burleson Blog 


 

 

 


 

 

 

 

 

Cluster Interconnect

Oracle RAC Cluster Tips by Burleson Consulting

This is an excerpt from the bestselling book Oracle Grid & Real Application Clusters.  To get immediate access to the code depot of working RAC scripts, buy it directly from the publisher and save more than 30%.


The cluster interconnect is a private network which is used to transfer cluster traffic, Oracle Resource directory information, and blocks for satisfying queries. There are two components involved in setting up and configuring the cluster interconnect, the physical hardware and the interconnect software. The physical connection between the nodes may be through a simple crossover cable or via a switch or hub.

There is a wide variety of technology. Fast Ethernet and Gigabit Ethernet connectivity provide high bandwidth and are widely used. However, such general purpose interconnect technology may not offer acceptable latency. To that end, special interconnect products like Memory Channel for HP Tru64 (TruCuster), Hyper Fabric Interconnect for HP-UX clusters, and SCI adapters for SunPlex Clusters should be preferred wherever possible.

In addition, there is Infiniband interconnect architecture for Linux based systems, which is being provided by vendors such as Mellanox and TopSpin. TopSpin Clustering kit provides Infiniband switch multiple ports, Fiber Channel Gateway for SAN storage or Ethernet gateway for NAS storage, 10 Gbps Infiniband Host Channel Bus Adapters.

In another example, SunPlex Cluster systems offers support for Remote Shared Memory technology (RSM), which enhances the performance of distributed applications by allowing for fast messaging between instances of a distributed application. RSM improves bandwidth and reduces the latency many-fold, allowing a SunPlex/Oracle RAC deployment to deliver superior performance.

The RSM API provides mechanisms upon which user applications, such as Oracle Real Application Clusters, can build low-latency, high bandwidth messaging to directly access interconnect hardware, bypassing the operating system. Utilizing this feature requires SCI-PCI interconnect in the Sun Cluster 3.0 environment. The following Table 6.1 shows the protocols used and the throughput achieved.

INTERCONNECT

LATENCY (?s)

PROTOCOL

THROUGHPUT

Memory Channel

3

3 RDG

100 MB/sec

Fast Ethernet

100

UDP

10 MB/sec

Gigabit Ethernet

100

UDP

UDP 100 MB/sec

Hyper Fabric

20

HMP

400 MB/sec

SCI

10

UDP

70 MB/sec

Table 6.1: Interconnect Types

If more than two nodes are interconnected in a cluster, they must be linked with a switch. Wherever possible, redundant interconnects must be provided by using dual switches. The following commands show which interconnect is used for UDP or TCP:

sqlplus> connect / as sysdba

oradebug setmypid

oradebug ipc

exit

The corresponding trace can be found in the user_dump_dest directory, and contains the following information in the last couple of lines:

SKGXPCTX: 0x32911a8 ctx

admno 0x12f7150d admport:

SSKGXPT 0x3291db8 flags SSKGXPT_READPENDING

    info for network 0

  socket no 9 IP 172.16.193.1 UDP 43307

  sflags SSKGXPT_WRITESSKGXPT_UP

  info for network 1

  socket no 0 IP 0.0.0.0 UDP 0

  sflags SSKGXPT_DOWN

context timestamp 0x1ca5

  no ports

When other protocols such as HMP or RDG are used, the trace file will not reveal the IP address.

Another method of finding the protocol is to look into the alert log file of the Oracle instance. During startup, a message is posted showing the IPC details. For example,

Wed Oct 30 05:28:55 2002

cluster interconnect IPC version: Oracle UDP/IP with Sun RSM disabled

IPC Vendor 1 proto 2 Version 1.0

Message Size

The message size is platform dependent. In Oracle9i Release 2 (9.2.0.1), message size was limited to 32K. Before 9.2.0.1, the Operating System UDP settings are overwritten and a 32k buffer is used.

Oracle9i allows larger message sizes, depending on the platform. To increase throughput on an interconnect, the UDP kernel parameters must be adjusted.

Oracle opens the UDP receive ports with a receive buffer size of 128k, but it also honors O/S settings and can even adjust the size based on the maximum block size seen, up to the O/S maximum. The operating system maximum for the UDP receive buffer size is:

Linux:                              256k

Solaris / Tru64 5.1:       1MB

HP:                                 2GB

When implementing the Sun Cluster with the clusterware provided by Veritas, the LMX message size is up to 64K, with a default of 32K.

 


This is an excerpt from the bestselling book Oracle Grid & Real Application Clusters, Rampant TechPress, by Mike Ault and Madhu Tumma.

You can buy it direct from the publisher for 30%-off and get instant access to the code depot of Oracle tuning scripts.

http://www.rampant-books.com/book_2004_1_10g_grid.htm


 

 
��  
 
 
Oracle Training at Sea
 
 
 
 
oracle dba poster
 

 
Follow us on Twitter 
 
Oracle performance tuning software 
 
Oracle Linux poster
 
 
 

 

Burleson is the American Team

Note: This Oracle documentation was created as a support and Oracle training reference for use by our DBA performance tuning consulting professionals.  Feel free to ask questions on our Oracle forum.

Verify experience! Anyone considering using the services of an Oracle support expert should independently investigate their credentials and experience, and not rely on advertisements and self-proclaimed expertise. All legitimate Oracle experts publish their Oracle qualifications.

Errata?  Oracle technology is changing and we strive to update our BC Oracle support information.  If you find an error or have a suggestion for improving our content, we would appreciate your feedback.  Just  e-mail:  

and include the URL for the page.


                    









Burleson Consulting

The Oracle of Database Support

Oracle Performance Tuning

Remote DBA Services


 

Copyright © 1996 -  2017

All rights reserved by Burleson

Oracle ® is the registered trademark of Oracle Corporation.

Remote Emergency Support provided by Conversational