OracleAS 10.1.3 & multicast communications Oracle App Server Tips by John Garmany

How  to communicate changes

Each release of the Oracle Application Server provides new capabilities, better stability, and increased usability.  The latest version of OracleAS, 10.1.3, is currently only available in the J2EE configuration.  If you want to use any other edition of the application server, you will need to use version 10.1.2.  However, even if you already use an earlier version, looking at the new capabilities of the latest J2EE edition will give you a preview of the great new features to come.  In this article we will focus on how the 10.1.3 Application Server uses multicast to replicate state information, cluster instances, and communicate between instances.

What is Multicast? 

            The term multicast basically refers to broadcasting a message across the entire network, rather than just to one other computer.  In a normal configuration, if server A is hosting an application and the application is no longer available, server A will notify the requesting server B that the application is no longer available.  Then server B will respond that it understands, and it will route all further request to another server where the application is still running.  In the case of a multicast, server A will simply announce to all the servers in its topology list that the application is shutting down.  In some cases servers do not respond to a multicast, so server A may not wait for a reply.   Server B will receive the notification from server A and route request to another server. 

            Multicasting was not invented by Oracle and is used in a number of applications on the Internet.  Because it allows one notification to be used by multiple recipients, it is an effective and efficient method to utilize in one-to-many communication.  Streaming audio/video and conferencing systems use multicast to efficiently deliver content to multiple users making efficient use of network resources.  There are standard ranges of IP address established for different types of multicast system.  Oracle allows you to use IPs from 224.0.0.1 to 239.255.255.255.  All installations included in the cluster must be listening on the same address (IP and port number).  The Oracle Universal Installer is the easiest way to setup multicasting.  Select the advanced installation at the opening screen and enter the IP and port in the Cluster Topology Configuration screen (example below).  Make sure that you select an open port and an IP address that will not conflict with any other application that may also be using multicast. 

 

 

            So multicast implies some functionality.  First, all the servers are listening for the notifications and keeping a list of server state and capabilities.  Each server needs to be able to manage the capabilities list internally and overcome missing a notification, such as updating their availability list when request to a certain server times out.    OracleAS uses multicasting in establishing and maintaining clusters

Clustering the Enterprise Way.

            Clustering is defined differently in different version of the application server, and between different components within the application server.  Basically all the definitions include the ability to group components/instances with common configurations for ease of management.  For now, we will focus on the differences between the 10.1.2.Enterprise Edition clustering and the 10.1.3 J2EE Edition clustering.

            In the Enterprise Edition of the Oracle Application Server all configuration data is maintained in the metadata repository.  This configuration data is located in the Oracle Internet Directory, stored in a 10gR1 database by default.  A cluster is created by defining the cluster name, and adding the first middle tier instance.  This first instance defines the configuration of the cluster.  Each subsequent middle tier instance that is added to the cluster will lose its current configuration and be reconfigured to exactly match the cluster configuration, to include the containers and applications deployed into the cluster configuration.  If an additional middle tier instance is require, a new instance can be installed and added to the cluster.  No configuration is required for the new instance before it is adding to the cluster.  This method of clustering is very efficient for managing large, complicated installations. 

Clustering in the J2EE Edition 10.1.3

            The J2EE edition or Java edition contains only the Oracle Container for Java (OC4J), the Oracle HTTP Server (OHS) and the processes needed to maintain the instance, such as Oracle Process Management and Notification (OPMN).   The J2EE Edition has no metadata repository to maintain the configuration data.  Configuration data is maintained in flat files using XML.  This leads to cluster configurations that are less restrictive.  In the J2EE edition, a cluster is defined as two or more loosely connected nodes. The nodes can be OHS and/or OC4J instances.  Many times one OHS will manage request for multiple OC4J containers.  Using the clustering capabilities, multiple OHS nodes can service request to multiple OC4J nodes.  Oracle uses multicast to enable these separate nodes to locate and monitor each of the other nodes in the configuration.  As a simple example, if you start an OC4J container, it will send a notification using multicast that it is running and what applications it is supporting.  The OHS nodes in the cluster will receive the multicast notification and add the OC4J container to their list of nodes to service requests.   Clustering using a multicast method provides you with lot of flexibility.

1. Each OC4J Container does not have to host the same applications.
2. The cluster nodes use Dynamic Discover to self managed the system topology.
3. Load balancing is automatic between nodes with the same capability.

Management of the cluster becomes easier because most of the configuration is maintained by each node. 

 

Multiple Containers with Multiple Capabilities

            Unlike with the Enterprise Instance cluster, the OC4J instances are not required to have the same configuration or even host the same applications.  Each OC4J container can host a unique application or one application can be hosted in multiple OC4J containers.  If multiple containers host the same application, the OHS instances will load balance request across those containers.

 

 

Since each OC4J instance will use multicast to notify other instances that it is running and what applications it supports, the administrator does not have to manually configure the instance linkages as containers/applications are added or removed.  The method used to mange the list of available nodes is called Dynamic Discovery.

Dynamic Discovery

            The key to establishing what instances are available and their capabilities is keeping a current topology map.  This is the job of Dynamic Discovery.  Dynamic Discovery allows each instance (OC4J or OHS) to listen to the multicast notifications and generate a list of the other instances are participating in this system topology.  As instances start or shutdown, the Oracle Notification Server (ONS) process for each instance will update its topology map.  It is the ONS that manages the multicast communication between nodes, including between the OHS and OC4J nodes.  Since the ONS process has been integrated into OPMN, the opmn.xml file contains the configuration information for discovery.  Below is an example containing a section of an opmn.xml file.

 

   <notification-server>

      <port local="6103" remote="6203" request="6006"/>

      <ssl enabled="true"

             wallet-file="$ORACLE_HOME\opmn\conf\ssl.wlt\default"/>

      <topology>

         <discover list="*225.0.0.1:6789"/>

      </topology>

   </notification-server>

If the OHS instance knows that multiple OC4J containers are hosting the same application, it will automatically load balance request among the OC4J containers.  In the OHS instance it is the mod_oc4j module that is responsible for load balancing request among available OC4J containers.  Load balancing is not as simple as it first appears because the OC4J container may maintain some type of session state from a previous request.  The classic example of session state is a shopping cart.  When a user places an item in their shopping cart, that item must be maintained across multiple requests to the server.  If the request is sent by the OHS instance to a new OC4J instance, the customer's shopping cart would suddenly be empty.  The mod_oc4j module must route sessions back to the correct OC4J container that is maintaining the session's state.  Multicast can also be used to replicate session state and we will cover that capability in a moment.

 

Application Server Control

            The Application Server Control application has been a key component in creating and managing clusters in all versions of the Application Server.  With OracleAS 10.1.3, AS Control has been rewritten as a light weight application, deployed into each OC4J container.  Oracle redesigned AS Control to use the integrated Java Management Extensions (JMX).  Using JMX allowed Oracle to eliminate the Oracle Management Agent, resulting in AS Control having a smaller footprint and utilizing fewer resources.  When you create a cluster in 10.1.3, one OC4J container is designated as the AS Control for the cluster.  All other container will set the AS Control application to not start.  No matter which OHS instance in a cluster you use to log into AS Control, you will be directed to the designated cluster AS Control.  Below is an example of AS Control displaying a cluster consisting of three instances.

 

 

 

Notice that the AS Control application is down on as1 and as3.  The green start burst next to AS Control on as2 indicates that it is the AS Control for the entire cluster.   In this cluster there are three OC4J containers and only one OHS instance (listed as the HTTP_Server).

            You can also monitor the cluster using the Oracle Process Management and Notification (OPMN) service.  You can request the cluster status on any node in the cluster.

 

C:\>D:\oracleas\10.1.3\OracleAS_1\opmn\bin\opmnctl @cluster status

 

Processes in Instance: as2_j2ee.svr3

-------------------+--------------------+---------+---------

ias-component      | process-type       |     pid | status

-------------------+--------------------+---------+---------

OC4J               | home               |    1676 | Alive

ASG                | ASG                |     N/A | Down

 

Processes in Instance: as3_j2ee.svr3

-------------------+--------------------+---------+---------

ias-component      | process-type       |     pid | status

-------------------+--------------------+---------+---------

OC4J               | home               |    2164 | Alive

ASG                | ASG                |     N/A | Down

 

Processes in Instance: as1_j2ee.svr3

-------------------+--------------------+---------+---------

ias-component      | process-type       |     pid | status

-------------------+--------------------+---------+---------

OC4J               | home               |    3024 | Alive

HTTP_Server        | HTTP_Server        |    3016 | Alive

ASG                | ASG                |     N/A | Down

Adding the ??l? parameter will display the ports currently used by each component.  Notice that this cluster does not have a name and the components in it are not configured the same.  If you deploy the petstore application into as3's home container, it will not automatically deploy into the other containers in the cluster.  However, if you want the ability to deploy an application into all of the containers in a cluster, you have that ability using a Group.

 

What's in a Name?  A Group!

            A Group consists of all the OC4J containers in a cluster, that have the same name.  Because they are managed as a group they must also have the same admin password.  In the example above, all the OC4J containers are named HOME.  When I installed the instances, I wanted to create a cluster that would allow me to deploy an application into all the OC4J containers in the cluster at the same time.  So I named them all the same name, HOME.  If we scroll down below the cluster information in AS Control we can see the Group configuration.

 

Here, all the OC4J containers that have the name HOME are grouped.  If I deploy an application into the HOME container in a Group, the application will be deployed into all the containers in that Group.  The example above only has one Group defined.  There is no restriction on the number of Groups in a cluster.  Each set of OC4J containers that share a name will be placed in a Group. 

            All this configuration information is constantly being monitored and updated by the ONS process using multicast.  As components start, stop or change configuration that change in state is broadcast to all the node at the same time.  But configuration data is not the only use for multicast in OracleAS 10.1.3.  This new version of the Application Server uses multicast to share session state information.

Islands in the Stream, No More!

            Session state is data that is specific to a particular web/application session.  It is stored by the application in web variable or java session beans.  In an earlier example we used a shopping cart as an example of session state.  It is the job of the OHS to route request from the same session to the OC4J container that holds that session's state.  But what happens when that container no longer is available?  OHS must route that session's request to a new container and the session data will be lost.  Previous version of OracleAS used Islands to share session state.  When OC4J containers were configured as part of an Island, each container would maintain the session state for the sessions in the other containers.  If an OC4J container was no longer available, OHS would route the request to an OC4J container in the same Island and the session state would be maintained.  The user would not know that the request was being serviced by another container.  The concept of Islands does not exist in OracleAS 10.1.3.  The new version uses three methods to share state between containers: the database, peer to peer, and multicast.  Here we are going to discuss using multicast to share session state among OC4J containers.

 

Multicasting Your State.

            Sharing session state information is also called clustering.  In this example, we are clustering applications rather than components.  Setting up session state replication is a manual process and we are going to cover the basics in this article.  You will want to spend some time in the manuals when creating you own configuration.  State replication is configured in the $ORACLE_HOME/j2ee/applidation-deployment/<name>/orion-application.xml file.  Add a replication tag defining when and what data is replicated.

 

<replication-policy trigger="onRequestEnd"

                scope="modifiedAttributes " />

 

In this example, only those attributes that changed are replicated and the replication takes place at the end of each request.  For multicast to be used, the multicast IP and port must also be specified.  This IP and port can be the same as the one used by the instance multicast or a separate address.  The only requirement is that all the OC4J containers in the replication must be configured to use the same multicast address.  Here is a complete section of the orion-application.xml file.

<cluster>

  <replication-policy trigger="onRequestEnd "

                   scope="modifiedAttributes " />

   <protocol>

      <multicast ip="225.0.0.1" port="6789" />

   </protocol>

   write-quota = 2

</cluster>

The write-quota parameter defines how many of the containers will replicate the session state.  In this example, the cluster has 3 OC4J containers and a wrote-quota of two so every container in the cluster will maintain this container's session state.  The default wrote-quota is 1, so if your cluster has more that two OC4J containers, the write-quota parameter should be increased.  When state is replicated using multicast, the receiving OC4J containers will acknowledge receiving the data.  The multicasting container will rebroadcast the state data until all the containers successfully receive it.  Because of the ability to update multiple OC4J containers with one notification, multicast is the preferred method of session state replication in OracleAS 10.1.3.

A Final Note.

As your application becomes more complex and the user load increases, it is easy to add additional instances to the topology to handle the growing workload.  As each instance is added, more configuration and state data is distributed by multicast.  One consideration is to use an additional network connection to separate the multicast network traffic from the application traffic.  This separates the multicast chatter from the request/response traffic of your application.

            This article has covered a lot of complicated ground in a short space to give you an idea of how OracleAS 10.1.3 uses multicast to share state and configuration information.  Multicast has the benefit of one network notification being received by multiple components.  This reduces network traffic will still insure that information is being shared.  The implementation of multicast is just one of the many great new features in OracleAS 10.1.3.  Check it out.

 

If you like Oracle tuning, see the book "Oracle Tuning: The Definitive Reference", with 950 pages of tuning tips and scripts.  You can buy it direct from the publisher for 30%-off and get instant access to the code depot of Oracle tuning scripts.