Stuck threads and JMS notifications

The third example demonstrate the use of JMS notifications.  The following real life example demonstrates the monitoring of a typical production issue.  Hanging back ends, blocked resources or not responding partner systems causes thread in WebLogic to not respond for a long time. WebLogic will mark them after a given amount of time as "stuck" which normally always indicates a production issue.

The following example will define a WLDF watch which will monitor the stuck thread counter of the default work manager in WebLogic.  If this counter goes above 0 then it will generate a JMS notification.  A JMS client is provided to receive these JMS messages.

Setup the JMS server and a JMS destination for the WLDF notifications

The following WLST cod will setup all the JMS elements which are necessary so that WLDF can send out JMS notifications. Please note that the JMS configurations are NOT part of the WLDF subsystem in WebLogic. It can be either an own dedicated JMS server or a queue on a shared JMS server.

Step 1: Create the File Store

cd('/')

cmo.createFileStore('WLDFFileStore')

cd('/FileStores/WLDFFileStore')

cmo.setDirectory('/tmp')

set('Targets',jarray.array([ObjectName('com.bea:Name=AdminServer,Type=Server')], ObjectName))

Step 2: Create the JMS Server for WLDF Notification

cd('/')

print 'Creating JMS Server.'

cmo.createJMSServer('WLDFJMSServer')

cd('/JMSServers/WLDFJMSServer')

cmo.setPersistentStore(getMBean('/FileStores/WLDFFileStore'))

cmo.addTarget(getMBean('/Servers/AdminServer'))

Step 3: Create the JMS Module for WLDF notification

cd('/')

cmo.createJMSSystemResource('WLDFJMSSystemResource')

cd('/JMSSystemResources/WLDFJMSSystemResource')

cmo.addTarget(getMBean('/Servers/AdminServer'))

cmo.createSubDeployment('WLDFSubdeDloyment')

Step 4: Create the Connection Factory

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource')

cmo.createConnectionFactory('WLDFConnectionfactory')

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource/ConnectionFactories/WLDFConnectionfactory')

cmo.setJNDIName('jms/myWLDFFactory')

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource/ConnectionFactories/WLDFConnectionfactory/SecurityParams/WLDFConnectionfactory')

cmo.setAttachJMSXUserId(false)

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource/ConnectionFactories/WLDFConnectionfactory/ClientParams/WLDFConnectionfactory')

cmo.setClientIdPolicy('Restricted')

cmo.setSubscriptionSharingPolicy('Exclusive')

cmo.setMessagesMaximum(1000)

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource/ConnectionFactories/WLDFConnectionfactory')

cmo.setDefaultTargetingEnabled(true)

Final Step 5: Create the Queue for our WLDF notifications

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource')

cmo.createQueue('WLDFMonitoringQueue')

cd('/JMSSystemResources/WLDFJMSSystemResource/JMSResource/WLDFJMSSystemResource/Queues/WLDFMonitoringQueue')

set('JNDIName','jms/WLDFSTuckThreadNotificationQueue')

set('SubDeploymentName','WLDFSubdeDloyment')

cd('/JMSSystemResources/WLDFJMSSystemResource/SubDeployments/WLDFSubdeDloyment')

cmo.addTarget(getMBean('/JMSServers/WLDFJMSServer'))

print 'WLDF JMS Resources are created'

After the following steps have been completed, WebLogic has created a JMS configuration. JMS configurations in WebLogic are stored in own configuration files which are saved in the "jms" subfolder of the <domain>/config directory.

The following configuration was created:

<weblogic-jms ...>

  <connection-factory name="WLDFConnectionfactory">

    <default-targeting-enabled>true</default-targeting-enabled>

    <jndi-name>jms/myWLDFFactory</jndi-name>

    <client-params>

      <client-id-policy>Restricted</client-id-policy>

      <subscription-sharing-policy>Exclusive</subscription-sharing-policy>

      <messages-maximum>1000</messages-maximum>

    </client-params>

    <security-params>

      <attach-jmsx-user-id>false</attach-jmsx-user-id>

    </security-params>

  </connection-factory>

  <queue name="WLDFMonitoringQueue">

    <sub-deployment-name>WLDFSubdeDloyment</sub-deployment-name>

    <jndi-name>jms/WLDFSTuckThreadNotificationQueue</jndi-name>

  </queue>

</weblogic-jms>

Setup the WLDF module

Next we have to define and setup the WLDF diagnostic module. As stated above, the WLDF configuration will make use of the JMS configuration defined above and which is defined outside of the WLDF module.

edit()

startEdit()

cd('/')

cmo.createWLDFSystemResource('StuckThreadDetectionModule')

cd('/WLDFSystemResources/StuckThreadDetectionModule')

cmo.setDescription('Module to detect stuck threads')

cd('/WLDFSystemResources/StuckThreadDetectionModule/WLDFResource/StuckThreadDetectionModule/WatchNotification/StuckThreadDetectionModule')

cmo.createJMSNotification('StuckThreadNotification')

cd('/WLDFSystemResources/StuckThreadDetectionModule/WLDFResource/StuckThreadDetectionModule/WatchNotification/StuckThreadDetectionModule/JMSNotifications/StuckThreadNotification')

cmo.setEnabled(true)

cmo.setDestinationJNDIName('jms/WLDFSTuckThreadNotificationQueue')

cmo.setConnectionFactoryJNDIName('jms/myWLDFFactory')

cd('/WLDFSystemResources/StuckThreadDetectionModule')

set('Targets',jarray.array([ObjectName('com.bea:Name=AdminServer,Type=Server')], ObjectName))

cd('/WLDFSystemResources/StuckThreadDetectionModule/WLDFResource/StuckThreadDetectionModule/WatchNotification/StuckThreadDetectionModule')

cmo.createWatch('StuckThreadWatch')

cd('/WLDFSystemResources/StuckThreadDetectionModule/WLDFResource/StuckThreadDetectionModule/WatchNotification/StuckThreadDetectionModule/Watches/StuckThreadWatch')

cmo.setRuleType('Harvester')

cmo.setEnabled(true)

cmo.setRuleExpression('(${ServerRuntime//[weblogic.management.runtime.WorkManagerRuntimeMBean]//StuckThreadCount} > 0)')

cmo.setAlarmType('AutomaticReset')

cmo.setAlarmResetPeriod(30000)

set('Notifications',jarray.array([ObjectName('com.bea:Name=StuckThreadNotification,Type=weblogic.diagnostics.descriptor.WLDFJMSNotificationBean,Parent=[base_domain]/WLDFSystemResources[StuckThreadDetectionModule],Path=WLDFResource[StuckThreadDetectionModule]/WatchNotification[StuckThreadDetectionModule]/JMSNotifications[StuckThreadNotification]')], ObjectName))

print 'WLDF diagnostic module is created'

activate()

WebLogic has created the following WLDF configuration in the "diagnostic" subfolder:

<wldf-resource ...>

  <name>StuckThreadDetectionModule</name>

  <watch-notification>

    <watch>

      <name>StuckThreadWatch</name>

      <enabled>true</enabled>

      <rule-type>Harvester</rule-type>

      <rule-expression>(${ServerRuntime//[weblogic.management.runtime.WorkManagerRuntimeMBean]//StuckThreadCount} &gt; 0)</rule-expression>

      <alarm-type>AutomaticReset</alarm-type>

      <alarm-reset-period>30000</alarm-reset-period>

      <notification>StuckThreadNotification</notification>

    </watch>

    <jms-notification>

      <name>StuckThreadNotification</name>

      <enabled>true</enabled>

      <destination-jndi-name>jms/WLDFSTuckThreadNotificationQueue</destination-jndi-name>

      <connection-factory-jndi-name>jms/myWLDFFactory</connection-factory-jndi-name>

    </jms-notification>

  </watch-notification>

</wldf-resource>

Define the JMS client

These notifications are only useful if clients will register with the WebLogic MBeans in order to receive these events.  The following JAVA code discusses how to write a simple JMX client to consume these notifications.

public class ReceiveJMSNotificationExample implements MessageListener

{

  public final static String JMS_CONNECTION_FACTORY="jms/myWLDFFactory";

  public final static String WLDF_NOTIFICATION_QUEUE="WLDFSTuckThreadNotificationQueue";

  private QueueConnectionFactory jmsConnectionFactory;

  private QueueConnection jmsConnection;

  private QueueSession jmsSession;

  private QueueReceiver jmsQueueReceiver;

  /**

   * Initialize the JMS layer and start listening

   * @param ctx

   * @throws Exception

   */

  public void init(Context ctx) throws Exception

  {

    jmsConnectionFactory = (QueueConnectionFactory) ctx.lookup(JMS_CONNECTION_FACTORY);

    jmsConnection = jmsConnectionFactory.createQueueConnection();

    jmsSession = jmsConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

    jmsQueueReceiver = jmsSession.createReceiver((Queue) ctx.lookup(WLDF_NOTIFICATION_QUEUE));

    jmsQueueReceiver.setMessageListener(this);

    jmsConnection.start();

  }

  /**

   * Action to do whenever a message is received. In this case we just print it.

   * For this scenario a good idea could be to connect to the server where the message is comming

   * from and issue a thread dump.

   */

  public void onMessage(Message myWLDFMessage)

  {

    try

    {

      if (myWLDFMessage instanceof TextMessage)

      {

        System.out.println("Message Received: "+  ((TextMessage)myWLDFMessage).getText());

      }

      else if (myWLDFMessage instanceof MapMessage)

      {

         System.out.println("Message Received: "+ myWLDFMessage.toString());

         MapMessage map = (MapMessage)myWLDFMessage;

         Enumeration en = map.getMapNames();

         while (en.hasMoreElements()){

            String nE = (String)en.nextElement();

            System.out.println("    "+nE+" = "+ map.getObject(nE).toString());

         }

      } 

      else

      {

        System.out.println("Message Received: "+ myWLDFMessage.toString());

      }

    }

    catch (JMSException ex) {

      ex.printStackTrace();

    }

  }

  /**

   * Cleanup the JMS layer and close resources

   * @throws JMSException

   */

  public void cleanupJMSResources() throws JMSException

  {

    jmsQueueReceiver.close();

    jmsSession.close();

    jmsConnection.close();

  }

  public static void main(String[] args) throws Exception

  {

    Hashtable<String,String> envProperties = new Hashtable<String,String>();

    envProperties.put(Context.INITIAL_CONTEXT_FACTORY, "weblogic.jndi.WLInitialContextFactory");

    envProperties.put(Context.PROVIDER_URL, "t3://localhost:12001");

    InitialContext ctx = new InitialContext(envProperties);

    ReceiveJMSNotificationExample myReceiver = new ReceiveJMSNotificationExample();

    myReceiver.init(ctx);

    System.out.println("WLDF Notification Listener is now listening " +

                       "for WLDF notification JMS Messages");

    // very bad implementation - just to make example short !!!!!!!

    Thread.sleep(Long.MAX_VALUE);

    myReceiver.cleanupJMSResources();

  }

}

Please note that we are distinguishing between the different types JMS messages we might receive. Also note that, (for simplicity), most error handling and a better way of sleeping in the main method has been deleted.

Run the example

When we start the example, it is assumed that the AdminServer is running, the JMS server is active and that the WLDF module is active. Then we start the client which will connect to the JMS server of the AdminServer and waits for messages.

If we are running into a stuck thread situation then the following message will be received. Note that WLS is sending a MapMessage with a number of information. For a better understanding of what is send from WebLogic the complete received messages is printed below:

WLDF Notificatin Listener is now listening for WLDF notification JMS Messages

Message Received: MapMessage[ID:<261079.1404683786387.0>]

    WatchAlarmResetPeriod = 30000

    WatchSeverityLevel = Notice

    WatchRule = (${ServerRuntime//[weblogic.management.runtime.WorkManagerRuntimeMBean]//StuckThreadCount} > 0)

    WatchDomainName = base_domain

    WatchModule = StuckThreadDetectionModule

    WatchData = com.bea:Name=weblogic.kernel.System,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=bea_wls_management_internal2,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=OneWayJTACoordinatorWM,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.jms.WLDFJMSServer.System,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=StuckThreadDetectionModule,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.Rejector,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=bea_wls_diagnostics,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=wl_oldBootStrap,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=HARVESTER_WM,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=direct,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=JmsDispatcher,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.kernel.WTC,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.kernel.Non-Blocking,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.admin.RMI,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=bea_wls_deployment_internal,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=wlhostmachinestats-030,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=DAPInfoWebLogic,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=WatchManagerEvents,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.jms.WLDFJMSServer.AsyncPush,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=ImageWorkManager,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=JmsAsyncQueue,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=WLDFJMSSystemResource,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.logging.DomainLogBroadcasterClient,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=bea_wls_internal,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.jms.WLDFJMSServer.Limited,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=consoleapp,Name=consoleWorkManager,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.logging.LogBroadcaster,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=UserLockout,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=bea_wls9_async_response,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=wls-wsat,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=DataRetirementWorkManager,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.kernel.Default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=dummy,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 1 com.bea:ApplicationRuntime=consoleapp,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=weblogic.nodemanager.ConfigPoler,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:Name=JTACoordinatorWM,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0 com.bea:ApplicationRuntime=mejb,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 0

    JMSNotificationName = StuckThreadNotification

    WatchAlarmType = AutomaticReset

    WatchRuleType = Harvester

    WatchName = StuckThreadWatch

    WatchServerName = AdminServer

    WatchTime = Jul 6, 2015 11:56:26 PM CEST

The reason which the "WatchData" section is so big is that WebLogic by default has a large number of WorkManagers. In our setup we did not restrict the notification to s specific instance, otherwise this section would be much smaller.

If you look carefully at the "WatchData" section you can see the reason for the notification.

com.bea:ApplicationRuntime=dummy,Name=default,ServerRuntime=AdminServer,Type=WorkManagerRuntime//StuckThreadCount = 1

This tell us that a thread in the "default" WorkManager and here in the application "dummy" got stuck.  And this is true as the "dummy" application has enforced the situation (it was just a very simple servlet which did nothing but slept for 15 min). So you get a number of very valuable information like which application, which server, when, which domain and more. If you have a proper unique naming for your WLS domains then it would be easy now to automatically connect to this domain and this server and enforce a thread-dump in order to get more information.