Oracle Linux DBA Benefits Don Burleson

• vmstat - Provides real-time reports on CPU consumption, CPU dispatcher run queue, RAM page in, scan rate, RAM page outs
• iostat - Provides detailed disk I/O information including disk enqueues and I/O latency
• nestat - Shows TCP/IP traffic and packet details
• watch - Shows top server task details including server load average
• ps - Shows process-level details on resource consumption

These utilities are exceptional for monitoring Oracle databases and they can be placed inside Linux shell scripts to take the output from the Linux commands and store the information inside Oracle tables.

For example, here is a utility that will take the output from Linux vmstat utility and insert the output into an Oracle table:

PATH=$ORACLE_HOME/bin:$PATH
export PATH
SERVER_NAME=`uname -a|awk '{print $2}'`
typeset -u SERVER_NAME
export SERVER_NAME

# sample every five minutes (300 seconds) . . . .
SAMPLE_TIME=300

while true
do
   vmstat ${SAMPLE_TIME} 2 > /tmp/msg$$


   # run vmstat and direct the output into the Oracle table . . .
   cat /tmp/msg$$|sed 1,3d | awk '{ printf("%s %s %s %s %s %s\n", $1, $8, $9,
   14, $15, $16) }' | while read RUNQUE PAGE_IN PAGE_OUT USER_CPU SYSTEM_CPU
   IDLE_CPU
   do

      $ORACLE_HOME/bin/sqlplus -s perfstat/perfstat@testsys1<<EOF
      insert into perfstat.stats\$vmstat
      values (
      sysdate,
      $SAMPLE_TIME,
      '$SERVER_NAME',
      $RUNQUE,
      $PAGE_IN,
      $PAGE_OUT,
      $USER_CPU,
      $SYSTEM_CPU,
      $IDLE_CPU,
      0
      );
      EXIT
      EOF
  done
done

rm /tmp/msg$$

Note: This type of operating system script is largely obsolete in Oracle10g because the Automated Workload Repository (AWR) captures this OS information automatically.

We can use the same script technique with the Linux iostat utility and shell scripts can be used to monitor disk I/O and place the details inside Oracle tables.

#!/bin/ksh

while true
do
   iostat -x 300 1|\
   sed 1,2d|\
   awk '{ printf("%s %s %s\n", $1, $4, $5) }' |\
   while read HDISK VMSTAT_IO_R VMSTAT_IO_W
   do

      if [ $VMSTAT_IO_R -gt 0 ] and [ $VMSTAT_IO_W -gt 0 ]
      then
          sqlplus -s perfstat/perfstat <<!
          insert into
          perfstat.stats\$iostat
          values
         (SYSDATE, 5, '$HDISK', $VMSTAT_IO_R,$VMSTAT_IO_W);
         exit
         !
       fi
   done
   sleep 300
done

As we see, the easy-to-use Linux monitoring utilities open-up a whole new world to the Oracle DBA. If you take the time to master the Linux command syntax and utilities you will gain the ability to diagnose any Oracle problem and always know the exact cause of any OS-related performance issue. Now let's take a look at some of the command-line utilities for Linux and see how they can simplify the tasks of the Oracle DBA.


Command Utilities for Linux

As we have noted, Linux has powerful command-line utilities that provide exceptional monitoring for every aspects of Oracle's interactions with the hardware environment. But there is more to Linux than the commands. The Oracle professional will also need to master shell scripting, task scheduling and specialized functions:

• The vi file editor - The vi editor is an extremely powerful text editor that is far more powerful and flexible than Windows-based editors such as notepad.
   
• Utility commands - Linux provides powerful system tools such as grep, sed and awk that allow you to create comprehensive OS monitoring facilities.
   
• Job scheduling - The Linux crontab utility allows for scheduling OS tasks at specific time intervals.
   
• Scripting - Shell environments such as Korn shell, C shell, Bourne shell and Bourne Again shell (bash) allow you to create sophisticated Linux monitoring scripts.

Linux tools are very powerful and flexible, and like any powerful tool, they are complex. It can take years for the staff to master Linux internals, using the powerful vi editor, and learning to program with UNIX shell scripts. Let's take a quick look at the types of scripts that a Linux DBA might employ for their Oracle environment.

There are many OS events that need to be monitored by the Oracle DBA and prior to Oracle10g the DBA would want to be notified of any server-side event.

Operating System Events:

     • File system is filling rapidly
     • Server is paging RAM
     • High CPU enqueues
     • Messages in Linux syslog (I/O errors, bus errors)

Oracle OS events:

     • Alert log message
     • Core dump
     • Background dump
     • User dump created

For example, here is an example of an actual UNIX script for Oracle to detect a almost-full filesystem:

#!/bin/ksh

for i in `df -k|grep /u0|awk '{ print $4 }'`
do
   filesize=`expr i`
   if [ $filesize -lt 100 ]
   then
      mailx -s "Oracle filesystem $i has less than 100k free."\
      bill.gates@mircosoft.com\
      lawrence_ellison@oracle.com
   fi
done

One of the most powerful utilities is grep which is used to quickly find files that contain specific strings. In fact, some Oracle DBA's using Windows will download a free copy of grep for PC's because of its power and ease of use. For example, the scripts below will cascade through the Linux directory structures, quickly finding all files that contain a specific string:

find . –print|xargs grep –i string

This script is far faster and easier to use than the Windows GUI search tool.

Linux Configuration

Configuring Linux for optimal performance is highly-dependent on the hardware. When you use a server like the UNISYS ES7000, the configuration is easy because the Linux kernel has already been optimized for the specific high-speed hardware components.

Because UNISYS Linux for Oracle is preconfigured for Oracle, you avoid the headaches of optimizing the OS for the Oracle software. In proprietary UNIX, proper configuration for Oracle is often overlooked and SUN, HP and AIX Oracle databases are often improperly configured.

For example, Oracle data files should be configured to bypass the UNIX server buffer cache and read the data blocks directly into the Oracle data buffers. However, a surprising number of proprietary UNIX shops have their I/O improperly configured, causing multiple buffer access whereby data blocks travel onto the UNIX Journaled File System (JFS) buffer cache, and then transferred into the Oracle data buffers.

This additional I/O overhead is a major problem and one that is automatic in Linux. Linux systems support direct I/O on a per-filehandle basis (which is much more flexible), and Oracle enables this O_DIRECT feature automatically. This is not true for those using Sun, HP and Veritas I/O, and tricky utilities must be implemented to enable direct I/O:

• Solaris - Uses the "forcedirectio" option. Oracle DBAs claim this option makes a huge difference in I/O speed for Sun servers.
   
• AIX - Uses the "dio" option.
   
• Veritas VxFS - (including HP-UX, Solaris and AIX), uses the "convosync=direct". It is also possible to enable direct I/O on a per-file basis using Veritas QIO; refer to the "qiostat" command and corresponding man page for hints.

In sum, using Linux with a hardware vendor such as UNISYS ensures that the Linux environment is optimized to run as quickly as possible. Now let's look at the issue of migrating to Linux from Windows and proprietary UNIX.

Migrating to Linux from Proprietary UNIX

Migrating into Linux can be very easy or very difficult, depending on your existing Oracle configuration. It is not uncommon to see small "dialect" differences between implementations of UNIX, especially with respect to display output and command arguments.

Those shops that find a migration difficult are those that have made extensive use of operating system utilities for their Oracle functions:

External scheduling of Oracle tasks

   - Crontab
   - Windows "at" scheduling

Extensive use of OS shell scripts

   - Automated e-mail alerts (new dumps, alert log messages, Linux syslog messages)
   - File system alerts
   - Server overload alerts

If you avoid the OS trap and use Oracle to schedule all Oracle tasks you can easily port your database across OS platforms. This is especially true if you migrate to Oracle10g where Oracle automatically collects OS metrics and has a built-in alert and scheduling mechanism:

• Migrate to Oracle10g
• Schedule all Oracle tasks with dbms_scheduler
• Use the Oracle dbms_alert package to replace OS scripting