Call now: 252-767-6166  
Oracle Training Oracle Support Development Oracle Apps

 
 Home
 E-mail Us
 Oracle Articles
New Oracle Articles


 Oracle Training
 Oracle Tips

 Oracle Forum
 Class Catalog


 Remote DBA
 Oracle Tuning
 Emergency 911
 RAC Support
 Apps Support
 Analysis
 Design
 Implementation
 Oracle Support


 SQL Tuning
 Security

 Oracle UNIX
 Oracle Linux
 Monitoring
 Remote s
upport
 Remote plans
 Remote
services
 Application Server

 Applications
 Oracle Forms
 Oracle Portal
 App Upgrades
 SQL Server
 Oracle Concepts
 Software Support

 Remote S
upport  
 Development  

 Implementation


 Consulting Staff
 Consulting Prices
 Help Wanted!

 


 Oracle Posters
 Oracle Books

 Oracle Scripts
 Ion
 Excel-DB  

Don Burleson Blog 


 

 

 


 

 

 

 
 

Interpreting Oracle RAC STATSPACK/AWR performance reports

Oracle 11g Tips by Donald Burleson

 

Interpreting an Oracle RAC (Real Application Clusters) STATSPACK/AWR performance reports can be tricky, and it's quite different from reading a standard STATSPACK report.

 

The most obvious difference is the shared "cache fusion" component, and it can skew the performance report data, often hiding the root cause of a performance problem.

 

For example, consider this RAC STATSPACK report:

 

Load Profile
~~~~~~~~~~~~                            Per Second       Per Transaction
                                   ---------------       ---------------
                  Redo size:              7,134.04              8,461.68
              Logical reads:             43,060.81             51,074.43
              Block changes:                 21.39                 25.38
             Physical reads:                154.16                182.84
            Physical writes:                 79.52                 94.31
                 User calls:                 68.52                 81.28
                     Parses:                 22.02                 26.12
                Hard parses:                  0.20                  0.24
                      Sorts:                 24.93                 29.57
                     Logons:                  1.29                  1.53
                   Executes:                 29.36                 34.83
               Transactions:                  0.84
 
  % Blocks changed per Read:    0.05    Recursive Call %:    18.71
 Rollback per transaction %:   21.31       Rows per Sort:    37.21
 
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            Buffer Nowait %:  100.00       Redo NoWait %:  100.00
            Buffer  Hit   %:   99.82    In-memory Sort %:  100.00
            Library Hit   %:   99.52        Soft Parse %:   99.09
         Execute to Parse %:   25.00         Latch Hit %:   99.92
Parse CPU to Parse Elapsd %:    2.98     % Non-Parse CPU:   99.32
 
 Shared Pool Statistics        Begin   End
                               ------  ------
             Memory Usage %:   81.11   83.06
    % SQL with executions>1:   71.32   71.46
  % Memory for SQL w/exec>1:   51.65   53.13
 
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~                                                     % Total
Event                                               Waits    Time (s) Ela Time
-------------------------------------------- ------------ ----------- --------
PX Deq Credit: send blkd                           14,513       1,864    46.27
CPU time                                                        1,002    24.89
library cache pin                                     413         486    12.07
PX qref latch                                         254         117     2.90
PX Deq: Table Q Sample                              1,272         110     2.74

 

Here we see high buffer hit ratios (which is deceptive), and note the "PX" entries in the top-5 timed events, which indicate that parallelism is turned-on and being used.  This, in turn, indicates the presence of full-scan activity (multiblock reads from large-table full-table-scans and/or index fast-fast-full scans).

Because this is an OLTP database, we next verify that parallel_automatic_tuning is turned-on, in this case a very bad thing, because it has influenced the database to make full-scan access more attractive than it should be.

parallel_adaptive_multi_user  TRUE
parallel_automatic_tuning     TRUE

We also see relatively high pinging on this RAC node.  This database cannot be partitioned (load balanced) by end-user usage (placing different "types" of end-users on different nodes), and there is the inevitable inter-node pinging as common data is requested between active nodes.

Global Cache Service - Workload Characteristics
-----------------------------------------------
 
Global cache hit ratio:                                    0.2
Ratio of current block defers:                             0.0
% of messages sent for buffer gets:                        0.2
% of remote buffer gets:                                   0.0
Ratio of I/O for coherence:                                0.6
Ratio of local vs remote work:                             6.0
Ratio of fusion vs physical writes:                        0.0

 

The core performance problems:

Upon closer inspection of this database we would the following problems, none of which were clear from the high-level summary statistics:

  • Unnecessary parallelism - Each node has only two CPU's and parallel_automatic_tuning required turning off.
     

  • Missing indexes - Missing indexes were invoking unnecessary large-table full-table scans.
     

  • I/O bottlenecks - A closer inspection of the I/O subsystem reveals super-slow I/O timings and clear disk enqueues.

 

Obfuscated details

Unlike a single-node STATSPACK/AWR report, a RAC statspack report is more subtle, and the top problems are not readily apparent from the top event statistics.  In this case, even though the data buffer hit ratio is 99%, there is still a huge I/O contention issue, which can be remedied with a larger db_cache_size:

Tablespace
------------------------------
                 Av      Av     Av                    Av        Buffer Av Buf
         Reads Reads/s Rd(ms) Blks/Rd       Writes Writes/s      Waits Wt(ms)
-------------- ------- ------ ------- ------------ -------- ---------- ------
VPPP_SP
        37,754      10 ######     4.7          137        0          7    1.4
CBPO_SP
        33,015       9 ######     1.2           15        0        716    2.0
TEMP
         9,265       3    1.7    30.1        9,350        3          0    0.0
PRODMAN
        12,560       3 ######     4.0          125        0          0    0.0
PRODDTAI
         1,219       0    7.5     1.0          990        0          0    0.0
PERFSTAT
           555       0    5.1     1.0        1,434        0          0    0.0
UNDOTBS1
             0       0    0.0                1,465        0          0    0.0
SYSTEM
           507       0 ######     6.8           56        0          0    0.0
FSDPDM_SP
            85       0 ######     3.2          136        0          0    0.0

 

Related notes on reading statspack reports:

 
   
Oracle Grid and Real Application Clusters

See working examples of Oracle Grid and RAC in the book Oracle Grid and Real Application Clusters.

Order directly from Rampant and save 30%. 
 

 


 

 

��  
 
 
Oracle Training at Sea
 
 
 
 
oracle dba poster
 

 
Follow us on Twitter 
 
Oracle performance tuning software 
 
Oracle Linux poster
 
 
 

 

Burleson is the American Team

Note: This Oracle documentation was created as a support and Oracle training reference for use by our DBA performance tuning consulting professionals.  Feel free to ask questions on our Oracle forum.

Verify experience! Anyone considering using the services of an Oracle support expert should independently investigate their credentials and experience, and not rely on advertisements and self-proclaimed expertise. All legitimate Oracle experts publish their Oracle qualifications.

Errata?  Oracle technology is changing and we strive to update our BC Oracle support information.  If you find an error or have a suggestion for improving our content, we would appreciate your feedback.  Just  e-mail:  

and include the URL for the page.


                    









Burleson Consulting

The Oracle of Database Support

Oracle Performance Tuning

Remote DBA Services


 

Copyright © 1996 -  2017

All rights reserved by Burleson

Oracle ® is the registered trademark of Oracle Corporation.

Remote Emergency Support provided by Conversational