 |
|
Interpreting Oracle RAC STATSPACK/AWR performance reports
Oracle 11g Tips by Donald Burleson
|
Interpreting an Oracle RAC (Real
Application Clusters) STATSPACK/AWR performance reports can be
tricky, and it's quite different from reading a standard
STATSPACK report.
The most obvious difference is the
shared "cache fusion" component, and it can skew the performance
report data, often hiding the root cause of a performance
problem.
For example, consider this RAC
STATSPACK report:
Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
--------------- ---------------
Redo size: 7,134.04 8,461.68
Logical reads: 43,060.81 51,074.43
Block changes: 21.39 25.38
Physical reads: 154.16 182.84
Physical writes: 79.52 94.31
User calls: 68.52 81.28
Parses: 22.02 26.12
Hard parses: 0.20 0.24
Sorts: 24.93 29.57
Logons: 1.29 1.53
Executes: 29.36 34.83
Transactions: 0.84
% Blocks changed per Read: 0.05 Recursive Call %: 18.71
Rollback per transaction %: 21.31 Rows per Sort: 37.21
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: 100.00 Redo NoWait %: 100.00
Buffer Hit %: 99.82 In-memory Sort %: 100.00
Library Hit %: 99.52 Soft Parse %: 99.09
Execute to Parse %: 25.00 Latch Hit %: 99.92
Parse CPU to Parse Elapsd %: 2.98 % Non-Parse CPU: 99.32
Shared Pool Statistics Begin End
------ ------
Memory Usage %: 81.11 83.06
% SQL with executions>1: 71.32 71.46
% Memory for SQL w/exec>1: 51.65 53.13
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ %
Total
Event Waits Time (s) Ela
Time
-------------------------------------------- ------------ -----------
--------
PX Deq Credit: send blkd 14,513 1,864
46.27
CPU time 1,002
24.89
library cache pin 413 486
12.07
PX qref latch 254 117
2.90
PX Deq: Table Q Sample 1,272 110
2.74
Here we see high buffer hit ratios (which
is deceptive), and note the "PX" entries in the top-5 timed events,
which indicate that parallelism is turned-on and being used. This,
in turn, indicates the presence of full-scan activity (multiblock reads
from large-table full-table-scans and/or index fast-fast-full scans).
Because this is an OLTP database, we next
verify that parallel_automatic_tuning is turned-on, in this case
a very bad thing, because it has influenced the database to make
full-scan access more attractive than it should be.
parallel_adaptive_multi_user TRUE
parallel_automatic_tuning TRUE
We also see relatively high pinging on
this RAC node. This database cannot be partitioned (load balanced)
by end-user usage (placing different "types" of end-users on different
nodes), and there is the inevitable inter-node pinging as common data is
requested between active nodes.
Global Cache Service -
Workload Characteristics
-----------------------------------------------
Global cache hit ratio: 0.2
Ratio of current block defers: 0.0
% of messages sent for buffer gets: 0.2
% of remote buffer gets: 0.0
Ratio of I/O for coherence: 0.6
Ratio of local vs remote work: 6.0
Ratio of fusion vs physical writes: 0.0
The core performance
problems:
Upon closer inspection of this database we
would the following problems, none of which were clear from the
high-level summary statistics:
-
Unnecessary
parallelism - Each node has only two CPU's and parallel_automatic_tuning
required turning off.
-
Missing indexes -
Missing indexes were invoking unnecessary large-table full-table scans.
-
I/O bottlenecks -
A closer inspection of the I/O subsystem reveals super-slow I/O timings and
clear disk enqueues.
Obfuscated details
Unlike a single-node STATSPACK/AWR report,
a RAC statspack report is more subtle, and the top problems are not
readily apparent from the top event statistics. In this case, even
though the data buffer hit ratio is 99%, there is still a huge I/O
contention issue, which can be remedied with a larger db_cache_size:
Tablespace
------------------------------
Av Av Av Av Buffer Av
Buf
Reads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)
-------------- ------- ------ ------- ------------ -------- ----------
------
VPPP_SP
37,754 10 ###### 4.7 137 0 7
1.4
CBPO_SP
33,015 9 ###### 1.2 15 0 716
2.0
TEMP
9,265 3 1.7 30.1 9,350 3 0
0.0
PRODMAN
12,560 3 ###### 4.0 125 0 0
0.0
PRODDTAI
1,219 0 7.5 1.0 990 0 0
0.0
PERFSTAT
555 0 5.1 1.0 1,434 0 0
0.0
UNDOTBS1
0 0 0.0 1,465 0 0
0.0
SYSTEM
507 0 ###### 6.8 56 0 0
0.0
FSDPDM_SP
85 0 ###### 3.2 136 0 0
0.0
Related notes on reading statspack
reports:
