Don't let script kiddies play with Silver Bullets
Oracle Database Tips by Donald Burleson
When publishing about the powerful
new holistic features of Oracle (such as the 11g
Performance Advisor), it's always important to remember that
these advanced tools are perilous in the hands of inexperienced
or inept DBA's. Also, see my related article on how
Dangerous Dilettantes Destroy
Back before Oracle introduced
their easy GUI interfaces and intelligent advisors, it was much
more difficult for neophytes to pose as Oracle experts and human
error was much lower than it is today. Oracle Corporation
says that human error is the greatest threat to database
stability, and in my experience running an Oracle emergency
support center, the real percentage is closer to 90%.
The problem of inexperience and
Oracle outages due to human error has several root causes. On one hand, we see desperate job candidates
(many from foreign countries) who
"puff" their job experience, often claiming that their unpaid
years of dabbling with Oracle on a PC constitutes real Oracle experience. Some
exaggerate their levels of understanding while some will
Posers and puffers can be identified and disqualified at the
Puffers and dilettantes are easily
spotted with the proper job interview tests, but the most
dangerous of all are the fraudsters. In the global Oracle
marketplace we see an alarming increase in "fake Oracle experts"
who fabricate their experience. See my related notes on
evaluating Oracle professionals for dishonesty.
The dangers of
phony Oracle experts
While some "questionable" Oracle
experts carefully hide their resumes, credentials and verifiable work
history, other script kiddies resort to outright deception.
Consider Ed Haskins, a poser who
created Robert Allen, a fake Oracle expert, giving him all
sorts of great credentials. Haskins used
a stock photo, built an elaborate deception scheme, all targeted
toward deceiving the IT management community. Here is the
actual text from Haskins charade:
Oracle Security Expert
My name is Robert Allen,
and I?m the Founder of OraSecure, Inc. With over 9
years of trial and error from painstaking research, I?m now
acknowledged as a promising top expert in the field of
Oracle database security.
For the past 7 years, I've consulted dozens of Fortune 1000
companies on all areas of Oracle Database Administration.
The areas I've focused my most attention to are Oracle
Security Features and Security Audits.
I've personally audited over 400 Oracle
instances over the years, as well as taught and mentored
more than 80 DBAs to do the same.
A excerpt from an Oracle
For details on this issue, read
here from eWeek how I helped to uncover this scammer.
Lets take a closer look at these perils
of inexperience and see ways that the savvy Oracle DBA manager
can minimize the risk of hiring qualified Oracle professionals
with verifiable credentials.
Playing Russian Roulette with Silver Bullets
In a real-world corporate shop,
one of the first things that a junior DBA learns is that
holistic tuning is essential to their effective job performance.
A typical DBA may have dozens of systems to support, and there
is no time to waste.
Achieving the proper baseline settings (based on
historical workloads) is the first step in all Oracle tuning efforts.
Should the DBA fail to optimize
their system for it's workload, SQL tuning can be un-done
with subsequent changes to any of the Silver Bullets, changes
that could alter the execution plans of thousands of SQL
SGA pool sizes (shared_pool_size,
db_23k_cache_size, etc.) - Pool sizes have a dramatic effect on Oracle
Materialized Views - MV can pre-join
popular tables together and prefetch commonly-referenced summary data.
Indexing - Intelligent indexing ensures
SQL query completion
with a minimum amount of buffer gets.
Optimizer knobs (optimizer_mode,
Optimizer_index_cost_adj) - Changing these global parms can have a
profound impact on Oracle performance (both good and bad).
I/O subsystem changes (RAID level,
multiple blocksizes, SSD) - I/O remains the most significant performance
bottleneck in most production databases.
Optimizer statistics - Applying CBO
histograms can have a huge benefit to an entire system.
These "broad brush" settings
change the way that Oracle processes data in a profound way, and
a single change can either make you a hero or get you fired.
Until the advent of the 11g
Performance Advisor , the DBA had to manually collect
representative workloads (using STATSPACK or AWR), and verify all
global changes in a test environment. This mission
critical tuning task requires knowledge of collecting and
analyzing workloads with statistical techniques, and for
advanced DBA's, knowledge of
Oracle Data Mining.
Even with the SPA, Oracle
professionals must have extensive knowledge of Oracle internals
The deceptive "ease" of Silver Bullets
Lewis makes a very important point about the
illusion of an "Easy Fix". To a beginner, holistic changes appear "easy",
since they only see the final act, making what appears to be a "quick fix" that
has a profound impact on performance (sadly, this is too-often a "bad" impact
when put into the hands of an inexperienced DBA). In the real-world, these
deceptively "easy" Silver Bullets are the end-result of weeks of careful
planning. Let's take a closer look at the importance of
real-word job experience when using the powerful and dangerous holistic tuning
Silver Bullets are only for worthy scholars
Lewis' points about an increasingly naive DBA
community are right on the money, especially about the damage that can occur
when dilettantes attempt holistic changes that are beyond their experience
These inexperienced DBA's" can be very dangerous when using the powerful Silver Bullet techniques. In my
emergency support I see case after case where a script kiddie has blindly run a
powerful Oracle script, just to "see what it does", crashing an entire database.
As a responsible author, I have a stern warning to
junior DBA's about about getting in over your head. The holistic
techniques in "Oracle
Silver Bullets", are intended only for experienced Oracle
DBA's, working professionals who understand the holistic
concepts of Oracle tuning. To learn best practices for Oracle tuning, see
my book "Oracle
Tuning: The Definitive Reference".
I think it's great how you addressed the 'silver Bullets? issue. It's funny, I
think the people that have a problem with your 'silver bullets? are those that
throw common sense out the window when they read it.
?Well, it's in the book, so I?ll run it. Oh no, it broke my database! Let's
blame Burleson! How was I supposed to know it had a 'shutdown immediate? in
it?! I thought I could run it on prod!?
I guess it goes to show that we really are in the ?Age of the Disclaimer?
nowadays. Coffee is hot, scripts should be tested. Go figure, huh? So maybe some
of those scripts don't always ?fix it?, and maybe some of them could make
performance worse. It's called 'trial and error.? They?re silver bullets, not
It reminds me of that song from South Park - The Movie, ?Blame Canada.?
'should we blame the matches, should we blame the fire?
Or the doctors who allowed him to expire? HECK NO!
Blame Canada, blame Canada!?
We could change that to these lyrics:
'should we blame Oracle? Should we blame the net?
Or the script I ran but haven't tested yet? HECK NO!
Blame Burleson! Blame Burleson!?
Steve Karam, OCM, Oracle ACE
There is one other egregious made by poseurs that is really easy to
spot. They come up with an oracle 'best practices' document that has
no relationship to the reality of large Oracle environment. One
classic 'best practice' is to use the ip address in the listener.ora
and tnsnames.ora instead of a dns alias. Try doing that on a site
that has multiple domains...
An Oracle ACE