Don't let script kiddies play with Silver Bullets Oracle Database Tips by Donald Burleson

 

When publishing about the powerful new holistic features of Oracle (such as the 11g SQL Performance Advisor), it's always important to remember that these advanced tools are perilous in the hands of inexperienced or inept DBA's.   Also, see my related article on how Dangerous Dilettantes Destroy Databases!

 

Back before Oracle introduced their easy GUI interfaces and intelligent advisors, it was much more difficult for neophytes to pose as Oracle experts and human error was much lower than it is today.  Oracle Corporation says that human error is the greatest threat to database stability, and in my experience running an Oracle emergency support center, the real percentage is closer to 90%.

 

The problem of inexperience and Oracle outages due to human error has several root causes.  On one hand, we see desperate job candidates (many from foreign countries) who "puff" their job experience, often claiming that their unpaid years of dabbling with Oracle on a PC constitutes real Oracle experience. Some exaggerate their levels of understanding while some will outright lie.

 

Posers and puffers can be identified and disqualified at the job interview

 

Puffers and dilettantes are easily spotted with the proper job interview tests, but the most dangerous of all are the fraudsters.  In the global Oracle marketplace we see an alarming increase in "fake Oracle experts" who fabricate their experience.  See my related notes on evaluating Oracle professionals for dishonesty.

 

The dangers of phony Oracle experts

 

While some "questionable" Oracle experts carefully hide their resumes, credentials and verifiable work history, other script kiddies resort to outright deception.  Consider Ed Haskins, a poser who created Robert Allen, a fake Oracle expert, giving him all sorts of great credentials.  Haskins used a stock photo, built an elaborate deception scheme, all targeted toward deceiving the IT management community.  Here is the actual text from Haskins charade:

 

Robert Allen Oracle Security Expert

My name is Robert Allen, and I?m the Founder of OraSecure, Inc.  With over 9 years of trial and error from painstaking research, I?m now acknowledged as a promising top expert in the field of Oracle database security. For the past 7 years, I've consulted dozens of Fortune 1000 companies on all areas of Oracle Database Administration.  The areas I've focused my most attention to are Oracle Security Features and Security Audits.  I've personally audited over 400 Oracle instances over the years, as well as taught and mentored more than 80 DBAs to do the same.

A excerpt from an Oracle Phishing site

For details on this issue, read here from eWeek how I helped to uncover this scammer.

 

Lets take a closer look at these perils of inexperience and see ways that the savvy Oracle DBA manager can minimize the risk of hiring qualified Oracle professionals with verifiable credentials.

 

Playing Russian Roulette with Silver Bullets

 

In a real-world corporate shop, one of the first things that a junior DBA learns is that holistic tuning is essential to their effective job performance.  A typical DBA may have dozens of systems to support, and there is no time to waste.

 

Achieving the proper baseline settings (based on the system's historical workloads) is the first step in all Oracle tuning efforts.  Should the DBA fail to optimize their system for it's workload, SQL tuning can be un-done with subsequent changes to any of the Silver Bullets, changes that could alter the execution plans of thousands of SQL statements:

• SGA pool sizes (shared_pool_size, db_23k_cache_size, etc.) - Pool sizes have a dramatic effect on Oracle performance.
   

• Materialized Views - MV can pre-join popular tables together and prefetch commonly-referenced summary data.
   

• Indexing - Intelligent indexing ensures SQL query completion with a minimum amount of buffer gets.
   

• Optimizer knobs (optimizer_mode, Optimizer_index_cost_adj) - Changing these global parms can have a profound impact on Oracle performance (both good and bad).
   

• I/O subsystem changes (RAID level, multiple blocksizes, SSD) - I/O remains the most significant performance bottleneck in most production databases.
   

• Optimizer statistics - Applying CBO histograms can have a huge benefit to an entire system.

These "broad brush" settings change the way that Oracle processes data in a profound way, and a single change can either make you a hero or get you fired.

 

Until the advent of the 11g SQL Performance Advisor , the DBA had to manually collect representative workloads (using STATSPACK or AWR), and verify all global changes in a test environment.  This mission critical tuning task requires knowledge of collecting and analyzing workloads with statistical techniques, and for advanced DBA's, knowledge of Oracle Data Mining. 

 

Even with the SPA, Oracle professionals must have extensive knowledge of Oracle internals and optimization. 

 

The deceptive "ease" of Silver Bullets

 

Lewis makes a very important point about the illusion of an "Easy Fix".  To a beginner, holistic changes appear "easy", since they only see the final act, making what appears to be a "quick fix" that has a profound impact on performance (sadly, this is too-often a "bad" impact when put into the hands of an inexperienced DBA).  In the real-world, these deceptively "easy" Silver Bullets are the end-result of weeks of careful planning.  Let's take a closer look at the importance of real-word job experience when using the powerful and dangerous holistic tuning activities.

 

Silver Bullets are only for worthy scholars

 

Lewis' points about an increasingly naive DBA community are right on the money, especially about the damage that can occur when dilettantes attempt holistic changes that are beyond their experience level. 

 

These inexperienced DBA's" can be very dangerous when using the powerful Silver Bullet techniques.  In my experience doing Oracle emergency support I see case after case where a script kiddie has blindly run a powerful Oracle script, just to "see what it does", crashing an entire database.

 

As a responsible author, I have a stern warning to junior DBA's about about getting in over your head.  The holistic techniques in "Oracle Silver Bullets", are intended only for experienced Oracle DBA's, working professionals who understand the holistic concepts of Oracle tuning.  To learn best practices for Oracle tuning, see my book "Oracle Tuning: The Definitive Reference".

---

Reader Comments:

 

I think it's great how you addressed the 'silver Bullets? issue. It's funny, I think the people that have a problem with your 'silver bullets? are those that throw common sense out the window when they read it.

?Well, it's in the book, so I?ll run it. Oh no, it broke my database! Let's blame Burleson! How was I supposed to know it had a 'shutdown immediate? in it?! I thought I could run it on prod!?

I guess it goes to show that we really are in the ?Age of the Disclaimer? nowadays. Coffee is hot, scripts should be tested. Go figure, huh? So maybe some of those scripts don't always ?fix it?, and maybe some of them could make performance worse. It's called 'trial and error.? They?re silver bullets, not miracles.

It reminds me of that song from South Park - The Movie, ?Blame Canada.?

'should we blame the matches, should we blame the fire?

Or the doctors who allowed him to expire? HECK NO!

Blame Canada, blame Canada!?

We could change that to these lyrics:

'should we blame Oracle? Should we blame the net?

Or the script I ran but haven't tested yet? HECK NO!

Blame Burleson! Blame Burleson!?
 

Steve Karam, OCM, Oracle ACE

 

---

There is one other egregious made by poseurs that is really easy to
spot.  They come up with an oracle 'best practices' document that has
no relationship to the reality of large Oracle environment.  One
classic 'best practice' is to use the ip address in the listener.ora
and tnsnames.ora instead of a dns alias.  Try doing that on a site
that has multiple domains...

 

An Oracle ACE