The Best Oracle
Resource on the Web
Costs and Benefits for High Availability Oracle9i
by Donald K. Burleson
Please see these updates to Oracle hardware
costs and benefits of server deconsolidation.
development managers are more challenged than ever to find the appropriate tools
and technologies for their mission-critical Oracle databases. With the explosive
interest in e-commerce, end users have high expectations. They demand systems on
the Web that can support thousands of concurrent transactions per second while
at the same time providing sub-second response time and continuous availability.
Many of the
world's most widely-read e-commerce systems use Oracle databases. Behemoths such as Amazon
and eBay use Oracle databases and support mind boggling transaction rates. And
they make it look easy. But the reality is that a tremendous amount of manual
effort goes into creating a Web architecture that provides a continuously
available solution for Oracle databases on the Web.
The Evolution of High-availability Systems
Over the past
five years, the demands upon Oracle e-commerce systems have been radically
changing. Consumers on the Web are no longer content with systems that are down
periodically and systems that don't provide sub-second response time.
Hence, it is the
challenge of the Oracle professional to create a database architecture that is
continually available, while at the same time providing blistering response
times, even when transaction rates exceed thousands of transactions per second.
eCommerce customer intolerance, continuous availability is a major goal of any
Oracle database that is deployed over the Internet. But how to we go about
getting continuous availability?
successful companies we can get an idea of some of the methods they use in order
to make sure that their systems are always available. Successful companies
employed replicated databases, such that the failure of any component of a
single database, including disk, RAM, CPU, or network, will not have a
catastrophic failure on the entire system. In addition to replication,
mechanisms are created to allow in-flight transactions to be automatically
reconnected to another fully replicated database management system, which is
most often located at a different geographical location in the primary database
This approach of
having replicated Oracle systems geographically distributed across the globe
also insures that connectivity over the Internet is always done in a uniform
fashion regardless of where in the world the connection is started.
The challenge in
this type of architecture is to ensure that all of the replicated systems remain
synchronized. Prior to the advent of Oracle9i, many e-commerce vendors used a
shadowed standby database. In a standby database, redo log images were fed from
the primary system into a standby database that was always in recovery mode. If
there was a failure of the primary server, the last redo logs could be fed into
the standby database, which could then be quickly started in takeover the
processing load. Of course, this approach had the serious drawback of requiring
at least 15 minutes downtime in case of primary hardware failover.
replication was also very little help, because snapshot refresh intervals were
not instantaneous enough to keep all the systems exactly synchronized. To
address this requirement for instantaneous system replication, third party
products such as Quest shareplex were developed in order to take Oracle redo log
images directly from Oracle's RAM wall buffer and generate SQL statements which
were then fed to the replicated databases.
of Continuous Availability
entire systems is a great solution for providing continuous availability we also
have to remember that there are replicated components within each individual
Oracle system. Let's take a look at each one of these components individually
and hope to understand how this kind of replication reduces the probability of
- Over the past decade, hardware devices have become increasingly more reliable.
However, in addition to this reliability, we also see that many disk storage
vendors are implementing sophisticated RAID technologies in order to ensure that
they have redundant backups of any disk device. This replication takes the form
of disk mirroring, which is also known as raid 0, and some companies will even
triple mirror their disks. Should an individual disk fail, many disk storage or
raise will telephone the vendor directly and the vendor will arrive at the
machine room and replace and re-synchronize the broken disk device without any
service interruption. For companies that are willing to investing triple
mirrored disks, the meantime between failures is expressed in decades, and these
companies can confidently assume that a failure of their disk I/O subsystem is
not going to cause a service interruption for their mission-critical system.
failure - Within any hardware architecture, automated failovers of CPU in
RAM memory are becoming increasingly sophisticated. In an SMP configuration for
UNIX, you may have 8, 16, 32, or even 64 individual CPUs configured as a
symmetric multiprocessing configuration. Should anyone of these individual
processors fail, the UNIX operating system is directed to bypass the bad CPU and
continue processing on the remaining processors. Just like with disk devices,
many hardware vendors have software that will automatically notify the vendor of
the hardware, and replacements can be installed without any service
Inside High-availability for Oracle Applications
Within the IT
industry there are two metrics that are commonly used to measure the time
between failures and the time required to recover from these failures. The first
is the meantime between failures which is commonly abbreviated MTBF. The MTBF
value specifies a statistical estimate on how long the expected configuration
will be able to run without unplanned data outage. The numbers for the MTBF can
be gathered from hardware vendors, and mathematical techniques can be applied
replicated systems to accurately predict the duration between unplanned data
measure is the meantime to recovery which is commonly abbreviated MTTR. The MTTR
was developed to measure the amount of downtime that will be incurred should
there be a catastrophic failure of some component within the computer system. Of
course, for continuously available systems the MTTR should be equal to zero, but
many companies who are unable to invest many millions of dollars in continuous
availability or often content to have systems with an MTTR which is measured in
less than 30 minutes.
- MTBF - Mean
time Between Failures
vendors can provide MTBF for all hardware.
- You must
measure your companies MTBF, noting causes of outages (human, hardware,
- MTTR - Mean
Time To Recovery
MTTR depends of nature of failure
vendors can guarantee MTTR, and this should be spelled-out in a binding
What Is Your Cost of Downtime?
The costs are a
huge issue when considering high-availability solutions, and many companies
carefully quantify all costs associated with database downtime. These costs
- Revenue loss -
Up to $100,000 per minute
loss - Factory workers sitting about
loss - Remember eBay?
However, 100 percent availability is very expensive, and most companies must
choose a tradeoff between high-availability and cost. The figure below shows the
average costs of database downtime for different industries (source: Giga
We take a look at the costs associated with system downtime, many Oracle
managers make a mistake of measuring only the actual tangible costs of having
the system down. Of course, having a mission critical system down is going to
result in lost revenue from sales, worker of productivity losses, but you always
have to keep in mind the intangible costs associated with the system downtime.
These intangible costs can include loss of customer goodwill which is often
measured in millions of dollars. One has to look no farther than the eBay system
to understand the dramatic impact it is unplanned system outage can have, not
only in lost revenue but on lost goodwill from the customer base.
We generally see
the costs of unplanned downtime broken down by industry experts. Leading the
pack are the financial markets for which unplanned downtime can run into many
millions of dollars per hour. We also see high cost for other financial systems
such as credit card companies, who have losses that can be expressed in many
hundreds of thousands of dollars per minute. Manufacturing systems are also
subject to very high cost, but these are direct costs in terms of lost sales,
Rather, lost sales for manufacturing operations often have to do with
interruption of the manufacturing process, and lost wages paid factory workers
who were no longer able to do their jobs.
The Causes of
Unplanned System Outages
We take close
look at the causes of unplanned systems outages, we see that the majority of
outages are equally divided between hardware failure and human error. Hardware
failure accounts for approximately one-third of all total system outages, with a
little more than one-third of system outages being attributable to some kind of
human failure. This human failure can take many forms, from a software patch
that was not thoroughly tested before moving it into production, or from an
operator error in the machine room. It is very easy for people in the IT
industry to add additional hardware layers in order to ensure continuous
availability up hardware level, but active steps must be taken whenever possible
to reduce the amount of human intervention.
If practice, many
companies are actively working to create Oracle Web system architectures that
minimize the amount of human intervention, by automating mundane tasks such as
state mounting, tape cataloging, database recovery procedures, and any other
procedures which might require the intervention of fallible human beings.
System Outages: Gartner Group
If you want
super-fast failover without any service interruption, using RAC with TAF, or a
customized webserver failover will provide this protection. Of course, there are
significant costs associated with the setup and testing of these tools.
Continuous Availability Costs
We take a look at
the specific costs of continuous availability systems, we see a split between
hardware costs and the costs of human beings. On the hardware end, we know that
massively parallel replicated machines can be quite expensive, and we also know
that mirroring disks will double the amount of disk devices required for the
On the human
factors and, we also see a significant expensive highly trained computer systems
professionals such as Oracle DBAs who were charged with building and testing the
continuous availability architecture for the system.
In fact, the
set-up and testing costs are so high for Oracle's Real Application Clusters with
Transparent Application Failover, that many companies in the real world are
considering building customized Web server failover solutions. Many of these
customized Web server failover solutions utilize an Oracle extension to the
Apache Web server, where the Web server programmer can automatically cache and
restart fail transactions should any single database engine stop responding.
Surrounding Continuous Availability
for companies to remember the continuous availability is not inexpensive. While
the software components are readily available, the human effort for the
installation, set up, and testing of continuous availability systems can be very
expensive and time-consuming. The set-up for Oracle time for using Real
Application Clusters, and the Transparent Application Failover product can run
in the many hundreds of hours of time for the installation, configuration, and
testing of these tools.
companies must take a look at their relative costs of downtime, and choose a
high availability or continuous availability solution that meets their economic
loss of downtime.
On one end of the
spectrum we know that large financial institutions have down times are expressed
in hundreds of thousands of dollars per minute. These kinds of companies cannot
tolerate any kind of unexpected outages, and they are willing to invest the
millions of dollars necessary in order to ensure that their systems are always
available, even in cases of catastrophic tragedies such as the events of
Moving down the
spectrum, companies they can tolerate a small amount of unplanned downtime
during the year can often use less expensive continuous availability solutions.
The solutions would involve standby databases, or traditional database recovery
using Oracle's RMAN utility.
In any case, it
is the job of Oracle management to weigh the costs of unexpected downtime with
the cost of additional layers of continuous availability protection. For systems
to be truly 24 by 7, many replicated servers across the globe may have to be
installed and sophisticated techniques put into place to ensure that these
replicated databases remain synchronized. For those companies willing to
tolerate a meantime between failures of less than one decade, replicated systems
can be placed within the same machine room over high-speed interconnect, and
these two systems can be only a few feet each other. Each system feeds
continuous updates to the other in order remain synchronized, and both systems
can be open and available for use by the end-user community.