Oracle Server Consolidation tips

Please see these updates to Oracle hardware architectures and the costs and benefits of server deconsolidation.

It is ironic that what was old is now brand-new again. When I started in "data processing" in the 1970's I was in an age when a single server would commonly host a dozen databases.

Things were much simpler in the pre-relational days of "glass house" servers. With IMS or IDMS on a mainframe, I would only need to perform an upgrade once, and I could easily gen-in the upgrades to dozens of "instances," all at my leisure. Back them there was "the" DBA, and few shops needed more than one DBA to manage all of the databases.

This all changed in the late '80s with the introduction on mini-computers. These cheap servers could be purchased for as little as $30k, a bargain when compared to the $3 million dollar cost of a mainframe. As minicomputers evolved into the UNIX-centric Oracle servers of the 1990s some shops found themselves with hundreds of servers, one for each Oracle database.

We also saw the age of multitiered applications in which a large system might be comprised of an Web server layer, an application server layer, and a database layer, each with dozens of individual servers (refer to figure 1).

Figure 1: The multi-server architecture of the 1990s.

For Oracle databases, the multi-server architecture employs Oracle Real Application Clusters, a processing architecture that allows multiple Oracle instances (often on separate servers) to access a common Oracle database.

Now we are seeing a resurgence in popularity of large monolithic servers.

These mainframe-on-a-stick servers may contain 16, 32, or even 64 CPUs and have processing capabilities that dwarf the traditional mainframe ancestors on the 1980s.

Many of these new pseudo-mainframes use the latest Intel Itanium II 64-bit chipset (refer to figure 2). Larry Ellison mentioned the Intel chips during his presentation at OracleWorld 2003 in San Francisco when he noted, :If you want the world's fastest processors, you are going to be forced to pay less."

Figure 2: The Intel Itanium II chipset [source Intel].

Some vendors are assembling these Intel processors into configurations that are surprisingly similar to their mainframe ancestors (refer to figure 3).

Figure 3: The UNISYS 16-way ES7000 Server.

These machines are capable of blistering performance and a recent UNISYS benchmark exceeded 250,000 transactions per minute on a Windows-based server using Oracle10g.

While the details of the benchmark are included in the link, the high speed was primarily as result of these factors:

• Multiple blocksizes

• 115 gigabyte total data buffer cache

• 78 gigabyte KEEP pool

• 16 CPU Server : Each a 64-bit Itanium 2 Processor

As a side-note, here are a few of the Oracle10g parameters that were used in the benchmark:

db_block_size = 2048
db_16k_cache_size = 15010M
db_8k_cache_size = 1024M
db_cache_size = 8096M
db_keep_cache_size = 78000M
db_recycle_cache_size = 15400M
db_writer_processes = 4
pga_aggregate_target = 0
sort_area_size = 65525

Also of interest is the Oracle10g Windows registry entry ora_lpenable for large page support:

"ORA_LPENABLE" = "1"

This tells Oracle to use the Large Page feature (available in Oracle9i 64-bit for Windows and Oracle10g 32-bit and 64-bit for Windows).

The Age of Consolidation

There are naysayers who argue that it is not a good idea to throw everything into a single server because it introduces a single point of failure. In reality, these large systems have redundant everything, and with the use of Oracle Streams for replication at different geographical locations, they are virtually unstoppable.

Everything from disk, CPU RAM, and internal busses are fault-tolerant and redundant, making the monolithic approach very compelling to large corporations. We also see the rapid depreciation rates for servers contributing to the move toward server consolidation.

Three year-old Oracle servers that cost over $100k and now worth less than $5k and IT managers are looking at a single-server approach for a variety of reasons:

• Lower costs : Monolithic server are extremely good at sharing computing resources between applications.

• Lower maintenance : Instead of maintaining 30-copies of Oracle and the OS, you only need to manage a single copy.

Part of the problem with single-server Oracle systems was the deliberate over-allocation of computing resources. Each system experiences periodic processing "spikes," and each server had to be equipped with additional resources to accommodate the infrequent demands of applications. This led to a condition in which hundreds of Oracle servers had unused CPU and RAM resources that could not be easily shared. In the following, (refer to figure 4), we see an Oracle server that has to have 70 percent more RAM than the average utilization, a significant waste of expensive resources, but required to maintain performance during infrequent bursts of activity.

Figure 4: Deliberate over-allocation of RAM resources on a single Oracle server.

Cost savings aside, we see other compelling reasons to consolidate dozens of Oracle instances onto a single server:

• Oracle server consolidation : Server consolidation technology can greatly reduce the number of Oracle database servers.

• Centralized management : A single server means a single copy of the Oracle software. Plus, the operating system controls resource allocation and the server will automatically balance the demands of many Oracle instances for processing cycles and RAM resources. Of course, the Oracle DBA still maintains control and can dedicate Oracle instances to a single CPU (processor affinity) or adjust the CPU dispatching priority (the UNIX "nice" command) of important Oracle tasks.

• Transparent high availability : If a CPU fails, the monolithic server can re-assign the processing without interruption. This is a more affordable and far simpler solution than Real Applications Clusters or Oracle9i Data Guard, either of which requires duplicate servers.

• Scalability : Using a single large server, additional CPU and RAM can be seamlessly added for increased performance.

• Reduced DBA workload : By consolidating server resources, the DBA has fewer servers to manage and need not be concerned with outgrowing their server.

So, what does this mean to the Oracle DBA? Clearly, less time will be spent installing and maintaining multiple copies of Oracle, and this will free-up time for the DBA to pursue more advanced tasks such as SQL tuning and database performance optimization. But the sad reality of server consolidation is that thousands of mediocre Oracle DBAs will loose their jobs to this trend. The best DBAs will continue to find work, but DBAs who were used for the repetitive tasks of installing upgrades on hundreds on small servers will be displaced.

A Peek into the Future

Over the next five years we will see other important changes in Oracle database administration, and this will forever change the way that Oracle DBAs perform their work:

• Fully-cached databases : Just as the UNISYS benchmark used 115 gigabyte data buffers, many Oracle systems will become fully cached.

• Solid-state Oracle : We will also see the advent of Solid-State Disk (SSD) as a faster replacement for the archaic spinning platters of magnetic-coated media.

• Back to the mainframe : The Oracle system of the future will run an entire corporate enterprise on two servers, a main server and a geographically distant failover server.

• Changing role of "the" DBA : A single DBA will be able to manage dozens of Oracle instances in a consolidated environment, and entire teams of DBAs will be dissolved, leaving a single "super DBA." This is reminiscent of the 1980s, when "the" DBA for a large corporation was required to have credentials (advanced degrees) and skills far exceeding the typical Oracle DBA of the late 1990s.

Having experienced the huge flood of need for Oracle DBAs in the early 1990s as the direct result of server de-consolidation, I welcome the return to the old days where I could manage dozens of Oracle instances within a single server environment.