Oracle Database Tips by Donald Burleson

The History of I/O Bandwidth

Historically, RAM I/O bandwidth grows one bit every 18 months, making the first decade of the 21st Century the era of 64-bit RAM technology:

1970's                       8  bit
1980's                     16  bit 
1990's                     32  bit 
2000's                     64  bit 
2010's                     28  bit 

As of 2006, the vast majority of hardware vendors (Sun, HP, IBM, UNISYS and Dell) offer 64-bit servers with far higher bandwidth than their ancient 32-bit predecessors.

However, RAM is quite different from other computer hardware such as disk and CPU.  Unlike CPU speed, which improves every year, RAM speed is constrained by the physics of silicon technology. Instead of speed improvements, there is a constant decline in price.  CPU speed also continues to outpace RAM speed and this means that RAM sub-systems must be localized to keep the CPUs running at full capacity.

"Moore's Law" states that CPU speed will constantly improve while process costs fall.  Unfortunately, this is not the case for RAM and disk, and Figure 2.1 shows that the "real" disk speeds have not improved significantly in the past 15 years:

As shown in Figure 2.1 CPU speed continues to double every few years, while the speed of disk and RAM cannot boast such a rapid rate of speed improvements.

For RAM, the speed has increase from 50 nanoseconds (one billionth of a second) to two nanoseconds, a 25x improvement over a 30-year period.  At access speeds of two-billionths of a second, today's DDR SDRAM is stressing the limits of silicon technology, and it's unlikely that significantly faster speeds will be seen in the next decade (Table 2.1).

Year            RAM Type                  Access Speed
1987           FPM                            50ns
1995           EDO                            50ns
1997           SDRAM                      15ns
1998           SDRAM                      10ns
1999           SDRAM                      7.50 ns
2000           DDR SDRAM             3.75 ns
2001           DDR SDRAM             3.00 ns
2002           DDR SDRAM             2.30 ns
2003           DDR SDRAM             2.00 ns

It is very clear than CPU speed will continue to outpace RAM speed and this has important ramifications for Oracle database processing.  The advent of Non-Uniform Memory Access (NUMA) is predicated on the fact that data storage (RAM) must be localized as close to the CPU as possible to maximize throughput (Figure 2.2):

NUMAhas been available for years in high-end UNIX servers running SMP(symmetric multi-processor) configurations. The vendors know that NUMA technology allows for faster communication between the distributed RAM in a multi-processor server environment.  NUMA is supported by Linux and Windows Advanced Server 2003 and is a feature of the Intel Itanium2 chipset, which is used in the latest Oracle server blades for Oracle Grid computing.

Oracle 10g has become NUMA -aware and the database engine can now exploit the high-speed L2 cache on the latest SMP servers.  According to David Ensor, a recognized Oracle tuning expert, author, and Former Vice President of the Oracle Corporation's Performance Group, the inordinate increase in CPU power has shifted the bottleneck of many systems to disk I/O, as the disk technology fails to keep-up with CPU.

The above book excerpt is from:

Oracle RAC & Tuning with Solid State Disk

Expert Secrets for High Performance Clustered Grid Computing

ISBN 0-9761573-5-7

Donald K. Burleson & Mike Ault

http://www.rampant-books.com/book_2005_2_rac_ssd_tuning.htm