Oracle I/O optimization on Sun Solaris Oracle Database Tips by Donald Burleson

Using direct I/O with Oracle keeps CPU consumption down and bypasses the double buffering within the Solaris Filesystem, buffers.  For complete details, see the book "Oracle Disk I/O Tuning", by Rampant TechPress.

Glen Faucett has a great note on the benefits of large RAM memory on Solaris, showing a 2x speed improvement when bypassing the buffer file system I/O.  Unbuffered I/O is a must with Oracle, using raw partitions or ASM:

A quick experiment was conducted on Oracle 10gR2 on a large memory machine (192GB).

• 1st test: DB cache was set to 1GB and the database was mounted on a buffered file system.
• 2nd test: DB cache was set to 50GB and the database was mounted direct - NOT buffered.

A 46GB table was populated, indexed, and then queried by 100 processes each requesting a range of data. A single row was retrieved at a time to simulate what would happen in an OLTP environment. The data was cached so that no IO occurred during any of the runs. When the dust had settled, the Oracle buffer cache provided a 2X speedup over buffered file systems. There was also a dramatic decrease in getmaps, xcalls, and system CPU time. The table below shows the results.

Cache	OS	Rows/sec	getmaps/sec	xcalls/sec	Usr	sys
FS	S9	287,114	86,516	2,600,000	71	28
DB	S9	695,700	296	3,254	94	5

 

Glen Faucett also notes tips for setting direct I/O on Sun Solaris Oracle servers using filesystemio_options=setall and forcedirectio:

The first step to avoiding buffered IO is to use the "FILESYSTEMIO_OPTIONS" parameter. When you use the "SETALL" option, this sets all the options for a particular filesystem to enable directio or async IO. Setting the FILSYSTEMIO_OPTIONS to anything other than "SETALL" could reduce performance.

With UFS, the only way to bypass the page cache is with directio. If you are using Oracle 9i or greater, then set the FILESYSTEMIO_OPTIONS=SETALL init.ora parameter. This the preferred way of enabling directio with Oracle. With this method, Oracle uses an api to enable directio when it opens database files. This method allows you to still use buffered IO for operations like backup and archiving. If you are using Oracle 8i, then the only way to enable directio with UFS is via the forcedirectio mount option.