Oracle Tips by Burleson

SCSI Bus Tips

SCSI buses on SUN can operate various bandwidths, speeds and throughputs (see table 3-2 to see the various bus specifications for SUN).  The prtconf Solaris command is used to report information that can be used to determine the speed of a particular SCSI device.  Most of the SCSI systems I have seen in recent years have been SCSI3 based.

Table 3-2: SUN SCSI Specifications

The scsi_options parameter can be set in the /etc/system file to limit bus speed or set other characteristics. You should check device documentation to determine if these settings need to be specified for a specific device.  The default scsi_options variable allows the widest range of capabilities that the SCSI host adapter can provide to be supported.

The default scsi_options value on Solaris 2.x works for both 5MB and 10MB devices.  The driver will negotiate with each device to determine if it is 10MB transfer capable or not.  If they are 10MB devices, 10MB transfer will be used.  If not, 5MB transfer will be used.

SCSI subsystem options - on SUN Solaris, a global word of options are available. The bits of the global word are broken down as:

The rest of the bits are reserved for future use.

The bits, which can be set are set using the hexadecimal values are shown in Table 3-3.

Table 3-3: SCIS Options Hexadecimal Values

The values for the various options are added to achieve the desired combination of features.  For example, a line in /etc/system that read:

        set scsi_options=0x3f8

means that the default options would be to allow WIDE SCSI, FAST SCSI, tagged commands, global parity, synchronous transfer, linked commands and global disconnect/reconnect, i.e. all currently supported options.

SCSI chains may be made of single-ended or differential connections.  They should not be mixed, as this may damage the equipment.  Differential connections permit longer chains, but the hardware is usually more expensive.  Single-ended chains must be less than 6 m in length; differential chains must be less than 20 m for synchronous connections or 25 m for asynchronous connections.

A Single ended SCSI connection uses "Normal" electrical signals, uses an open collector to the SCSI bus. The maximum length for SCSI-1 is a 6 meter cable with stubs of max 10cm allowed to connect a device to the main-cable. Most devices are single ended.

A differential SCSI connection uses two wires to drive one signal.  It has a maximum cable length of 25 meters.  Differential SCSI is electrically incompatible with single ended devices, allowed in SCSI-1 and upwards based systems.

The SCSI target numbers represent attachment points on the SCSI chain.  Each target number may include as many as 8 devices (luns or logical unit numbers).  Embedded SCSI devices only include one lun.

Higher target numbers receive better service and on a narrow bus, the target priorities run 7 -> 0.  On a wide bus, they run 7 -> 0, then 15 -> 8.  The host adapter is usually 7.  This can cause problems where busy disks and tape devices share a SCSI bus, since tape devices are usually assigned target 6.  If possible, isolate tape devices to their own SCSI bus.

If you are running older drives with Solaris 2.7 and above, you may run into a situation where there are many bus over-runs and errors.  This is usually caused by having one or more disks on the SCSI bus that improperly implement the tagged queuing option (or don't support it at all!).

For a quick fix, append this line to /etc/system and reboot:

set scsi_options & ~0x80

This turns off the Tagged Command Queuing.  Tagged command queuing has also been seen to cause problems between Suns and some RAID implementations.  However, the best fix is to replace the older drives with newer drives that will probably offer higher speed and greater capacity anyway.

Nevertheless, verify that your drives are incapable of supporting tagged queuing before making this change, since this will seriously degrade performance on disks that do properly support tagged command queuing.  Setting the SCSI options for the target drives that can't support the option is therefore the preferred solution.

In Solaris 2.4 and later versions, you can set those options per SCSI bus.  See your systems man entries or paper manual instructions for isp(7) and esp(7).

For some disks, decreasing the maximum number of queued commands by setting an option for the sd driver (SCSI Driver) using the host configuration variable sd_max_throttle, is all that is necessary:

set sd:sd_max_throttle=10

In later Solaris releases, you can specify scsi_options per target or per SCSI bus.  See esp(7d), isp(7d), from which this example /kernel/drv/esp.conf file is derived:

name="esp" parent="/iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000"  reg=0xf,0x800000,0x40
target1-scsi-options=0x58
scsi-options=0x178;

It has also been reported that some hardware RAIDs support a number of different LUNs (logical disks), but these LUNs share a common set of I/O buffers between them.  This can cause SCSI QFULL conditions on those devices that do not have commands queued.  Since the usual algorithm is to retry the command when a previous command is completed, Solaris doesn't handle this situation very well.

The workaround is to decrease sd_max_throttle such that there's always at least 1 slot available for each LUN,  e.g., if you have 3 LUNs and your RAID supports a maximum of 64 outstanding commands,   sd_max_throttle must be at most 31.  Any two LUNs can get 31 requests and you still have two slots left over for number 3.

For some hardware RAIDs, it has been found that decreasing sd_max_throttle improved performance due to better load balancing among LUNs.  It might be worth a shot if your hardware based RAID system seems sluggish. 

There are several additional host configuration variables that can be specified in addition to sd_max_throttle and in relation to the SCSI interface on SUN Solaris.  These are usually specified in the /etc/system file as was shown above.

The following variables in the /etc/systems file should be set to maximize system performance. When any of these variables are changed, the system must be rebooted for changes to take effect.

sd_max_throttle -- The sd_max_throttle variable sets the maximum number of commands that the SCSI sd driver will attempt to queue to a single HBA driver.  The default value is 256.  This variable must be set to a value less than or equal to the maximum queue depth of each LUN connected to each instance of the sd driver.  If this is not done, then commands may be rejected because of a full queue condition and the sd driver instance that receives the queue full message will throttle down sd_max_throttle to 1.  This obviously will result in degraded performance.  The variable is set in the /etc/system file as follows:

set sd:sd_max_throttle=20