ORA-15080: synchronous I/O operation to a disk failed tips

Question:  I am getting this error, a ORA-15080, and my RAC node crashed after killing a session with an active transaction:

ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 9084 logical extent 0 of file 266 in group 2 on disk 7 allocation unit 385054

How do I fix the ORA-15080 error?  I am also seeing the ORA-27063 error:

ORA-27063: number of bytes read/written is incorrect

Cause: the number of bytes read/written as returned by aiowait does not match the original number, additional information indicates both these numbers

Action: check errno

Answer:  The ORA-15080 error is an operating system error, normally caused when a part of the disk I/O subsystem is not operational.  This could be an individual disk failure, and you will want to run the iostat utility against the disk to ensure that you are not experiencing I/O corruption.  Your system administrator should immediately check your disks strace output and ensure that you can do dd-read and dd-write operations against file number 266.

Only Oracle technical support will be able to give you the exact reasons for the node crash (likely a bug), but it is probably related to a sporadic disk hardware failure, that's my guess.  Check for any trace or dump files that may have been created when the node crashed, and immediately check the surviving nodes to ensure that they are working properly.

The oerr utility shows this for the ORA-15080 error:

ORA-15080: synchronous I/O operation to a disk failed

Cause A synchronous I/O operation invoked on a disk has failed.

Action Make sure that all the disks are operational.

This node crash and the ORA-15080 error may be related to MOSC note 393958.1 titled "Disk has been offline In ASM Diskgroup and has 2 entries in v$asm_disk".  This note mentions the ORA-15080 error is a similar context to yours, and it provides a solution for the ORA-15080 error.

Regarding this message "WARNING: failed to write mirror side 1 of virtual extent 9084 logical extent 0 of file 266", you can use the dba_data_files view to look-up the disk that contains data file number 266. 

This is just a warning message, on a write to a mirrored disk.

If possible, I would take the time to check each of the other nodes and ensure that they are operational by checking the alert log for errors, and if the entire cluster is down, I would bounce the entire RAC cluster and give the disks a chance to synchronize themselves.

If this is production impact problem and you are experiencing a system-down error, open a severity-one service request (emergency) on MOSC, and the Oracle technical support team can immediately assist you.