Exadata X5 Hybrid Rack | OVM and Physical Database Nodes in One Rack.

For my own curiosity, I was creating OEDA for latest Exadata X5 half rack and found that we can have Physical compute nodes and OVM Compute nodes in same cluster & in Same Rack.

Change Exadata Write-Back Flash cache in Rolling Mode.

---Make sure you have both of this files into one directory.

 [root@dummyCNadm01 ~]# pwd  
 /root  
 rwxr--r-- 1 root root 141286 May 12 13:47 setWBFC.sh  
 -rwxr-xr-x 1 root root  683 May 12 13:47 wbfc_FLUSH.sh  

----From compute Node. Run Pre-check to see if cell servers are ready to flip over WBFC .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling -p  
 ./setWBFC.sh: Using log directory '/tmp'  
 ./setWBFC.sh: Log File '/tmp/setWBFC_85166_2015-05-12-13:50:03.log' created successfully  
 2015-05-12 13:50:03  
 Starting ./setWBFC.sh on dummyCNadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -p (Perform pre-req checks only)  
  -t 21600  
  -x 0  
 2015-05-12 13:50:03  
 Performing pre-req checks.....  
 2015-05-12 13:50:03  
 Creating baseline inventory for griddisks  
 2015-05-12 13:50:06  
 Creating baseline inventory for flashdisks  
 2015-05-12 13:50:08  
 Creating baseline inventory for flashsize  
 2015-05-12 13:50:10  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 13:50:10  
 Checking cell nodes are valid storage servers...  
 2015-05-12 13:50:10  
 All cells are valid Exadata storage cells.  
 2015-05-12 13:50:10  
 Checking Exadata Storage Software Versions...  
 2015-05-12 13:50:15  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 13:50:15  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 13:50:17  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 13:50:17  
 Checking for active ASM operations....  
 2015-05-12 13:50:18  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 13:50:18  
 Checking griddisk status across all cells....  
 2015-05-12 13:50:20  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 13:50:20  
 Checking flash cache status.....  
 2015-05-12 13:50:21  
 Flashcache status normal:            [PASSED]  
 2015-05-12 13:50:21  
 Checking that all FlashDisks are present...  
 2015-05-12 13:50:22  
 FlashDisk validation:              [PASSED]  
 2015-05-12 13:50:22  
 Checking current flash cache mode.....  
 2015-05-12 13:50:23  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 13:50:23  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 13:50:24  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  

---- Turn write back ON in rolling fashion .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling  
 2015-05-12 14:59:41  
 Starting ./setWBFC.sh on dummy01dbadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -t 21600  
  -x 0  
 2015-05-12 14:59:41  
 Performing pre-req checks.....  
 2015-05-12 14:59:41  
 Creating baseline inventory for griddisks  
 2015-05-12 14:59:42  
 Creating baseline inventory for flashdisks  
 2015-05-12 14:59:43  
 Creating baseline inventory for flashsize  
 2015-05-12 14:59:44  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 14:59:44  
 Checking cell nodes are valid storage servers...  
 2015-05-12 14:59:44  
 All cells are valid Exadata storage cells.  
 2015-05-12 14:59:44  
 Checking Exadata Storage Software Versions...  
 2015-05-12 14:59:48  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 14:59:48  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 14:59:51  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 14:59:51  
 Checking for active ASM operations....  
 2015-05-12 14:59:51  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 14:59:51  
 Checking griddisk status across all cells....  
 2015-05-12 14:59:54  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 14:59:54  
 Checking flash cache status.....  
 2015-05-12 14:59:55  
 Flashcache status normal:            [PASSED]  
 2015-05-12 14:59:55  
 Checking that all FlashDisks are present...  
 2015-05-12 14:59:57  
 FlashDisk validation:              [PASSED]  
 2015-05-12 14:59:57  
 Checking current flash cache mode.....  
 2015-05-12 14:59:58  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 14:59:58  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 14:59:59  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  
 2015-05-12 14:59:59  
 Changing flash cache to WriteBack ROLLING....  
 2015-05-12 14:59:59  
 STEP 0: Checking gridisk status on cell: dummy01celadm01  
 2015-05-12 15:00:01  
 STEP 0 completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:04  
 STEP 1: Dropping flashcache on cell: dummy01celadm01  
 2015-05-12 15:00:36  
 STEP 1: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:00:39  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:00:39  
 STEP 4: Set the flashCachMode on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:00:40  
 STEP 4: Completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:43  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:00:43  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:00:43  
 STEP 7: Creating flashcache on cell: dummy01celadm01  
 2015-05-12 15:01:32  
 STEP 7: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:01:35  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm01  
 2015-05-12 15:01:36  
 STEP 8: Completed sucessfully on cell: dummy01celadm01  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:01:39  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:01:39  
 STEP 0: Checking gridisk status on cell: dummy01celadm02  
 2015-05-12 15:01:41  
 STEP 0 completed successfully on cell: dummy01celadm02  
 2015-05-12 15:01:44  
 STEP 1: Dropping flashcache on cell: dummy01celadm02  
 2015-05-12 15:02:19  
 STEP 1: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:02:22  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:02:22  
 STEP 4: Set the flashCachMode on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:02:23  
 STEP 4: Completed successfully on cell: dummy01celadm02  
 2015-05-12 15:02:26  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:02:26  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:02:26  
 STEP 7: Creating flashcache on cell: dummy01celadm02  
 2015-05-12 15:03:21  
 STEP 7: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:03:24  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm02  
 2015-05-12 15:03:25  
 STEP 8: Completed sucessfully on cell: dummy01celadm02  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:03:28  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:03:28  
 STEP 0: Checking gridisk status on cell: dummy01celadm03  
 2015-05-12 15:03:29  
 STEP 0 completed successfully on cell: dummy01celadm03  
 2015-05-12 15:03:32  
 STEP 1: Dropping flashcache on cell: dummy01celadm03  
 2015-05-12 15:04:07  
 STEP 1: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:04:10  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:04:10  
 STEP 4: Set the flashCachMode on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:04:10  
 STEP 4: Completed successfully on cell: dummy01celadm03  
 2015-05-12 15:04:13  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:04:13  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:04:13  
 STEP 7: Creating flashcache on cell: dummy01celadm03  
 2015-05-12 15:05:09  
 STEP 7: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Completed sucessfully on cell: dummy01celadm03  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:05:15  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:05:15  
 STEP 0: Checking gridisk status on cell: dummy01celadm04  
 2015-05-12 15:05:17  
 STEP 0 completed successfully on cell: dummy01celadm04  
 2015-05-12 15:05:20  
 STEP 1: Dropping flashcache on cell: dummy01celadm04  
 2015-05-12 15:05:53  
 STEP 1: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:05:56  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:05:56  
 STEP 4: Set the flashCachMode on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:06:01  
 STEP 4: Completed successfully on cell: dummy01celadm04  
 2015-05-12 15:06:04  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:06:04  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:06:04  
 STEP 7: Creating flashcache on cell: dummy01celadm04  
 2015-05-12 15:06:47  
 STEP 7: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:06:50  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm04  
 2015-05-12 15:06:51  
 STEP 8: Completed sucessfully on cell: dummy01celadm04  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:06:54  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:06:54  
 STEP 0: Checking gridisk status on cell: dummy01celadm05  
 2015-05-12 15:06:55  
 STEP 0 completed successfully on cell: dummy01celadm05  
 2015-05-12 15:06:58  
 STEP 1: Dropping flashcache on cell: dummy01celadm05  
 2015-05-12 15:07:40  
 STEP 1: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:07:43  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:07:43  
 STEP 4: Set the flashCachMode on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:07:43  
 STEP 4: Completed successfully on cell: dummy01celadm05  
 2015-05-12 15:07:46  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:07:46  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:07:46  
 STEP 7: Creating flashcache on cell: dummy01celadm05  
 2015-05-12 15:08:37  
 STEP 7: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:08:40  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm05  
 2015-05-12 15:08:41  
 STEP 8: Completed sucessfully on cell: dummy01celadm05  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:08:44  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:08:44  
 STEP 0: Checking gridisk status on cell: dummy01celadm06  
 2015-05-12 15:08:46  
 STEP 0 completed successfully on cell: dummy01celadm06  
 2015-05-12 15:08:49  
 STEP 1: Dropping flashcache on cell: dummy01celadm06  
 2015-05-12 15:09:23  
 STEP 1: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:09:26  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:09:26  
 STEP 4: Set the flashCachMode on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:09:30  
 STEP 4: Completed successfully on cell: dummy01celadm06  
 2015-05-12 15:09:33  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:09:33  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:09:33  
 STEP 7: Creating flashcache on cell: dummy01celadm06  
 2015-05-12 15:10:19  
 STEP 7: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:10:22  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm06  
 2015-05-12 15:10:23  
 STEP 8: Completed sucessfully on cell: dummy01celadm06  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:10:26  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:10:26  
 STEP 0: Checking gridisk status on cell: dummy01celadm07  
 2015-05-12 15:10:28  
 STEP 0 completed successfully on cell: dummy01celadm07  
 2015-05-12 15:10:31  
 STEP 1: Dropping flashcache on cell: dummy01celadm07  
 2015-05-12 15:11:04  
 STEP 1: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:11:07  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:11:07  
 STEP 4: Set the flashCachMode on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:11:13  
 STEP 4: Completed successfully on cell: dummy01celadm07  
 2015-05-12 15:11:16  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:11:16  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:11:16  
 STEP 7: Creating flashcache on cell: dummy01celadm07  
 2015-05-12 15:12:02  
 STEP 7: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:12:05  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm07  
 2015-05-12 15:12:06  
 STEP 8: Completed sucessfully on cell: dummy01celadm07  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:12:09  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:12:09  
 Validating inventory for griddisks  
 2015-05-12 15:12:11  
 Validation of griddisk:             [PASSED]  
 2015-05-12 15:12:11  
 Validating inventory for flashdisks  
 2015-05-12 15:12:13  
 Validation of flashdisk:             [PASSED]  
 2015-05-12 15:12:13  
 Validating inventory for flashsize  
 2015-05-12 15:12:14  
 Validation of flashsize:             [PASSED]  
 2015-05-12 15:12:14  
 Setting flash cache to WriteBack completed successfully.  

dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"

cp preserving permissions for | Operation not supported | NFS

This issue was in Exalogic ZFS , when we had exported shares via NFS3 over to few Linux Guest VMs.

 [root@dummyserver uxyz]# cp -p /uxyz/log.txt /uxyz/test.log  
 cp: preserving permissions for `/uxyz/test.log': Operation not supported  

----As you can see + sign for ACL permissions

 [root@dummyserver ~]# cd /uxyz  
 [root@dummyserver uxyz]# ll  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x+ 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 test.txt  

/etc/fstab file entry looked like this .

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp  

--to overcome this issue. append "noacl" at end of line.

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp,noacl  

----dont try to mount -o noacl,remount /u11
----unmount properly

 [root@dummyserver ~]# umount /uxyz  
 [root@dummyserver ~]# mount /uxyz  

------Magic. .

 [appUSER@dummyserver ~]$ cp -p /uxyz/log.txt /uxyz/test.log  
 [appUSER@dummyserver ~]$ ll  

-----There is no + sign for ACL.

 [root@dummyserver uxyz]# ll  
 total 10  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.log  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.txt  

---Reference.

Solaris 10 NFS client fails in ACL or permissions when copying files to Linux NFS server (Doc ID 1917349.1)
cp ACL Error Creating OSB Domain, 'Operation not supported' (Doc ID 1959764.1)

Exadata Local Read-only file system | End_request: I/O error

---On database Nodes we wont be able to write on / or any local file system.

 [root@dummyhostname01 ~]# df  
 Filesystem       1K-blocks    Used  Available Use% Mounted on  
 /dev/mapper/VGExaDb-LVDbSys1  
             30963708  22372264   7018580 77% /  
 tmpfs         264152064     4  264152060  1% /dev/shm  
 /dev/sda1         516040   40016   449812  9% /boot  
 /dev/mapper/VGExaDb-LVDbOra1  
             103212320  25806260  72163180 27% /u01  
 [root@dummyhostname01 oracle.cellos]# cd conf  
 [root@dummyhostname01 conf]# ls  
 ls: reading directory .: Input/output error  
 [root@dummyhostname01 log]# cd /u01  
 [root@dummyhostname01 u01]# ls  
 app lost+found  
 [root@dummyhostname01 u01]# ll  
 total 20  
 drwxr-xr-x 5 root oinstall 4096 Mar 23 19:11 app  
 drwx------ 2 root root   16384 Mar 10 19:26 lost+found  
 [root@dummyhostname01 u01]# touch test  
 touch: cannot touch `test': Read-only file system  

---On looking at tail -f /var/log/messeges

 May 5 02:18:24 dummyhostname01 adclient[21135]: INFO <fd:25 sudo(100222)> client.sudo Set credentials for user 'root': mapping misconfiguration. Passing user to next service module.  
 May 5 02:18:26 dummyhostname01 kernel: megaraid_sas: Iop2SysDoorbellIntfor scsi0  
 May 5 02:18:27 dummyhostname01 kernel: megasas: Found FW in FAULT state, will reset adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: resetting fusion adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: Reset not supported, killing adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Unhandled error code  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] CDB: Write(10): 2a 00 16 b5 5d 40 00 00 08 00  
 May 5 02:18:27 dummyhostname01 kernel: blk_update_request: 4 callbacks suppressed  
 May 5 02:18:27 dummyhostname01 kernel: end_request: I/O error, dev sda, sector 380984640  
 May 5 02:18:27 dummyhostname01 kernel: Buffer I/O error on device dm-3, logical block 17606304  

---firmware / hardware diagnostic on Database Node.
---As you can not write on Local Filesystem you could write on NFS mounts if there is present on DBNODE.

 dmesg > /NFS_MOUNT/sundiag/dmesg.txt   
 ipmitool sunoem cli force 'show /SP/console/history' > /NFS_MOUNT/sundiag/console.out   
 /opt/MegaRAID/MegaCli/MegaCli64 -FwTermLog Dsply -a0 >/NFS_MOUNT/sundiag/fwterm.txt   
 [root@dummyhostname01 sundiag]# cat /NFS_MOUNT/sundiag/fwterm.txt  
 User specified controller is not present.  
 Failed to get CpController object.  
 Exit Code: 0x01  
 /opt/MegaRAID/MegaCli/MegaCli64 -AdpEventLog -GetEvents -f /NFS_MOUNT/sundiag/events.txt -a0   
 lspci -vvv > /NFS_MOUNT/sundiag/lspci.out  

Solution.

--Finally reboot / restart force fully from ILOM.

 -> stop -f /SYS  
 Are you sure you want to immediately stop /SYS (y/n)? y  
 Stopping /SYS immediately