Exadata X5 Hybrid Rack | OVM and Physical Database Nodes in One Rack.

For my own curiosity, I was creating OEDA for latest Exadata X5 half rack and found that we can have Physical compute nodes and OVM Compute nodes in same cluster & in Same Rack.

Change Exadata Write-Back Flash cache in Rolling Mode.

---Make sure you have both of this files into one directory.

 [root@dummyCNadm01 ~]# pwd  
 /root  
 rwxr--r-- 1 root root 141286 May 12 13:47 setWBFC.sh  
 -rwxr-xr-x 1 root root  683 May 12 13:47 wbfc_FLUSH.sh  

----From compute Node. Run Pre-check to see if cell servers are ready to flip over WBFC .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling -p  
 ./setWBFC.sh: Using log directory '/tmp'  
 ./setWBFC.sh: Log File '/tmp/setWBFC_85166_2015-05-12-13:50:03.log' created successfully  
 2015-05-12 13:50:03  
 Starting ./setWBFC.sh on dummyCNadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -p (Perform pre-req checks only)  
  -t 21600  
  -x 0  
 2015-05-12 13:50:03  
 Performing pre-req checks.....  
 2015-05-12 13:50:03  
 Creating baseline inventory for griddisks  
 2015-05-12 13:50:06  
 Creating baseline inventory for flashdisks  
 2015-05-12 13:50:08  
 Creating baseline inventory for flashsize  
 2015-05-12 13:50:10  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 13:50:10  
 Checking cell nodes are valid storage servers...  
 2015-05-12 13:50:10  
 All cells are valid Exadata storage cells.  
 2015-05-12 13:50:10  
 Checking Exadata Storage Software Versions...  
 2015-05-12 13:50:15  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 13:50:15  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 13:50:17  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 13:50:17  
 Checking for active ASM operations....  
 2015-05-12 13:50:18  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 13:50:18  
 Checking griddisk status across all cells....  
 2015-05-12 13:50:20  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 13:50:20  
 Checking flash cache status.....  
 2015-05-12 13:50:21  
 Flashcache status normal:            [PASSED]  
 2015-05-12 13:50:21  
 Checking that all FlashDisks are present...  
 2015-05-12 13:50:22  
 FlashDisk validation:              [PASSED]  
 2015-05-12 13:50:22  
 Checking current flash cache mode.....  
 2015-05-12 13:50:23  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 13:50:23  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 13:50:24  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  

---- Turn write back ON in rolling fashion .

 [root@dummyCNadm01 ~]# ./setWBFC.sh -g cell_group -l /tmp -m WriteBack -o rolling  
 2015-05-12 14:59:41  
 Starting ./setWBFC.sh on dummy01dbadm01  
 Version: 1.0.0.1.6.20140716  
 Command line options used:  
  -g cell_group  
  -o rolling  
  -m WriteBack  
  -t 21600  
  -x 0  
 2015-05-12 14:59:41  
 Performing pre-req checks.....  
 2015-05-12 14:59:41  
 Creating baseline inventory for griddisks  
 2015-05-12 14:59:42  
 Creating baseline inventory for flashdisks  
 2015-05-12 14:59:43  
 Creating baseline inventory for flashsize  
 2015-05-12 14:59:44  
 dcli present and in PATH.            [PASSED]  
 2015-05-12 14:59:44  
 Checking cell nodes are valid storage servers...  
 2015-05-12 14:59:44  
 All cells are valid Exadata storage cells.  
 2015-05-12 14:59:44  
 Checking Exadata Storage Software Versions...  
 2015-05-12 14:59:48  
 Software versions of the following cells:  
 dummy01celadm01: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm02: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm03: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm04: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm05: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm06: 12.1.2.1.1.150316.2           [PASSED]  
 dummy01celadm07: 12.1.2.1.1.150316.2           [PASSED]  
 2015-05-12 14:59:48  
 Checking Grid Infrastructure Software Version...  
 2015-05-12 14:59:51  
 Grid Infrastructure version: 12.1.0.2.00     [PASSED]  
 2015-05-12 14:59:51  
 Checking for active ASM operations....  
 2015-05-12 14:59:51  
 Check for no active ASM operations:       [PASSED]  
 2015-05-12 14:59:51  
 Checking griddisk status across all cells....  
 2015-05-12 14:59:54  
 All griddisks across all cells have asmdeactivationoutcome = Yes  
 All griddisks across all cells are ONLINE  
 Griddisk checks:                 [PASSED]  
 2015-05-12 14:59:54  
 Checking flash cache status.....  
 2015-05-12 14:59:55  
 Flashcache status normal:            [PASSED]  
 2015-05-12 14:59:55  
 Checking that all FlashDisks are present...  
 2015-05-12 14:59:57  
 FlashDisk validation:              [PASSED]  
 2015-05-12 14:59:57  
 Checking current flash cache mode.....  
 2015-05-12 14:59:58  
 Flashcache not already in target mode:      [PASSED]  
 2015-05-12 14:59:58  
 All pre-req checks completed:          [PASSED]  
 2015-05-12 14:59:59  
 dummy01celadm01: flashcache size: 5.82122802734375T  
 dummy01celadm02: flashcache size: 5.82122802734375T  
 dummy01celadm03: flashcache size: 5.82122802734375T  
 dummy01celadm04: flashcache size: 5.82122802734375T  
 dummy01celadm05: flashcache size: 5.82122802734375T  
 dummy01celadm06: flashcache size: 5.82122802734375T  
 dummy01celadm07: flashcache size: 5.82122802734375T  
 There are 7 storage cells to process.  
 2015-05-12 14:59:59  
 Changing flash cache to WriteBack ROLLING....  
 2015-05-12 14:59:59  
 STEP 0: Checking gridisk status on cell: dummy01celadm01  
 2015-05-12 15:00:01  
 STEP 0 completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:04  
 STEP 1: Dropping flashcache on cell: dummy01celadm01  
 2015-05-12 15:00:36  
 STEP 1: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:00:39  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:00:39  
 STEP 4: Set the flashCachMode on cell: dummy01celadm01  
 2015-05-12 15:00:39  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:00:40  
 STEP 4: Completed successfully on cell: dummy01celadm01  
 2015-05-12 15:00:43  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:00:43  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:00:43  
 STEP 7: Creating flashcache on cell: dummy01celadm01  
 2015-05-12 15:01:32  
 STEP 7: Completed sucessfully on cell: dummy01celadm01  
 2015-05-12 15:01:35  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm01  
 2015-05-12 15:01:36  
 STEP 8: Completed sucessfully on cell: dummy01celadm01  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:01:39  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:01:39  
 STEP 0: Checking gridisk status on cell: dummy01celadm02  
 2015-05-12 15:01:41  
 STEP 0 completed successfully on cell: dummy01celadm02  
 2015-05-12 15:01:44  
 STEP 1: Dropping flashcache on cell: dummy01celadm02  
 2015-05-12 15:02:19  
 STEP 1: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:02:22  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:02:22  
 STEP 4: Set the flashCachMode on cell: dummy01celadm02  
 2015-05-12 15:02:22  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:02:23  
 STEP 4: Completed successfully on cell: dummy01celadm02  
 2015-05-12 15:02:26  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:02:26  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:02:26  
 STEP 7: Creating flashcache on cell: dummy01celadm02  
 2015-05-12 15:03:21  
 STEP 7: Completed sucessfully on cell: dummy01celadm02  
 2015-05-12 15:03:24  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm02  
 2015-05-12 15:03:25  
 STEP 8: Completed sucessfully on cell: dummy01celadm02  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:03:28  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:03:28  
 STEP 0: Checking gridisk status on cell: dummy01celadm03  
 2015-05-12 15:03:29  
 STEP 0 completed successfully on cell: dummy01celadm03  
 2015-05-12 15:03:32  
 STEP 1: Dropping flashcache on cell: dummy01celadm03  
 2015-05-12 15:04:07  
 STEP 1: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:04:10  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:04:10  
 STEP 4: Set the flashCachMode on cell: dummy01celadm03  
 2015-05-12 15:04:10  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:04:10  
 STEP 4: Completed successfully on cell: dummy01celadm03  
 2015-05-12 15:04:13  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:04:13  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:04:13  
 STEP 7: Creating flashcache on cell: dummy01celadm03  
 2015-05-12 15:05:09  
 STEP 7: Completed sucessfully on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm03  
 2015-05-12 15:05:12  
 STEP 8: Completed sucessfully on cell: dummy01celadm03  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:05:15  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:05:15  
 STEP 0: Checking gridisk status on cell: dummy01celadm04  
 2015-05-12 15:05:17  
 STEP 0 completed successfully on cell: dummy01celadm04  
 2015-05-12 15:05:20  
 STEP 1: Dropping flashcache on cell: dummy01celadm04  
 2015-05-12 15:05:53  
 STEP 1: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:05:56  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:05:56  
 STEP 4: Set the flashCachMode on cell: dummy01celadm04  
 2015-05-12 15:05:56  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:06:01  
 STEP 4: Completed successfully on cell: dummy01celadm04  
 2015-05-12 15:06:04  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:06:04  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:06:04  
 STEP 7: Creating flashcache on cell: dummy01celadm04  
 2015-05-12 15:06:47  
 STEP 7: Completed sucessfully on cell: dummy01celadm04  
 2015-05-12 15:06:50  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm04  
 2015-05-12 15:06:51  
 STEP 8: Completed sucessfully on cell: dummy01celadm04  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:06:54  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:06:54  
 STEP 0: Checking gridisk status on cell: dummy01celadm05  
 2015-05-12 15:06:55  
 STEP 0 completed successfully on cell: dummy01celadm05  
 2015-05-12 15:06:58  
 STEP 1: Dropping flashcache on cell: dummy01celadm05  
 2015-05-12 15:07:40  
 STEP 1: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:07:43  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:07:43  
 STEP 4: Set the flashCachMode on cell: dummy01celadm05  
 2015-05-12 15:07:43  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:07:43  
 STEP 4: Completed successfully on cell: dummy01celadm05  
 2015-05-12 15:07:46  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:07:46  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:07:46  
 STEP 7: Creating flashcache on cell: dummy01celadm05  
 2015-05-12 15:08:37  
 STEP 7: Completed sucessfully on cell: dummy01celadm05  
 2015-05-12 15:08:40  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm05  
 2015-05-12 15:08:41  
 STEP 8: Completed sucessfully on cell: dummy01celadm05  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:08:44  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:08:44  
 STEP 0: Checking gridisk status on cell: dummy01celadm06  
 2015-05-12 15:08:46  
 STEP 0 completed successfully on cell: dummy01celadm06  
 2015-05-12 15:08:49  
 STEP 1: Dropping flashcache on cell: dummy01celadm06  
 2015-05-12 15:09:23  
 STEP 1: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:09:26  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:09:26  
 STEP 4: Set the flashCachMode on cell: dummy01celadm06  
 2015-05-12 15:09:26  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:09:30  
 STEP 4: Completed successfully on cell: dummy01celadm06  
 2015-05-12 15:09:33  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:09:33  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:09:33  
 STEP 7: Creating flashcache on cell: dummy01celadm06  
 2015-05-12 15:10:19  
 STEP 7: Completed sucessfully on cell: dummy01celadm06  
 2015-05-12 15:10:22  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm06  
 2015-05-12 15:10:23  
 STEP 8: Completed sucessfully on cell: dummy01celadm06  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:10:26  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:10:26  
 STEP 0: Checking gridisk status on cell: dummy01celadm07  
 2015-05-12 15:10:28  
 STEP 0 completed successfully on cell: dummy01celadm07  
 2015-05-12 15:10:31  
 STEP 1: Dropping flashcache on cell: dummy01celadm07  
 2015-05-12 15:11:04  
 STEP 1: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 Skipping STEP 2: Inactivating grid disks not required.  
 2015-05-12 15:11:07  
 Skipping STEP 3: Shutdown of cellsrv not required.  
 2015-05-12 15:11:07  
 STEP 4: Set the flashCachMode on cell: dummy01celadm07  
 2015-05-12 15:11:07  
 STEP 4: Setting flashCacheMode to WriteBack  
 2015-05-12 15:11:13  
 STEP 4: Completed successfully on cell: dummy01celadm07  
 2015-05-12 15:11:16  
 Skipping STEP 5: Restart of cellsrv not required.  
 2015-05-12 15:11:16  
 Skipping STEP 6: Acativating grid disks not required.  
 2015-05-12 15:11:16  
 STEP 7: Creating flashcache on cell: dummy01celadm07  
 2015-05-12 15:12:02  
 STEP 7: Completed sucessfully on cell: dummy01celadm07  
 2015-05-12 15:12:05  
 STEP 8: Verifying flashCacheMode on cell: dummy01celadm07  
 2015-05-12 15:12:06  
 STEP 8: Completed sucessfully on cell: dummy01celadm07  
 Flash Cache mode is now WriteBack  
 2015-05-12 15:12:09  
 Skipping STEP 9: Waiting for grid disks to sync not required.  
 2015-05-12 15:12:09  
 Validating inventory for griddisks  
 2015-05-12 15:12:11  
 Validation of griddisk:             [PASSED]  
 2015-05-12 15:12:11  
 Validating inventory for flashdisks  
 2015-05-12 15:12:13  
 Validation of flashdisk:             [PASSED]  
 2015-05-12 15:12:13  
 Validating inventory for flashsize  
 2015-05-12 15:12:14  
 Validation of flashsize:             [PASSED]  
 2015-05-12 15:12:14  
 Setting flash cache to WriteBack completed successfully.  

dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"

cp preserving permissions for | Operation not supported | NFS

This issue was in Exalogic ZFS , when we had exported shares via NFS3 over to few Linux Guest VMs.

 [root@dummyserver uxyz]# cp -p /uxyz/log.txt /uxyz/test.log  
 cp: preserving permissions for `/uxyz/test.log': Operation not supported  

----As you can see + sign for ACL permissions

 [root@dummyserver ~]# cd /uxyz  
 [root@dummyserver uxyz]# ll  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x+ 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r--+ 1 appUSER dba 0 May 6 15:59 test.txt  

/etc/fstab file entry looked like this .

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp  

--to overcome this issue. append "noacl" at end of line.

 000.00.0.9:/export/share-2-uxyz     /uxyz      nfs rw,nointr,bg,hard,timeo=600,wsize=65536,rsize=65536,nfsvers=3,tcp,noacl  

----dont try to mount -o noacl,remount /u11
----unmount properly

 [root@dummyserver ~]# umount /uxyz  
 [root@dummyserver ~]# mount /uxyz  

------Magic. .

 [appUSER@dummyserver ~]$ cp -p /uxyz/log.txt /uxyz/test.log  
 [appUSER@dummyserver ~]$ ll  

-----There is no + sign for ACL.

 [root@dummyserver uxyz]# ll  
 total 10  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 bkp.txt  
 drwxr-sr-x 3 appUSER dba 3 Apr 25 02:22 home  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 log.txt  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.log  
 -rw-r--r-- 1 appUSER dba 0 May 6 15:59 test.txt  

---Reference.

Solaris 10 NFS client fails in ACL or permissions when copying files to Linux NFS server (Doc ID 1917349.1)
cp ACL Error Creating OSB Domain, 'Operation not supported' (Doc ID 1959764.1)

Exadata Local Read-only file system | End_request: I/O error

---On database Nodes we wont be able to write on / or any local file system.

 [root@dummyhostname01 ~]# df  
 Filesystem       1K-blocks    Used  Available Use% Mounted on  
 /dev/mapper/VGExaDb-LVDbSys1  
             30963708  22372264   7018580 77% /  
 tmpfs         264152064     4  264152060  1% /dev/shm  
 /dev/sda1         516040   40016   449812  9% /boot  
 /dev/mapper/VGExaDb-LVDbOra1  
             103212320  25806260  72163180 27% /u01  
 [root@dummyhostname01 oracle.cellos]# cd conf  
 [root@dummyhostname01 conf]# ls  
 ls: reading directory .: Input/output error  
 [root@dummyhostname01 log]# cd /u01  
 [root@dummyhostname01 u01]# ls  
 app lost+found  
 [root@dummyhostname01 u01]# ll  
 total 20  
 drwxr-xr-x 5 root oinstall 4096 Mar 23 19:11 app  
 drwx------ 2 root root   16384 Mar 10 19:26 lost+found  
 [root@dummyhostname01 u01]# touch test  
 touch: cannot touch `test': Read-only file system  

---On looking at tail -f /var/log/messeges

 May 5 02:18:24 dummyhostname01 adclient[21135]: INFO <fd:25 sudo(100222)> client.sudo Set credentials for user 'root': mapping misconfiguration. Passing user to next service module.  
 May 5 02:18:26 dummyhostname01 kernel: megaraid_sas: Iop2SysDoorbellIntfor scsi0  
 May 5 02:18:27 dummyhostname01 kernel: megasas: Found FW in FAULT state, will reset adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: resetting fusion adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: megaraid_sas: Reset not supported, killing adapter scsi0.  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Unhandled error code  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK  
 May 5 02:18:27 dummyhostname01 kernel: sd 0:2:0:0: [sda] CDB: Write(10): 2a 00 16 b5 5d 40 00 00 08 00  
 May 5 02:18:27 dummyhostname01 kernel: blk_update_request: 4 callbacks suppressed  
 May 5 02:18:27 dummyhostname01 kernel: end_request: I/O error, dev sda, sector 380984640  
 May 5 02:18:27 dummyhostname01 kernel: Buffer I/O error on device dm-3, logical block 17606304  

---firmware / hardware diagnostic on Database Node.
---As you can not write on Local Filesystem you could write on NFS mounts if there is present on DBNODE.

 dmesg > /NFS_MOUNT/sundiag/dmesg.txt   
 ipmitool sunoem cli force 'show /SP/console/history' > /NFS_MOUNT/sundiag/console.out   
 /opt/MegaRAID/MegaCli/MegaCli64 -FwTermLog Dsply -a0 >/NFS_MOUNT/sundiag/fwterm.txt   
 [root@dummyhostname01 sundiag]# cat /NFS_MOUNT/sundiag/fwterm.txt  
 User specified controller is not present.  
 Failed to get CpController object.  
 Exit Code: 0x01  
 /opt/MegaRAID/MegaCli/MegaCli64 -AdpEventLog -GetEvents -f /NFS_MOUNT/sundiag/events.txt -a0   
 lspci -vvv > /NFS_MOUNT/sundiag/lspci.out  

Solution.

--Finally reboot / restart force fully from ILOM.

 -> stop -f /SYS  
 Are you sure you want to immediately stop /SYS (y/n)? y  
 Stopping /SYS immediately  

Find DML SQL statement literal values caused Replicate to Abend.

Generally in Goldengate replicat apply SQL using bind variable so when error occurs report file or error log will have SQL statement with bind
variable. This is hard to catch which literal values caused a problem at Target side. Below is example to see Which literal values caused a
problem.

Error log / Reporfile (Normal Mode)

2013-05-15 11:08:31  WARNING OGG-01004  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Aborted grouped transaction on 'DUMMY_ADMIN_6.DUMMYEVENT', Database error 1403 (OCI Error ORA-01403: no data found, SQL ).

2013-05-15 11:08:31  WARNING OGG-01154  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
SQL error 1403 mapping DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT OCI Error ORA-01403: no data found, SQL .

2013-05-15 11:08:31  ERROR   OGG-01296  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Error mapping from DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT.

Edit below parameters in replicat and start again.

NOBINARYCHARS
NODYNSQL
SHOWSYNTAX

Error log / Reporfile (Actual value)

2013-05-15 11:17:35  WARNING OGG-01004  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Aborted grouped transaction on 'DUMMY_ADMIN_6.DUMMYEVENT', Database error 1403 (OCI Error ORA-01403: no data found, SQL ).
2013-05-15 11:17:35  WARNING OGG-01154  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  

SQL error 1403 mapping DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT OCI Error ORA-01403: no data found, SQL .
2013-05-15 11:17:35  ERROR   OGG-01296  Oracle GoldenGate Delivery for Oracle, rbmdr.prm:  
Error mapping from DUMMY_ADMIN.DUMMYEVENT to DUMMY_ADMIN_6.DUMMYEVENT.

RS-700 Celloflsrv hang detected on Exadata 12.1.1.1

RS-700 [Celloflsrv hang detected. It will be terminated] [SYS_121111_140712] [] [] [] [] [] [] [] [] [] []
Server Model Oracle Corporation SUN SERVER X4-2L High Capacity
Release Version 12.1.1.1.1
Release Label OSS_12.1.1.1.1_LINUX.X64_140712

This is Bug 19132065 - Oracle Linux semtimedop() wakeups by timeout are lagging causing offload operations to fail (which may degrade performance) and errors similar to one or more of the following:
? ORA-700 [Offload issue job timed out]
? ORA-700 [Offload group not open]
? RS-700 [Celloflsrv hang detected. It will be terminated]

This bug affects related to 12.1.1.1. storage Version.
It is due to DB Node RCU delayed and cause Offload job to fail on Cellservices .
it affects database performance not availability.
Error ocure mostly when cellserv tried to do Read optimization.
reducing Delay in RCU is work around accross whole stack.

Step 1: Set rcu_delay for runtime

# echo 1 > /proc/sys/kernel/rcu_delay
Verify the setting
# cat /proc/sys/kernel/rcu_delay
1

Step 2: Set rcu_delay in /etc/sysctl.conf for proper setting upon reboot

Add "kernel.rcu_delay=1" to /etc/sysctl.conf

Step 3: Restart cellsrv on storage servers

CellCLI> alter cell restart services cellsrv;

This workaround is automatically applied in the following cases:
When a new system is deployed with Exadata 11.2.3.3.1 or 12.1.1.1.1 using OEDA Sep 2014 or later.
When storage servers are upgraded to 11.2.3.3.1 or 12.1.1.1.1 and the patchmgr plugins patch is properly staged before running patchmgr, as documented.
When database servers are upgraded to 11.2.3.3.1 or 12.1.1.1.1 using dbnodeupdate.sh v3.58 or later.

References https://www.kernel.org/doc/Documentation/RCU/whatisRCU.txt

Exadata Schedule Exachk using OEM and command Line | Exadata healthcheck OEM

There is New Plugin to Install and schedule Exachk on Exadata/ Exalogic system Through 12c Grid control.

Plugin has to first deploy on OMS server and then Exadata / Exalogic Agents.

12c Home page setup (right upper corner) -> extensibility -> Plugin -> Engineered system -> Oracle Exadata healthcheck.

Once it deploy on 12c Grid OMS and Exadata agent.
You can create Monitoring Targets for Exadata Healthcheck Metrics.

https://docs.oracle.com/cd/E24628_01/install.121/e27420/toc.htm#PICHK105 

To schedule Using command Line.

Download Latest Exachk from
Oracle Exadata Database Machine exachk or HealthCheck (Doc ID 1070954.1).
Exachk Health-Check Tool for Exalogic (Doc ID 1449226.1)


Note : Exachk must run from Enterprise Vserver in Virtual Exalogic Environment.

---check Version

./exachk -v
EXACHK  VERSION: 12.1.0.2.1

---Set User equivelency between DB node to storage cell and IB switches 

./exachk -initpresetup   

---setup auto restart and Exachk Deamon. 

./exachk -initsetup
Setting up exachk auto restart functionality using inittab
Starting exachk daemon. . .  .

---Check auto restart and deamon status . 

./exachk -initcheck
Auto restart functionality is configured.
exachk daemon is running. PID : 00000

---schedule exachk. 

AUTORUN_SCHEDULE * * * *       :- Automatic run at specific time in daemon mode.
                 - - - -
                 ? ? ? ?
                 ? ? ? +----- day of week (0 - 6) (0 to 6 are Sunday to Saturday)
                 ? ? +---------- month (1 - 12)
                 ? +--------------- day of month (1 - 31)
                 +-------------------- hour (0 - 23)

./exachk -set "AUTORUN_SCHEDULE=23 * * 0;AUTORUN_FLAGS= -a -o v;NOTIFICATION_EMAIL=firstname.lastname@company.com;PASSWORD_CHECK_INTERVAL=1;"

---List all parameter for exachk. 

./exachk -get all