RMAN 04006 and ORA 17627 | ORA 01017 invalid username/password while RMAN duplicate.

While duplicate database By RMAN using active database command.
I had trouble of connecting from RMAN to non-mounted standby instance.
It was connecting to sqlplus perfectly.

I did copy password file from primary. made connection from tns to non-mounted instance using SQLPLUS.
it worked fine but when i tries from RMAN it gaves error.


ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-01017: invalid username/password; logon denied
ORA-17629: Cannot connect to the remote database server

Then I recreated password file with below command IGNORECASE=Y is the one which i was missing.
You also need to put sec_case_sensitive_logon=false in your non-mounted standby instance init file.

export ORACLE_SID=DUMMY
orapwd file=$ORACLE_HOME/dbs/orapw$ORACLE_SID password=sys password entries=10 force=y/n ignorecase=y 

This works like a charm. ...!!!

Goldengate Discard Monitoring and Alert.

This script is to monitor Discard , It should run Each Hour at 59 Min.

It will send email if Any discard found in Any Processes. Please customize it according to your Need.

It is still under Development and raw but it does the Job.

#!/usr/bin/ksh
##############################################################
# To get Accurate Result schedule it at each Hour like 59 * * * * from Cron
##############################################################
# Export variables
##############################################################
export ORACLE_HOME=/usr/local/opt/oracle/11r2
export GG_HOME=/usr/local/opt/oracle/ggs/112106
export PATH=$ORACLE_HOME/bin:$GG_HOME:$PATH
##############################################################
# Customized "info all" command to get Replicat Processes
##############################################################
X=`${GG_HOME}/ggsci< /tmp/info.log
stats $i,Hourly
exit
!
echo "______________________________"
##############################################################
# Check and send Email if any discard Found. 
##############################################################
Y=`grep discards /tmp/info.log |tr "." " "|awk '{print $3}'`
echo $Y
for c in $Y
do
if [ $c -gt 0 ];then
echo "send email for process $i"
mailx -s "Discards found in $i" emailadd@comapny.com < /tmp/info.log
cat /tmp/info.log
else
echo "dont send email"
fi
done
done
###############################################################

Identify and Diagnostic of Hardware Failure using ILOM on Exadata

To identify Faulty Hardware on Exadata , it use the Sun ILOM.

Below are some steps to Identify and Diagnostics using ILOMs on Exadata Machine using command Lines.

You can Take snapshot and Upload to SR , using ILOM snapshot ASR will help you to Diagnose and provide Further support.

To Take snapshot, Login into ILOM.

Oracle(R) Integrated Lights Out Manager

Version 3.1.2.10 r74387

Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.

-> help
The help command is used to view information about commands and targets

Usage: help [-format wrap|nowrap] [-o|-output terse|verbose]
[|legal|targets|| ]

Special characters used in the help command are
[]   encloses optional keywords or options
<>   encloses a description of the keyword
     (If <> is not present, an actual keyword is indicated)
|    indicates a choice of keywords or options

help               displays description if this target and its
properties
help     displays description of this property of this target
help targets               displays a list of targets
help legal                 displays the product legal notice

Commands are:
cd
create
delete
dump
exit
help
load
reset
set
show
start
stop
version
--- Choose which type of snapshot you want to Take. 

->set /SP/diag/snapshot dataset=data   [normal|full]
->set /SP/diag/snapshot dump_uri= or   ftp://username:pwd@host_ip_address/~

Identify Hardware Failure.

1) Method.

-> show /SP/faultmgmt

 /SP/faultmgmt
    Targets:
        shell
        0 (/SYS/MB/P0/D7)

2) Method. Which is Very Detailed.

-> show -o table -level all /SP/faultmgmt


Target              | Property               | Value
--------------------+------------------------+---------------------------------
/SP/faultmgmt/0     | fru                    | /SYS/MB/P0/D7
/SP/faultmgmt/0/    | class                  | fault.memory.intel.sb.dimm_ce
 faults/0           |                        |
/SP/faultmgmt/0/    | sunw-msg-id            | SPX86-8004-CE
 faults/0           |                        |
/SP/faultmgmt/0/    | component              | /SYS/MB/P0/D7
 faults/0           |                        |
/SP/faultmgmt/0/    | uuid                   | 34d4bfaa-dummy-ebc8-f95a-dummy-
 faults/0           |                        | d17a
/SP/faultmgmt/0/    | timestamp              | 2013-10-05/23:13:06
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_part_number        | 001-0003
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_dash_level         | 01
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_rev_level          | 50
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_serial_number      | 0000dummy00000dummy
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_manufacturer       | Hynix Semiconductor Inc.
 faults/0           |                        |
/SP/faultmgmt/0/    | fru_name               | 8192MB DDR3 SDRAM DIMM
 faults/0           |                        |
/SP/faultmgmt/0/    | system_manufacturer    | Oracle Corporation
 faults/0           |                        |
/SP/faultmgmt/0/    | system_name            | Exadata X3-2
 faults/0           |                        |
/SP/faultmgmt/0/    | system_part_number     | Exadata X3-2
 faults/0           |                        |
/SP/faultmgmt/0/    | system_serial_number   | AK00122916
 faults/0           |                        |
/SP/faultmgmt/0/    | chassis_manufacturer   | Oracle Corporation
 faults/0           |                        |
/SP/faultmgmt/0/    | chassis_name           | SUN FIRE X4270 M3
 faults/0           |                        |
/SP/faultmgmt/0/    | chassis_part_number    | 700000000
 faults/0           |                        |
/SP/faultmgmt/0/    | chassis_serial_number  | 1323XXXXX03F
 faults/0           |                        |
/SP/faultmgmt/0/    | system_component_manuf | Oracle Corporation
 faults/0           | acturer                |
/SP/faultmgmt/0/    | system_component_name  | SUN FIRE X4270 M3
 faults/0           |                        |
/SP/faultmgmt/0/    | system_component_part_ | 70000000
 faults/0           | number                 |
/SP/faultmgmt/0/    | system_component_seria | 1323XXXXX03F
 faults/0           | l_number               |
/SP/faultmgmt/0/    | serd_count             | 0x7b
 faults/0           |                        |
/SP/faultmgmt/0/    | _list_idx              | 0
 faults/0           |                        |
/SP/faultmgmt/0/    | _list_sz               | 1
 faults/0           |                        | 

Rman Backup Shell script

This is backup script to schedule in crontab.
Create three rman command file first.
These Rman commandfile will be called by Below shell script to Kick of which Type of Backup you want.

1) Full backup
2) Inremental Backup
3) Archivelog Backup

To run below backup script provide argument as

./script_name FULL|INCR|ARCH
Backup script

#!/bin/ksh
###########################################################
#######CHECK ARGUMENTS
#
if [ $# -lt 2 ] ;
then
echo "Usage :script_name ORACLE_SID FULL|INCR|ARCH POLICY"
exit
else
echo "Enough Argument Passed"
fi
###########################################################export ALTER SESSION NLS_DATE_FORMAT='DD-MON-YYYY HH24:MI'
export ORACLE_SID=$1
echo $ORACLE_SID
if [[ $( grep -iwc "$ORACLE_SID" /etc/oratab ) -eq 1 ]]
then
echo "ORACLE SID Found"
echo "$ORACLE_SID is instance"
ORACLE_HOME=`grep -iw $ORACLE_SID /etc/oratab | cut -d ":" -f2`
TNS_ADMIN=$ORACLE_HOME/network/admin
PATH=$ORACLE_HOME/bin:$PATH
RMAN=$ORACLE_HOME/bin/rman
echo "Current Oracle Instance :-------------: $ORACLE_SID"
echo "Current Oracle Home :-----------------: $ORACLE_HOME"
echo "Current TNS admin :-------------------: $TNS_ADMIN"
echo "Current RMAN :------------------------: $RMAN"
else
echo "$ORACLE_SID" not found in Oratab file.
exit
fi
###########################################################
export BKTYPE=$2
echo $BKTYPE
case $BKTYPE in
FULL) print "\n You selected FULL DB BACKUP"
export CMD=/u01/test_disk_bkp.rcv
echo $CMD
;;
INCR) print "\n You selected Incremental Backup"
export CMD=/u01/test_disk_incr_bkp.rcv
echo $CMD
;;
ARCH) print "\n You selected Achivelog backup"
export CMD=/u01/test_disk_arch.rcv
echo $CMD
;;
*) print "\n Invalid Argument"
print " choose FULL or INCR or ARCH"
sleep 0
exit
;;
esac
###########################################################
rman  <connect target /
connect catalog rman/dummy@RMAN_CATALOG_STRING
@$CMD
exit
EOF
####################################################################################
RETURN_STATUS=$?
if [ $RETURN_STATUS = 0 ]; then
  echo "$ORACLE_SID Backup successful" | mail -s "$ORACLE_SID backup Completed"
   else
  echo "$ORACLE_SID Backup not successful" | mail -s "$ORACLE_SID backup failed"  
fi
####################################################################################

OGG-01028 Formatting error

Extract can be abend if Parameters are not Set correctly.
Extract was abend due to below Error , after searching on Metalink I found that it is considered as DB bug which fixed in 11.2.0.4.
But there is workaround.

Error

2013-09-24 04:08:30  ERROR   OGG-01028  Formatting error on: table name DUMMY.DUMMY_INQ, rowid AADYhHAHVAAJLIcAAA, XID 8.26.543865, 
position (Seqno 7314, RBA 212298768). Error converting timestamp with timezone from Oracle to ASCII format for column DUMMY_UPDATE.

Metalink Suggestion

GoldenGate extract abends: OGG-01028 Formatting error on: table name xxxx.xxxx rowid , XID nnnn.nn.nnnn, position (Seqno n, RBA n). (Doc ID 1558791.1)

Oracle GoldenGate - Version 11.2.1.0.1 and later
Information in this document applies to any platform.

This is related to database bug 15947884 (base bug 14053498).

In House WorkAround

--Add SETENV parameter before userid 
SETENV (NLS_LANG="AMERICAN_AMERICA.US7ASCII")
--Add SETENV in Extract
TRANLOGOPTIONS INCLUDEREGIONID

Find Transaction SQL from RBA

logdump> open 
logdump> GHDR ON
logdump> DETAIL DATA
logdump> HEADERTOKEN
logdump> GGSTOKEN detail
logdump> pos  
TokenID x47 'G' Record Header    Info x01  Length 1180
TokenID x48 'H' GHDR             Info x00  Length   55
 4504 0041 0417 0fff 02f2 052c 5ca7 4670 0000 0000 | E..A.......,\.Fp....
 05c2 b45c 0001 83f6 0352 0000 0001 4e53 5044 4241 | ...\.....R....dummy
 2e4e 5350 5f49 4e56 454e 544f 5259 00             | .DUMMY_INVENTORY.
TokenID x44 'D' Data             Info x00  Length 1047
TokenID x54 'T' GGS Tokens       Info x00  Length   58
TokenID x5a 'Z' Record Trailer   Info x01  Length 1180
___________________________________________________________________
Hdr-Ind    :     E  (x45)     Partition  :     .  (x04)
UndoFlag   :     .  (x00)     BeforeAfter:     A  (x41)
RecLength  :  1047  (x0417)   IO Time    : 2013/06/12 14:08:51.000.944
IOType     :    15  (x0f)     OrigNode   :   255  (xff)
TransInd   :     .  (x03)     FormatType :     R  (x52)
SyskeyLen  :     0  (x00)     Incomplete :     .  (x00)
---------------------------------------------------------------
---------------------------------------------------------------
--AuditRBA is Redo sequence Number , AuditPos is SCN
AuditRBA   :      99318       AuditPos   : 96646236   
---------------------------------------------------------------
---------------------------------------------------------------
Continued  :     N  (x00)     RecCount   :     1  (x01)

2013/06/12 14:08:51.000.944 FieldComp            Len  1047 RBA 433498317
Name: DUMMYDBA.DUMMY_INVENTORY
After  Image:                                             Partition 4   G  s
 0000 000e 0000 000a 4d37 3266 3833 6464 3034 0001 | ........M72f83dd04..
 0003 0000 4e00 0200 0c00 0000 0847 5244 4e43 4130 | ....N........GRDNCA0
 3100 0300 0900 0000 0531 3256 5450 0004 0009 0000 | 1........12VTP......
 0005 3131 5437 5600 0500 0a00 0000 0633 3130 3231 | ..11T7V........31021
 3700 0600 0a00 0000 0634 3130 3630 3600 0700 0300 | 7........410606.....
 004e 0008 0003 0000 4e00 0900 0300 004e 000a 0008 | .N......N......N....
 0000 0004 4654 544e 000b 0015 ffff 3139 3030 2d30 | ....FTTN......1900-0
Column     0 (x0000), Len    14 (x000e)
 0000 000a 4d37 3266 3833 6464 3034                | ....M72f83dd04
Column     1 (x0001), Len     3 (x0003)
 0000 4e                                           | ..N
Column     2 (x0002), Len    12 (x000c)
 0000 0008 4752 444e 4341 3031                     | ....GRDNCA01
Column     3 (x0003), Len     9 (x0009)
 0000 0005 3132 5654 50                            | ....12VTP
Column     4 (x0004), Len     9 (x0009)
 0000 0005 3131 5437 56                            | ....11T7V
Column     5 (x0005), Len    10 (x000a)
 0000 0006 3331 3032 3137                          | ....310217
Column     6 (x0006), Len    10 (x000a)
 0000 0006 3431 3036 3036                          | ....410606
Column    57 (x0039), Len     4 (x0004)
 ffff 3139 3030 2d30 312d 3031 3a30 303a 3030 3a30 | ..1900-01-01:00:00:0
 30                                                | 0
Column    78 (x004e), Len    21 (x0015)
 ffff 3139 3030 2d30 312d 3031 3a30 303a 3030 3a30 | ..1900-01-01:00:00:0
 30                                                | 0
Column    79 (x004f), Len     4 (x0004)
 ffff 0000                                         | ....
Column    80 (x0050), Len     5 (x0005)
 0000 0001 59                                      | ....Y

GGS tokens:
TokenID x52 'R' ORAROWID         Info x00  Length   20
 4141 4338 6c39 4143 6d41 4141 7a68 3141 4148 0001 | AAC8l9ACmAAAzh1AAH..
TokenID x4c 'L' LOGCSN           Info x00  Length   14
 3133 3332 3539 3536 3332 3432 3337                | 13325956324237
------XID Transactions. 
TokenID x36 '6' TRANID           Info x00  Length   12
 3332 2e32 382e 3831 3139 3133                     | 32.28.811913 

Transaction XID will be at end of Logdump record format.


This way we can find XID,redo log sequence and SCN of Record.

Now how to find Original transaction.

Go to source database.

Find Location of Archivelog sequence which is in AuditRBA in our case it is 99318.

spool output.txt
ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD HH24:MI:SS';
set numformat 9999999999999999999999
set long 999999
set lines 190
set pages 555
exec dbms_logmnr.add_logfile(' complete path of arc seq 99318 '); 
exec dbms_logmnr.start_logmnr(options=>dbms_logmnr.dict_from_online_catalog);
select username, os_username, DATA_OBJ#, rbasqn, RBABLK, rbabyte, RS_ID, row_id, rollback, seg_name,seg_owner,operation, timestamp, sql_redo, scn, cscn from v$logmnr_contents where xidusn || '.' || xidslt || '.' || xidsqn='32.28.811913';
EXECUTE DBMS_LOGMNR.END_LOGMNR;
spool off;

Goldengate User Creation.

This is SQL script to Create GG Admin User. Script Run DBMS_PACKAGE and grants according to Oracle database Version.

It also created Dedicated Tablespace for Goldengate User.

prompt "Enter Goldengate Tablespace Name"
def GGATE_ADMIN_TABLESPACE_NAME=&1
prompt "Enter Goldengate Tablespace Location"
def GG_ADMIN_TABLESPACE_LOCATION=&2
prompt "Enter Goldengate Admin UserName"
def GOLDENGATE_ADMIN_USER=&3
prompt "Enter Goldengate username password"
def GOLDENGATE_ADMIN_USER_PWD=&4
set serveroutput on;
prompt "-------------------------------------------------------------------------"
prompt "This script first drop Old tablespace and Goldengate user if same name exist"
prompt "Provide DISKGROUP NAME LIKE +DATA if Tablespace location is on ASM  "
PROMPT "OR"
prompt "provide filesystem with datafile name like /u01/app/datafile/GGADMIN.dbf"
prompt "-------------------------------------------------------------------------"
spool GG_ADMIN_USER_CREATION.LOG
prompt "Goldengate User must have its own Tablespace it can not be shared with"
Prompt "shared with any other user or ddl_setup script will fail"
--prompt "=======first read above instruction clearly============"
prompt "-------------------------------------------------------------------------"
prompt "------------------------Disable DDL--------------------------------------"
@ddl_disable.sql
SET VERIFY OFF;
prompt "-----------------------Droping any existing Tablespace-------------------"
DROP TABLESPACE &GGATE_ADMIN_TABLESPACE_NAME INCLUDING CONTENTS AND DATAFILES;
prompt "-----------------------Droping any existing User-------------------------"
DROP USER &GOLDENGATE_ADMIN_USER CASCADE;
Prompt "-----------------------creating new Tablespace --------------------------"
CREATE TABLESPACE &GGATE_ADMIN_TABLESPACE_NAME  datafile '&GG_ADMIN_TABLESPACE_LOCATION' SIZE 1500M autoextend on;
--PROMPT "ENTER PASSWORD FOR NEW GOLDENGATE ADMIN USER"
prompt "-----------------------creating new Goldengate Admin User----------------"
CREATE USER &GOLDENGATE_ADMIN_USER IDENTIFIED BY &GOLDENGATE_ADMIN_USER_PWD DEFAULT TABLESPACE &GGATE_ADMIN_TABLESPACE_NAME TEMPORARY TABLESPACE TEMP;
GRANT CONNECT,alter session,CREATE SESSION,RESOURCE,CREATE TABLE TO &GOLDENGATE_ADMIN_USER;
GRANT FLASHBACK ANY TABLE TO &GOLDENGATE_ADMIN_USER;
grant SELECT ANY TRANSACTION to &GOLDENGATE_ADMIN_USER;
grant SELECT ANY TABLE to &GOLDENGATE_ADMIN_USER;
grant SELECT ANY DICTIONARY to &GOLDENGATE_ADMIN_USER;
grant CREATE TABLE to &GOLDENGATE_ADMIN_USER;
grant ALTER SESSION to &GOLDENGATE_ADMIN_USER;
grant UNLIMITED TABLESPACE  to &GOLDENGATE_ADMIN_USER;
grant EXECUTE ON utl_file to &GOLDENGATE_ADMIN_USER;
grant select on v_$instance to &GOLDENGATE_ADMIN_USER;
grant select on v_$database to &GOLDENGATE_ADMIN_USER;
grant LOCK ANY TABLE to &GOLDENGATE_ADMIN_USER;
PROMPT "----------DBA PRIVILEGES WILL BE GRANTED TO GOLDENGATE_ADMIN_USER--------"
GRANT DBA TO &GOLDENGATE_ADMIN_USER;
Prompt "-------------------------------------------------------------------------"
Prompt "-------------------------------------------------------------------------"
Prompt " Running PLSQL BLOCK to execute procedure and grants according to Oracle version"
Prompt "-------------------------------------------------------------------------"
Prompt "-------------------------------------------------------------------------"
Prompt
Prompt


declare
v1 VARCHAR2(200);
v2 varchar2(200);
V3 varchar2(1000);
l_sql varchar2(1000);
begin
select version into v1 from DBA_REGISTRY WHERE COMP_NAME LIKE '%Catalog Views%';
 CASE
      WHEN v1 like '10.2%' THEN
DBMS_OUTPUT.PUT_LINE('it is Oracle version 10.2');
execute immediate 'BEGIN dbms_streams_auth.grant_admin_privilege(''&GOLDENGATE_ADMIN_USER''); END;';
EXECUTE IMMEDIATE 'grant insert on system.logmnr_restart_ckpt$ to &GOLDENGATE_ADMIN_USER';
EXECUTE IMMEDIATE 'grant update on sys.streams$_capture_process to &GOLDENGATE_ADMIN_USER';
EXECUTE IMMEDIATE 'grant become user to &GOLDENGATE_ADMIN_USER';
dbms_output.put_line('Executing dbms_streams_auth.grant_admin_privilege');
WHEN v1 like '11.1%' THEN
DBMS_OUTPUT.PUT_LINE('it is Oracle version 11.1');
execute immediate 'BEGIN dbms_streams_auth.grant_admin_privilege(''&GOLDENGATE_ADMIN_USER''); END;';
EXECUTE IMMEDIATE 'grant become user to &GOLDENGATE_ADMIN_USER';
dbms_output.put_line('Executing dbms_streams_auth.grant_admin_privilege');
WHEN v1 like '11.2.0.3%' THEN
DBMS_OUTPUT.PUT_LINE('it is Oracle version 11.2.0.3');
EXECUTE IMMEDIATE 'begin dbms_goldengate_auth.grant_admin_privilege(''&GOLDENGATE_ADMIN_USER''); END;';
dbms_output.put_line('Executing dbms_goldengate_auth.grant_admin_privilege');
    END CASE;
exception when others
then
dbms_output.put_line('ERROR IS ' || SQLCODE || SQLERRM);
end;
/
prompt "---------------------script end-------------------------------------------"
prompt
prompt
prompt "~~~~~~~~please verify GG_ADMIN_USER_CREATION.LOG for Errors ~~~~~~~~~~~~~~"
prompt
prompt
prompt
spool OFF;