TNS-03505: Failed to resolve name

Tnsentry was something like below.

TXCOM =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = TEST11v)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = TEST12v)(PORT = 1521))
    (LOAD_BALANCE = yes)
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = TXCOM.world)
    )
  )
  

it gave error TNS-03505: Failed to resolve name , which mostly happen due to bad syntax.

TEST11 | DUMMY1 | /u00/app/oracle/product/10.2.0/DB/network/admin
> tnsping TXCOM

TNS Ping Utility for Linux: Version 10.2.0.4.0 - Production on 04-JUN-2012 09:55:54

Copyright (c) 1997,  2007, Oracle.  All rights reserved.

Used parameter files:

TNS-03505: Failed to resolve name

This need to check why its not able to find TNSnames. before that make sure you corrected syntax of tnsentry in tnsnames.ora

lets check what is $TNS_ADMIN set to.

TEST11 | DUMMY1 | /u00/app/oracle/product/10.2.0/DB/network/admin
> echo $TNS_ADMIN
/u00/app/oracle/product/10.2.0/ASM/network/admin
TEST11 | DUMMY1 | /u00/app/oracle/product/10.2.0/DB/network/admin
> export TNS_ADMIN=/u00/app/oracle/product/10.2.0/DB/network/admin
TEST11 | DUMMY1 | /u00/app/oracle/product/10.2.0/DB/network/admin
> tnsping TXCOM

TNS Ping Utility for Linux: Version 10.2.0.4.0 - Production on 04-JUN-2012 09:58:00

Copyright (c) 1997,  2007, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = TEST11v)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = dukegotd99v)(PORT = 1521)) (LOAD_BALANCE = yes) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TXCOM.world)))
OK (0 msec)
TEST11 | DUMMY1 | /u00/app/oracle/product/10.2.0/DB/network/admin

ORA-12514 on +ASM , ASM & ORA-12514 , ASM tnsnames.ora

Environment was 3 Node RAC running on Grid Infrastructure. 11.2.0.1

I tried to connect to +ASM instances using sqlplus from same node using TNSnames but i got ORA-12514.

SQL*Plus: Release 11.2.0.1.0 Production on Tue May 29 19:44:36 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

ERROR:
ORA-12514: TNS:listener does not currently know of service requested in connect
descriptor


Enter user-name: 

Both Listener is running.

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM $ps -ef | grep tns
oracle    9661     1  0 18:54 ?        00:00:00 /u00/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle    9696     1  0 18:55 ?        00:00:00 /u00/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit

Below is my configuration.

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $vi listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))             # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                                         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                                         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                                         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON                                               # line added by Agent

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $vi endpoints_listener.ora
LISTENER_RACG1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG1-VIP)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.100.121)(PORT=1521)(IP=FIRST))))  # line added by Agent

Tnsentry.

+ASM =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = RACG-SCAN)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = +ASM)
    )
  )

Here are services supported by from both listener.

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $lsnrctl services listener

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 29-MAY-2012 19:45:05

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         LOCAL SERVER
Service "RACT.localdomain.com" has 1 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         LOCAL SERVER
Service "RACTXDB.localdomain.com" has 1 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG1.localdomain.com)(PORT=54564))
The command completed successfully

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $lsnrctl services listener_scan2

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 29-MAY-2012 19:45:24

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
Services Summary...
Service "RACT.localdomain.com" has 3 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG1-VIP)(PORT=1521)))
  Instance "RACT2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG2-VIP)(PORT=1521)))
  Instance "RACT3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG3-VIP)(PORT=1521)))
Service "RACTXDB.localdomain.com" has 3 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG1.localdomain.com)(PORT=54564))
  Instance "RACT2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG2.localdomain.com)(PORT=65286))
  Instance "RACT3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG3.localdomain.com)(PORT=39505))
The command completed successfully

Here What we needed to do ....

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue May 29 19:46:21 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> show parameter remote

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
remote_listener                      string
remote_login_passwordfile            string      EXCLUSIVE
remote_os_authent                    boolean     FALSE
remote_os_roles                      boolean     FALSE

Just add REMOTE_LISTENER pointing to SCAN ip address.

SQL> ALTER SYSTEM SET remote_listener='RACG-SCAN:1521' scope=both;

System altered.

SQL> ALTER SYSTEM REGISTER;

System altered.

As you can see how it reflected in scan listener

RACG1@:/u00/app/11.2.0/grid/network/admin :+ASM1 $lsnrctl services listener_scan2

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 29-MAY-2012 19:48:42

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
Services Summary...
Service "+ASM" has 3 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:3 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG1-VIP)(PORT=1521)))
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG2-VIP)(PORT=1521)))
  Instance "+ASM3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG3-VIP)(PORT=1521)))
Service "RACT.localdomain.com" has 3 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG1-VIP)(PORT=1521)))
  Instance "RACT2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:1 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG2-VIP)(PORT=1521)))
  Instance "RACT3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=RACG3-VIP)(PORT=1521)))
Service "RACTXDB.localdomain.com" has 3 instance(s).
  Instance "RACT1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG1.localdomain.com)(PORT=54564))
  Instance "RACT2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG2.localdomain.com)(PORT=65286))
  Instance "RACT3", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER 
         (ADDRESS=(PROTOCOL=tcp)(HOST=RACG3.localdomain.com)(PORT=39505))
The command completed successfully

Uninstall Oracle 11g on Unix/ Remove Oracle 11g Unix

Here is procedure to Uninstall 11g Oracle Home. earlier it used to done by OUI.

DEVL:/u00/app/oracle/product/11.2.0/db_1/deinstall :TEST$./deinstall
Checking for required files and bootstrapping ...
Please wait ...
Location of logs /tmp/deinstall2012-05-07_07-55-03-AM/logs/

############ ORACLE DEINSTALL & DECONFIG TOOL START ############


######################## CHECK OPERATION START ########################
Install check configuration START


Checking for existence of the Oracle home location /u00/app/oracle/product/11.2.0/db_1
Oracle Home type selected for de-install is: SIDB
Oracle Base selected for de-install is: /u00/app/oracle
Checking for existence of central inventory location /u00/app/oraInventory

Install check configuration END


Network Configuration check config START

Network de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/netdc_check8067592795646126333.log

Specify all Single Instance listeners that are to be de-configured [LISTENER]:

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/databasedc_check8855114124755634650.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home [TEST]:

###### For Database 'TEST' ######

Single Instance Database
The diagnostic destination location of the database: /u00/app/oracle/diag/rdbms/test
Storage type used by the Database: FS
Database file location: /u01/test/data
Flash recovery area location: Does not exist
database spfile location: /u00/app/oracle/product/11.2.0/db_1/dbs/spfileTEST.ora

The details of database(s) TEST have been discovered automatically. Do you still want to modify the details of TEST database(s)? [n]: n

Database Check Configuration END

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/emcadc_check.log

Checking configuration for database TEST
Enterprise Manager Configuration Assistant END
Oracle Configuration Manager check START
OCM check log file location : /tmp/deinstall2012-05-07_07-55-03-AM/logs//ocm_check3012.log
Oracle Configuration Manager check END

######################### CHECK OPERATION END #########################


####################### CHECK OPERATION SUMMARY #######################
Oracle Home selected for de-install is: /u00/app/oracle/product/11.2.0/db_1
Inventory Location where the Oracle home registered is: /u00/app/oraInventory
Following Single Instance listener(s) will be de-configured: LISTENER
The following databases were selected for de-configuration : TEST
Database unique name : TEST
Storage used : FS
No Enterprise Manager configuration to be updated for any database(s)
No Enterprise Manager ASM targets to update
No Enterprise Manager listener targets to migrate
Checking the config status for CCR
Oracle Home exists with CCR directory, but CCR is not configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/tmp/deinstall2012-05-07_07-55-03-AM/logs/deinstall_deconfig2012-05-07_07-56-11-AM.out'
Any error messages from this session will be written to: '/tmp/deinstall2012-05-07_07-55-03-AM/logs/deinstall_deconfig2012-05-07_07-56-11-AM.err'

######################## CLEAN OPERATION START ########################

Enterprise Manager Configuration Assistant START

EMCA de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/emcadc_clean.log

Updating Enterprise Manager ASM targets (if any)
Updating Enterprise Manager listener targets (if any)
Enterprise Manager Configuration Assistant END
Database de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/databasedc_clean1221138661467301182.log
Database Clean Configuration START TEST
This operation may take few minutes.
Database Clean Configuration END TEST

Network Configuration clean config START

Network de-configuration trace file location: /tmp/deinstall2012-05-07_07-55-03-AM/logs/netdc_clean6816269865994649600.log

De-configuring Single Instance listener(s): LISTENER

De-configuring listener: LISTENER
    Stopping listener: LISTENER
    Warning: Failed to stop listener. Listener may not be running.
    Deleting listener: LISTENER
    Listener deleted successfully.
Listener de-configured successfully.

De-configuring Listener configuration file...
Listener configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file...
Local Net Service Names configuration file de-configured successfully.

De-configuring backup files...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /tmp/deinstall2012-05-07_07-55-03-AM/logs//ocm_clean3012.log
Oracle Configuration Manager clean END
Oracle Universal Installer clean START

Detach Oracle home '/u00/app/oracle/product/11.2.0/db_1' from the central inventory on the local node : Done

Delete directory '/u00/app/oracle/product/11.2.0/db_1' on the local node : Done

Delete directory '/u00/app/oraInventory' on the local node : Done

Delete directory '/u00/app/oracle' on the local node : Done

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


Oracle install clean START

Clean install operation removing temporary directory '/tmp/install' on node 'DEVL'

Oracle install clean END

Moved default properties file /tmp/deinstall2012-05-07_07-55-03-AM/response/deinstall_OraDb11g_home1.rsp as /tmp/deinstall2012-05-07_07-55-03-AM/response/deinstall_OraDb11g_home1.rsp0

######################### CLEAN OPERATION END #########################


####################### CLEAN OPERATION SUMMARY #######################
Successfully de-configured the following database instances : TEST
Following Single Instance listener(s) were de-configured successfully: LISTENER
Cleaning the config for CCR
As CCR is not configured, so skipping the cleaning of CCR configuration
CCR clean is finished
Successfully detached Oracle home '/u00/app/oracle/product/11.2.0/db_1' from the central inventory on the local node.
Successfully deleted directory '/u00/app/oracle/product/11.2.0/db_1' on the local node.
Successfully deleted directory '/u00/app/oraInventory' on the local node.
Successfully deleted directory '/u00/app/oracle' on the local node.
Oracle Universal Installer cleanup was successful.


Run 'rm -rf /etc/oraInst.loc' as root on node(s) 'DEVL' at the end of the session.

Oracle install successfully cleaned up the temporary directories.
#######################################################################


############# ORACLE DEINSTALL & DECONFIG TOOL END #############

DEVL:/u00/app/oracle/product/11.2.0/db_1/deinstall :TEST$

export fails with ORA-00600 [qmtGetColumnInfo1]

I was taking export as below .

/testc/qual/orasw10/app/oracle/admin/testdummy6/scripts (Dummy_server) $ cat export.sh
export ORACLE_HOME=/testc/qual/orasw10/app/oracle/product/10.2.0.3
export PATH=/testc/qual/orasw10/app/oracle/product/10.2.0.3/bin:$PATH
export ORACLE_SID=testdummy6
exp userid=system/**** owner=stscorp file=/testc/qual/testdummy6/arch/stscorp.dmp consistent=y STATISTICS=None

and i have hit below error.

Export: Release 10.2.0.4.0 - Production on Tue May 1 14:20:29 2012

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application testing options
Export done in US7ASCII character set and AL16UTF16 NCHAR character set
server uses WE8ISO8859P1 character set (possible charset conversion)

About to export specified users ...
. exporting pre-schema procedural objects and actions
. exporting foreign function library names for user dummy_schema
. exporting PUBLIC type synonyms
. exporting private type synonyms
. exporting object type definitions for user dummy_schema
About to export dummy_schema's objects ...
. exporting database links
. exporting sequence numbers
. exporting cluster definitionsWarning: Entry/Exit code is optimized.  Cannot restore context (UNWIND 22)

EXP-00056: ORACLE error 600 encountered
ORA-00600: internal error code, arguments: [qmtGetColumnInfo1], [], [], [], [], [], [], []
EXP-00000: Export terminated unsuccessfully

I check that is there any cluster tables are in schema or not.

SQL> SELECT OWNER,SEGMENT_NAME,TABLESPACE_NAME FROM DBA_SEGMENTS WHERE SEGMENT_TYPE='CLUSTER';

OWNER                          SEGMENT_NAME         TABLESPACE_NAME
------------------------------ -------------------- ------------------------------
SYS                            C_USER#              SYSTEM
SYS                            C_TS#                SYSTEM
SYS                            C_OBJ#               SYSTEM
SYS                            C_FILE#_BLOCK#       SYSTEM
SYS                            C_COBJ#              SYSTEM
SYS                            C_MLOG#              SYSTEM
SYS                            C_TOID_VERSION#      SYSTEM
SYS                            C_RG#                SYSTEM
SYS                            C_OBJ#_INTCOL#       SYSTEM
SYS                            SMON_SCN_TO_TIME     SYSTEM

so i have open alert log, and found below contents.

Tue May  1 14:20:37 2012
Errors in file /testc/qual/testdummy6/admin/udump/testdummy6_ora_17082.trc:
ORA-00600: internal error code, arguments: [qmtGetColumnInfo1], [], [], [], [], [], [], []

Below is call stack from trace file.

/testc/qual/testdummy6/admin/udump/testdummy6_ora_17082.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application testing options
ORACLE_HOME = /testc/qual/orasw10/app/oracle/product/10.2.0.4_2
System name:    HP-UX
Node name:      Dummy_server
Release:        B.11.11
Version:        U
Machine:        9000/800
Instance name: testdummy6
Redo thread mounted by this instance: 1
Oracle process number: 25
Unix process pid: 17082, image: oracle@Dummy_server (TNS V1-V3)

*** ACTION NAME:() 2012-05-01 14:20:37.159
*** MODULE NAME:(exp@Dummy_server (TNS V1-V3)) 2012-05-01 14:20:37.159
*** SERVICE NAME:(SYS$USERS) 2012-05-01 14:20:37.159
*** SESSION ID:(142.3367) 2012-05-01 14:20:37.159
*** 2012-05-01 14:20:37.159
ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [qmtGetColumnInfo1], [], [], [], [], [], [], []
No current SQL statement being executed.
----- Call Stack Trace -----
calling              call     entry                argument values in hex
location             type     point                (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+40          ?        ksedst1()            4000000001ADAB5B ?
                                                   000000000 ?
                                                   800003F9FFFFF1B8 ?
                                                   800003FFC0012ED8 ?
ksedmp()+424         ?        ksedst()             800003F9FFFFF1B8 ?
                                                   800003FFC0012ED8 ?
                                                   800003FFC0014400 ?
                                                   1000000010 ?
ksfdmp()+32          ?        ksedmp()             800003FFC0012980 ?
                                                   4000000001B35633 ?
                                                   000000006 ? 000000000 ?
kgerinv()+152        ?        ksfdmp()             000000006 ? 000000000 ?
                                                   C00000007FFFFFFF ?
                                                   4020000000000000 ?
kgeasnmierr()+88     ?        kgerinv()            000000000 ?
                                                   8000000100147A00 ?
                                                   000000458 ?
                                                   8000000000000001 ?
qmtkGetColumnInfo()  ?        kgeasnmierr()        4000000080 ? 00000000C ?
+140                                               C000000134155608 ?
                                                   C000000134B371B4 ?
koksxmlticm()+3908   ?        qmtkGetColumnInfo()  800003F7C006A9F0 ?
                                                   C000000134156DA0 ?
                                                   000000000 ?
                                                   8000000100013468 ?
koksicm()+1120       ?        koksxmlticm()        C000000134156848 ?
                                                   5101002000 ? 000000000 ?
                                                   000000001 ?
kokaccx()+1304       ?        koksicm()            000000001 ?
                                                   800003FFC000D560 ?
                                                   C00000013FF6F8B0 ?
                                                   1000000000000000 ?
kokacmc()+64         ?        kokaccx()            C000000134B37228 ?
                                                   4000000001BD19B3 ?
                                                   00005E8A8 ?
                                                   800003FFC0010E88 ?

I have put this call stack into Metalink Note

ORA-600/ORA-7445 Error Look-up Tool [ID 153788.1]

Which suggest me to export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH.

but no luck again.So I check dba_registry and found that Oracle XML Database is invalid in XDB schema.

COMP_NAME                                     SCHEMA               STATUS      VERSION
--------------------------------------------- -------------------- ----------- ------------------------------
Oracle Enterprise Manager                     SYSMAN               VALID       10.2.0.4.0
Spatial                                       MDSYS                VALID       10.2.0.4.0
Oracle interMedia                             ORDSYS               VALID       10.2.0.4.0
OLAP Catalog                                  OLAPSYS              VALID       10.2.0.4.0
Oracle XML Database                           XDB                  INVALID     10.2.0.4.0
Oracle Text                                   CTXSYS               VALID       10.2.0.4.0
Oracle Expression Filter                      EXFSYS               VALID       10.2.0.4.0
Oracle Rule Manager                           EXFSYS               VALID       10.2.0.4.0
Oracle Workspace Manager                      WMSYS                VALID       10.2.0.4.3
Oracle Data Mining                            DMSYS                VALID       10.2.0.4.0
Oracle Database Catalog Views                 SYS                  VALID       10.2.0.4.0
Oracle Database Packages and Types            SYS                  VALID       10.2.0.4.0
JServer JAVA Virtual Machine                  SYS                  VALID       10.2.0.4.0
Oracle XDK                                    SYS                  VALID       10.2.0.4.0
Oracle Database Java Packages                 SYS                  VALID       10.2.0.4.0
OLAP Analytic Workspace                       SYS                  VALID       10.2.0.4.0
Oracle OLAP API                               SYS                  VALID       10.2.0.4.0

This time i have search Metalink again found something similar.

Datapump Export expdp Fails With ORA-600 [unable to load XDB library] [ID 736049.1]

where solution was given as below.

As the SYSDBA user, please run the following scripts :-

SQL>@/rdbms/admin/catmeta.sql
SQL>@/rdbms/admin/catmet2.sql
SQL>@/rdbms/admin/utlrp.sql

I executed above scripts and tried again this time i got below error.

Export: Release 10.2.0.4.0 - Production on Tue May 1 14:50:44 2012

Copyright (c) 1982, 2007, Oracle.  All rights reserved.


Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application testing options
Export done in US7ASCII character set and AL16UTF16 NCHAR character set
server uses WE8ISO8859P1 character set (possible charset conversion)

About to export specified users ...
. exporting pre-schema procedural objects and actions
. exporting foreign function library names for user dummy_schema
. exporting PUBLIC type synonyms
. exporting private type synonyms
. exporting object type definitions for user dummy_schema
About to export dummy_schema's objects ...
. exporting database links
. exporting sequence numbers
. exporting cluster definitions
EXP-00056: ORACLE error 932 encountered
ORA-00932: inconsistent datatypes: expected BLOB, CLOB got CHAR
EXP-00000: Export terminated unsuccessfully

Since it was 10.2.0.4 ,I tried to run export using Datapump. It went through.

Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 4.590 GB
Processing object type SCHEMA_EXPORT/USER
Processing object type SCHEMA_EXPORT/SYSTEM_GRANT
Processing object type SCHEMA_EXPORT/ROLE_GRANT
Processing object type SCHEMA_EXPORT/DEFAULT_ROLE
Processing object type SCHEMA_EXPORT/TABLESPACE_QUOTA
Processing object type SCHEMA_EXPORT/PASSWORD_HISTORY
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/SEQUENCE/SEQUENCE
Processing object type SCHEMA_EXPORT/SEQUENCE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type SCHEMA_EXPORT/TABLE/AUDIT_OBJ
Processing object type SCHEMA_EXPORT/FUNCTION/FUNCTION
Processing object type SCHEMA_EXPORT/FUNCTION/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/FUNCTION/ALTER_FUNCTION
Processing object type SCHEMA_EXPORT/VIEW/VIEW
Processing object type SCHEMA_EXPORT/VIEW/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/CONSTRAINT/REF_CONSTRAINT
Processing object type SCHEMA_EXPORT/TABLE/TRIGGER
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS

Rename Diskgroup.

1) Unmount Diskgroup on All Nodes.

RACG1@:/home/oracle :+ASM1 $asmcmd -p
ASMCMD [+] > umount RACG

RACG2@:/home/oracle :+ASM2 $asmcmd -p
ASMCMD [+] > umount RACG

2) Check on Diskgroup is not mounted.

RACG1@:/dev/oracleasm/disks :+ASM1 $crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRSDG.dg
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
ora.LISTENER.lsnr
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
ora.RACG.dg
               OFFLINE OFFLINE      racg1
               OFFLINE OFFLINE      racg2
ora.asm
               ONLINE  ONLINE       racg1                    Started
               ONLINE  ONLINE       racg2                    Started
ora.eons
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
ora.gsd
               OFFLINE OFFLINE      racg1
               OFFLINE OFFLINE      racg2
ora.net1.network
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
ora.ons
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
ora.registry.acfs
               ONLINE  ONLINE       racg1
               ONLINE  ONLINE       racg2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racg2
ora.oc4j
      1        OFFLINE OFFLINE
ora.racg1.vip
      1        ONLINE  ONLINE       racg1
ora.racg2.vip
      1        ONLINE  ONLINE       racg2
ora.scan1.vip
      1        ONLINE  ONLINE       racg2

3) Now lets rename Diskgroup from +RACG to +RACD

RACG1@:/dev/oracleasm/disks :+ASM1 $renamedg phase=both dgname=RACG newdgname=RACD verbose=true
NOTE: No asm libraries found in the system

Parsing parameters..

Parameters in effect:

         Old DG name       : RACG
         New DG name          : RACD
         Phases               :
                 Phase 1
                 Phase 2
         Discovery str        : (null)
         Clean              : TRUE
         Raw only           : TRUE
renamedg operation: phase=both dgname=RACG newdgname=RACD verbose=true
Executing phase 1
Discovering the group
Performing discovery with string:
KFNDG-00407: file not found; arguments: []

Terminating kgfd context 0x2ba7587250a0

- it failed with KFNDG-00407: file not found; arguments: []
-Lets try again , Add asm_diskstring='/dev/oracleasm/disks/*' to command.

RACG1@:/dev/oracleasm/disks :+ASM1 $renamedg phase=both dgname=RACG newdgname=RACD verbose=true asm_diskstring='/dev/oracleasm/disks/*'
NOTE: No asm libraries found in the system

Parsing parameters..

Parameters in effect:

         Old DG name       : RACG
         New DG name          : RACD
         Phases               :
                 Phase 1
                 Phase 2
         Discovery str        : /dev/oracleasm/disks/*
         Clean              : TRUE
         Raw only           : TRUE
renamedg operation: phase=both dgname=RACG newdgname=RACD verbose=true asm_diskstring=/dev/oracleasm/disks/*
Executing phase 1
Discovering the group
Performing discovery with string:/dev/oracleasm/disks/*
Identified disk UFS:/dev/oracleasm/disks/DISK2 with disk number:0 and timestamp (32969226 -1320570880)
Identified disk UFS:/dev/oracleasm/disks/DISK3 with disk number:1 and timestamp (32969226 -1320570880)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:/dev/oracleasm/disks/*
Identified disk UFS:/dev/oracleasm/disks/DISK2 with disk number:0 and timestamp (32969226 -1320570880)
Identified disk UFS:/dev/oracleasm/disks/DISK3 with disk number:1 and timestamp (32969226 -1320570880)
Checking if the diskgroup is mounted
Checking disk number:0
Checking disk number:1
Checking if diskgroup is used by CSS
Generating configuration file..
KFNDG-00305: file not found

Terminating kgfd context 0x2b03e7d4d0a0

-Again it failed with KFNDG-00305: file not found
-This time Add confirm=true config=/tmp/renamedg to command.

RACG1@:/dev/oracleasm/disks :+ASM1 $renamedg phase=both dgname=RACG newdgname=RACD confirm=true config=/tmp/renamedg verbose=true asm_diskstring='/dev/oracleasm/disks/*'
NOTE: No asm libraries found in the system

Parsing parameters..

Parameters in effect:

         Old DG name       : RACG
         New DG name          : RACD
         Phases               :
                 Phase 1
                 Phase 2
         Discovery str        : /dev/oracleasm/disks/*
         Confirm            : TRUE
         Clean              : TRUE
         Raw only           : TRUE
renamedg operation: phase=both dgname=RACG newdgname=RACD confirm=true config=/tmp/renamedg verbose=true asm_diskstring=/dev/oracleasm/disks/*
Executing phase 1
Discovering the group
Performing discovery with string:/dev/oracleasm/disks/*
Identified disk UFS:/dev/oracleasm/disks/DISK2 with disk number:0 and timestamp (32969226 -1320570880)
Identified disk UFS:/dev/oracleasm/disks/DISK3 with disk number:1 and timestamp (32969226 -1320570880)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:/dev/oracleasm/disks/*
Identified disk UFS:/dev/oracleasm/disks/DISK2 with disk number:0 and timestamp (32969226 -1320570880)
Identified disk UFS:/dev/oracleasm/disks/DISK3 with disk number:1 and timestamp (32969226 -1320570880)
Checking if the diskgroup is mounted
Checking disk number:0
Checking disk number:1
Checking if diskgroup is used by CSS
Generating configuration file..
Completed phase 1
Executing phase 2
Looking for /dev/oracleasm/disks/DISK2
Modifying the header
Looking for /dev/oracleasm/disks/DISK3
Modifying the header
Completed phase 2
Terminating kgfd context 0x2b2b6ce750a0

And it successfully Renamed.

4) Mount Renamed Diskgroup +RACD on All Nodes.

RACG1@:/home/oracle :+ASM1 $asmcmd mount RACD
RACG2@:/home/oracle :+ASM2 $asmcmd mount RACD

5) Confirm on All Nodes.

RACG1@:/home/oracle :+ASM1 $asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576      1019      623                0             623              0             N  CRSDG/
MOUNTED  EXTERN  N         512   4096  1048576      2038     1941                0            1941              0             N  RACD/

RACG2@:/home/oracle :+ASM2 $asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576      1019      623                0             623              0             N  CRSDG/
MOUNTED  EXTERN  N         512   4096  1048576      2038     1941                0            1941              0             N  RACD/

Zero down-time Migration using Goldengate of Oracle database from 8i to 11g.

Hi

Few months back , I did migration of  Oracle database from 8i to 11gR1 on cross-platform using GoldenGate V10 .  3 minute downtime was required.
I would like to share a Overview of whole plan , which could be helpful for reference.

Fundamentally this method works in most of the migration using Goldengate .
Data Loading from source to target has to be decided.

It could be either export/import, ETL tool, Transportable Tablespace,Backup.

Note: challenges we faced was to capture & replicate CLOB data.

We moved  4 TB Oracle 8i database from NY to North Carolina. It was geographically separated but in same Timezone.

Some facts.

• Data moved across databases using Export / Import utility. 
• FTP used to move dumpfiles from Source to target destination server. (NFS will be better)
• Goldengate was configure to replicate schema to schema. (Tables which as has CLOB data were excluded). 
• Index were created on Target database for big tables or table subject to heavy transactions to boost replicate on target db.( Index were dropped after migration ).
• Big Tables or Heavy transactions Tables can be split into few capture processes using @RANGE parameter.
• Triggers were disable for performance reason.  its not mandatory to disable triggers.  
• As source database was Oracle 8i , Trandata need to add for every table on source Database.  
• Handlecolision parameter must be used on replicate side. 
• 3 Minutes downtime was Achieved. 
• There should be enough space to accommodate remote trailfiles on Target server.

Below is step-by-step Migration from 8i to 11g.

Timestamp	Source	Target	Comments.
	New york (8i )	North Carolina (11g)
Sunday 8:00	Configure goldengate for capture and datapump processes.	Configure goldengate for Replicate processes.	Test this configureation in Dev/Qa.
Monday 8:00	Start capture and datapump processes.	Dont start replicate now.	Trailfiles from capture are transferred by datapump. This will be accumulate on target server.
Monday 9:00	start export from 8i database (Mark SCN when export started.)		lag for replicate will increase but dont start replicate yet.
Tuesday 22:00	Export completed. start FTP of dumpfile to target server.
Wednesday 14:00	FTP completed.	Start Import on Target database using dumpfiles.
Friday 20:00		Import Finished.	Index created & trigger were disabled after Import finished.
Friday 22:00		Start replicat using atscn / Use HANDLECOLLITION	it will apply all changes from trailfiles which was accumulated on target server til now.  Lag will decrease slowly for replicate.
Sunday 10:00		Replicate applied all changes from trailfiles.	Monitor Lag for capture and datapump process.
sunday 11:00 (cut-over time)	when there are least transactions happens or Lock db for further login and wait till capture shows zero lag  , after this stop capture. when lag is zero for datapump ,stop datapump	wait till replicate apply all changes from last trailfile  , lag should be zero for replicate. After this stop replicate.	This whole downtime depends on volume of transaction and latency between source and target in term of Goldengate. Drop indexes & Enable all triggers.
Sunday 11:03 (cut-over time)	Redirect db connection point to target db.	Redirect db connection point to target db.	Migration completed.

We started capture process to capture all transaction on source db, till export and import finish.

HANDLECOLLISION was specially used on replicate side , so we can avoid duplicate transactions.

Cut-over is explained.

Source side.
Monitor lag of capture process on source side.
When there are very low transactions happen or we could also use approach to lock db for further loggin. Now when all capture processes shows zero lag. stop the all capture process. Once this is complete wait till datapump reach to zero lag, Now stop datapump too.
Idea behind this is to make sure that No transaction occurs after we stop capture.
As of now All transactions are captured , transmitted to target side on remote trailfile.

Target side.

Wait till replicate apply all changes till end of trailfile. There should be zero lag for all replicate processes.
Once this done , stop replicate.


Now change connection string to target database. which will redirect connection to new 11g database.

Drop indexes which was created to boost replicate processes. enable all triggers.

----------2nd approach-----------------

• configure GG  extract/pump/replicat
• start extract & pump (Make sure remote trailfile is generated), You can also take SCN based backup by RMAN. 
• Take export of Whole Source database ,based on SCN. 
• FTP dumpfiles to Target database server.
• Import into Target database.
• start replicat ,AFTERCSN  
• Let replicat catchup.  
• once extract/pump/replicat shows zero lag/checkpoint ,stop further login into source DB. 
• switch tns to Target db. 

This whole cut-over time could be estimate and strongly depend on volume of transaction & latency between capture and replicate. Measure the redo generation & latency at peak-time.


Happy Migration.

How to "kill Semaphore of Oracle Instance" / "Remove Shared memory of Oracle Instnace"

We had Oracle instance running , Pmon of Instance was also visible .
While connecting to Oracle database it looked Normal , But while query V$database.It throws error.

So I had take look into Tail of alert log , Instance was cleanly shutdown. On Attempt to startup it gave below error.

ORA-01081: cannot start already-running ORACLE - shut it down first

So,

• PMON is visible,
• Alertlog shows clean shutdown of instnace last time.
• you cant query even v$database or v$instnace.

This whole scenario leads to conclusion that semaphore of Instance is still locked.

And here we go .

To see how many Oracle symaphores/Shared memory are running.

ipcs | grep oracle
m  916291602 0xc0a8ed2c --rw-rw----    oracle       dba
m  154730522 0xee8c7308 --rw-rw----    oracle       dba
m   34111516 0x8fe194f8 --rw-rw----    oracle       dba
m  975011880 0xa1b69514 --rw-rw----    oracle       dba
s  237666346 0xe0695054 --ra-ra----    oracle       dba
s 1828749356 0x74616ad0 --ra-ra----    oracle       dba
s 1802698797 0xf9afd36c --ra-ra----    oracle       dba
s 1491337262 0xf40cfa70 --ra-ra----    oracle       dba

To identify which symaphore is connected to which Oracle instance .

echo $ORACLE_SID
VVCBDMQ1

sysresv is command to find symaphore attached to Instance.It will also shows shared memory address.

sysresv
IPC Resources for ORACLE_SID "VVCBDMQ1" :
Shared Memory:
ID              KEY
34111516        0x8fe194f8     
Semaphores:
ID              KEY
237666346       0xe0695054
Oracle Instance not alive for sid "VVCBDMQ1"

To confirm do as below

ipcs -a | grep oracle | grep  | grep 

ipcs -a | grep oracle | grep 34111516 | grep 0x8fe194f8
ipcs -a | grep oracle | grep 237666346 | grep 0xe0695054

To clean up symaphore.

ipcrm -s 34111516   <- ID
ipcrm -M 0x8fe194f8 <- key 

Check again to confirm it.

sysresv
IPC Resources for ORACLE_SID "VVCBDMQ1" :
Shared Memory
ID              KEY
No shared memory segments used
Semaphores:
ID              KEY
No semaphore resources used
Oracle Instance not alive for sid "VVCBDMQ1"