Friday, August 3, 2012

ORA-01041: internal error. hostdef extension doesn't exist

I had complete crash of 3 NODE RAC on 11.2.0.2 on All Node.

Cluster it self is quite Intelligent. Once i started up the cluster.

I rebooted all node twice. It brought all Resources except Database.


ora.cvu
      1        ONLINE  ONLINE       pnetn1
ora.edwp.db
      1        OFFLINE OFFLINE
      2        OFFLINE OFFLINE
      3        OFFLINE OFFLINE
ora.oc4j
      1        ONLINE  ONLINE       pnetn1
ora.pnetn1.vip
      1        ONLINE  ONLINE       pnetn1
ora.pnetn2.vip
      1        ONLINE  ONLINE       pnetn2
ora.pnetn3.vip
      1        ONLINE  ONLINE       pnetn3
ora.scan1.vip
      1        ONLINE  ONLINE       pnetn3
ora.scan2.vip
      1        ONLINE  ONLINE       pnetn2
ora.scan3.vip
      1        ONLINE  ONLINE       pnetn1
[root@PNETN1 ~]# crsctl start resource ora.edwp.db
CRS-2672: Attempting to start 'ora.edwp.db' on 'pnetn3'
CRS-2679: Attempting to clean 'ora.edwp.db' on 'pnetn3'
CRS-2679: Attempting to clean 'ora.edwp.db' on 'pnetn2'
CRS-2678: 'ora.edwp.db' on 'pnetn3' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5807: Agent failed to process the message

So i went on to Node 3 and tried to start Instance manually.
it ended up in

ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist

PNETN3@:/home/oracle :EDWP3 $sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Aug 2 11:56:40 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

---Tried to start it 

SQL> startup
ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist
SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

What could be the reason behind it, All resources are up except DB.

so i seen this file "lkinstEDWP3" which locked into $ORACLE_HOME/dbs.


drwxr-xr-x 74 oracle oinstall 4096 Jul 25 12:56 ..
-rw-r--r--  1 oracle oinstall 2851 Jul 25 12:57 init.ora
-rw-r-----  1 oracle oinstall 1536 Jul 26 06:37 orapwEDWP3
-rw-r-----  1 oracle oinstall   39 Jul 26 06:40 initEDWP3.ora
-rw-rw----  1 oracle asmadmin 1544 Jul 26 06:42 hc_EDWP3.dat
-rw-r-----  1 oracle asmadmin    0 Aug  2 11:57 lkinstEDWP3
drwxr-xr-x  2 oracle oinstall 4096 Aug  2 11:57 .
PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $rm -rf lkinstEDWP3
PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $ll
total 24
drwxr-xr-x 74 oracle oinstall 4096 Jul 25 12:56 ..
-rw-r--r--  1 oracle oinstall 2851 Jul 25 12:57 init.ora
-rw-r-----  1 oracle oinstall 1536 Jul 26 06:37 orapwEDWP3
-rw-r-----  1 oracle oinstall   39 Jul 26 06:40 initEDWP3.ora
-rw-rw----  1 oracle asmadmin 1544 Jul 26 06:42 hc_EDWP3.dat
drwxr-xr-x  2 oracle oinstall 4096 Aug  2 11:57 .
PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $cd ..
PNETN3@:/u01/app/oracle/11.2.0/db_1 :EDWP3 $ll

Again tried to start it.


PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $sqlplus /as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Aug 2 12:12:13 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area  342388736 bytes
Fixed Size                  2226352 bytes
Variable Size             306186064 bytes
Database Buffers           29360128 bytes
Redo Buffers                4616192 bytes
Database mounted.
Database opened.
SQL>
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0
With the Partitioning, Real Application Clusters, Automatic Storage Manageme
Data Mining and Real Application Testing options

No comments:

Post a Comment