Cluster it self is quite Intelligent. Once i started up the cluster.
I rebooted all node twice. It brought all Resources except Database.
ora.cvu
1 ONLINE ONLINE pnetn1
ora.edwp.db
1 OFFLINE OFFLINE
2 OFFLINE OFFLINE
3 OFFLINE OFFLINE
ora.oc4j
1 ONLINE ONLINE pnetn1
ora.pnetn1.vip
1 ONLINE ONLINE pnetn1
ora.pnetn2.vip
1 ONLINE ONLINE pnetn2
ora.pnetn3.vip
1 ONLINE ONLINE pnetn3
ora.scan1.vip
1 ONLINE ONLINE pnetn3
ora.scan2.vip
1 ONLINE ONLINE pnetn2
ora.scan3.vip
1 ONLINE ONLINE pnetn1
[root@PNETN1 ~]# crsctl start resource ora.edwp.db
CRS-2672: Attempting to start 'ora.edwp.db' on 'pnetn3'
CRS-2679: Attempting to clean 'ora.edwp.db' on 'pnetn3'
CRS-2679: Attempting to clean 'ora.edwp.db' on 'pnetn2'
CRS-2678: 'ora.edwp.db' on 'pnetn3' has experienced an unrecoverable failure
CRS-0267: Human intervention required to resume its availability.
CRS-5807: Agent failed to process the message
So i went on to Node 3 and tried to start Instance manually.
it ended up in
ORA-24324: service handle not initialized
ORA-01041: internal error. hostdef extension doesn't exist
PNETN3@:/home/oracle :EDWP3 $sqlplus / as sysdba SQL*Plus: Release 11.2.0.2.0 Production on Thu Aug 2 11:56:40 2012 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options ---Tried to start it SQL> startup ORA-24324: service handle not initialized ORA-01041: internal error. hostdef extension doesn't exist SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, Data Mining and Real Application Testing options
What could be the reason behind it, All resources are up except DB.
so i seen this file "lkinstEDWP3" which locked into $ORACLE_HOME/dbs.
drwxr-xr-x 74 oracle oinstall 4096 Jul 25 12:56 .. -rw-r--r-- 1 oracle oinstall 2851 Jul 25 12:57 init.ora -rw-r----- 1 oracle oinstall 1536 Jul 26 06:37 orapwEDWP3 -rw-r----- 1 oracle oinstall 39 Jul 26 06:40 initEDWP3.ora -rw-rw---- 1 oracle asmadmin 1544 Jul 26 06:42 hc_EDWP3.dat -rw-r----- 1 oracle asmadmin 0 Aug 2 11:57 lkinstEDWP3 drwxr-xr-x 2 oracle oinstall 4096 Aug 2 11:57 . PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $rm -rf lkinstEDWP3 PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $ll total 24 drwxr-xr-x 74 oracle oinstall 4096 Jul 25 12:56 .. -rw-r--r-- 1 oracle oinstall 2851 Jul 25 12:57 init.ora -rw-r----- 1 oracle oinstall 1536 Jul 26 06:37 orapwEDWP3 -rw-r----- 1 oracle oinstall 39 Jul 26 06:40 initEDWP3.ora -rw-rw---- 1 oracle asmadmin 1544 Jul 26 06:42 hc_EDWP3.dat drwxr-xr-x 2 oracle oinstall 4096 Aug 2 11:57 . PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $cd .. PNETN3@:/u01/app/oracle/11.2.0/db_1 :EDWP3 $ll
Again tried to start it.
PNETN3@:/u01/app/oracle/11.2.0/db_1/dbs :EDWP3 $sqlplus /as sysdba SQL*Plus: Release 11.2.0.2.0 Production on Thu Aug 2 12:12:13 2012 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to an idle instance. SQL> startup ORACLE instance started. Total System Global Area 342388736 bytes Fixed Size 2226352 bytes Variable Size 306186064 bytes Database Buffers 29360128 bytes Redo Buffers 4616192 bytes Database mounted. Database opened. SQL> SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 With the Partitioning, Real Application Clusters, Automatic Storage Manageme Data Mining and Real Application Testing options
