All About Oracle...: July 2012

Monday, July 30, 2012

Start Extract to Re-positioning / How to start extract to point-in-time.

Labels: alter extract begin, GOLDENGATE, OGG-01515 positioning to begin time, re position extract process, start extract, start replicat afterscn, start replicat atcsn

we mostly add extract as below

add extract finance ,tranlog begin now

This will make extract to start when you added extract.So the transactions which commited before we added extract will not capture.

Add the Extract group by issuing the ADD EXTRACT command with a timestamp as the
BEGIN option, or by using ADD EXTRACT with the SEQNO and RBA options. It is best to give
Extract a known start point at which to begin extracting data, rather than by using the
NOW argument.

ADD EXTRACT finance, TRANLOG, BEGIN 2011-01-01 12:00:00.000000

If data that Extract needs during processing was not retained, either in online or archived
logs, one of the following corrective actions might be required:
? Alter Extract to capture from a later point in time for which log data is available (and
accept possible data loss on the target).

ALTER EXTRACT
[,ADD EXTRACT {PARAMETERS} ]
[, UPGRADE INTEGRATED TRANLOG]
[, DOWNGRADE INTEGRATED TRANLOG]
[, THREAD ]
[, ETROLLOVER]

ALTER EXTRACT finance, TRANLOG, BEGIN 2011-01-01 12:00:00.000000

If you has to start EXTRACT at certain RBA , this could be achieved as below.

ALTER EXTRACT finance,EXTSEQNO <trailfile#> ,EXTRBA <rba>

You can also ALTER recovery Checkpoint. but this is not wise and Even Oracle does not support.

Alter the ioextseqno and ioextrba of recovery checkpoint back to its original recovery checkpoint position

Below seqno & RBA can be seen by INFO ,detail
In this example,

Alter extract , ioextseqno , ioextrba

To determine the oldest log that Extract might need at any
given point, issue the SEND EXTRACT command with the SHOWTRANS option.

You can also start Replicat at Point in time using , where CSN (commit sequence number ) is SCN (system change number) for Oracle database.

START REPLICAT
[SKIPTRANSACTION | ATCSN | AFTERCSN ]

if your replicat is abended and you ALTER it with BEGIN NOW, its status will change to STOPPED.

Friday, July 27, 2012

Configure Parallel DEGREE / INSTANCES of Table / PARALLEL execution running slow.

Labels: alter table parallel degree default instances, configure parallel degree of table, Exadata, parallel execution running slow, parallel running slow, Performance Tuning, table parallel degree

This is demo about Query which performing Parallel but takes lot of time.
Configuring Table parallel degree / instances degree incorrect can result adverse effect.

Platform is Exadata-X2.

Elapsed time has dropped from 2 minutes to 20 seconds.

SELECT count(1)
FROM DUMMY_TPLGY
WHERE (DUMMY_TPLGY.DVIC_TYPE = 'optical_node'
OR DUMMY_TPLGY.DVIC_TYPE = 'amplifier')
AND DUMMY_TPLGY.SNPSHT_DTM in
(SELECT SNPSHT_DTM
FROM DUMMY_TPLGY_CTRL
WHERE AVL_IND = 'Y'
AND SNPSHT_DTM >
(SELECT MAX (DAY_KEY)
FROM RCM_CTRL
WHERE TBL_NAME = 'DUMMY_DIM' AND AVL_IND = 'Y'));

Lets see What explain plan says .

set timing on 
set time on 
set long 999999
set lines 180
set pages 555

12:24:11 SQL> explain plan for
12:24:11   2  SELECT count(1)
12:24:11   3  FROM DUMMY_TPLGY
12:24:11   4  WHERE (DUMMY_TPLGY.DVIC_TYPE = 'optical_node'
12:24:11   5  OR DUMMY_TPLGY.DVIC_TYPE = 'amplifier')
12:24:11   6  AND DUMMY_TPLGY.SNPSHT_DTM in
12:24:11   7  (SELECT SNPSHT_DTM
12:24:11   8  FROM DUMMY_TPLGY_CTRL
12:24:11   9  WHERE AVL_IND = 'Y'
12:24:11  10  AND SNPSHT_DTM >
12:24:11  11  (SELECT MAX (DAY_KEY)
12:24:11  12  FROM RCM_CTRL
12:24:11  13  WHERE TBL_NAME = 'DUMMY_DIM' AND AVL_IND = 'Y'));


Explained.

Elapsed: 00:00:02.05

As you can see cost are too much high for full table scan.

12:24:15 SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 231818000

--------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name                | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                     |     1 |    26 | 50332   (1)| 00:10:04 |       |       |        |         |          |
|   1 |  SORT AGGREGATE               |                     |     1 |    26 |            |          |       |       |        |         |          |
|   2 |   NESTED LOOPS                |                     |  3434K|    85M| 49646   (1)| 00:09:56 |       |       |        |         |          |
|*  3 |    TABLE ACCESS BY INDEX ROWID| DUMMY_TPLGY_CTRL     |     3 |    30 |     1   (0)| 00:00:01 |       |       |        |         |          |
|*  4 |     INDEX RANGE SCAN          | XPK_DUMMY_TPLGY_CTRL |     1 |       |     1   (0)| 00:00:01 |       |       |        |         |          |
|   5 |      SORT AGGREGATE           |                     |     1 |    26 |            |          |       |       |        |         |          |
|   6 |       PX COORDINATOR          |                     |       |       |            |          |       |       |        |         |          |
|   7 |        PX SEND QC (RANDOM)    | :TQ10000            |     1 |    26 |            |          |       |       |  Q1,00 |    P->S | QC (RAND)|
|   8 |         SORT AGGREGATE        |                     |     1 |    26 |            |          |       |       |  Q1,00 |    PCWP |          |
|   9 |          PX BLOCK ITERATOR    |                     |   382 |  9932 |    24   (0)| 00:00:01 |       |       |  Q1,00 |    PCWC |          |
|* 10 |           TABLE ACCESS FULL   | RCM_CTRL            |   382 |  9932 |    24   (0)| 00:00:01 |       |       |  Q1,00 |    PCWP |          |
|  11 |    PARTITION LIST ITERATOR    |                     |  1174K|    17M| 16548   (1)| 00:03:19 |   KEY |   KEY |        |         |          |
|* 12 |     TABLE ACCESS FULL         | DUMMY_TPLGY          |  1174K|    17M| 16548   (1)| 00:03:19 |   KEY |   KEY |        |         |          |
--------------------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter("AVL_IND"='Y')
   4 - access("SNPSHT_DTM"> (SELECT MAX(SYS_OP_CSR(SYS_OP_MSR(MAX("DAY_KEY")),0)) FROM "RCM"."RCM_CTRL" "RCM_CTRL" WHERE
              "TBL_NAME"='DUMMY_DIM' AND "AVL_IND"='Y'))
  10 - filter("TBL_NAME"='DUMMY_DIM' AND "AVL_IND"='Y')
  12 - filter(("DUMMY_TPLGY"."DVIC_TYPE"='amplifier' OR "DUMMY_TPLGY"."DVIC_TYPE"='optical_node') AND
              "DUMMY_TPLGY"."SNPSHT_DTM"="SNPSHT_DTM")

Note
-----
   - dynamic sampling used for this statement (level=6)

33 rows selected.

Lets see actual execution Time .

Elapsed: 00:00:01.01
12:24:28 SQL>
12:24:28 SQL>
12:28:14 SQL> SELECT count(1)
12:28:15   2  FROM DUMMY_TPLGY
12:28:15   3  WHERE (DUMMY_TPLGY.DVIC_TYPE = 'optical_node'
12:28:15   4  OR DUMMY_TPLGY.DVIC_TYPE = 'amplifier')
12:28:15   5  AND DUMMY_TPLGY.SNPSHT_DTM in
12:28:15   6  (SELECT SNPSHT_DTM
12:28:15   7  FROM DUMMY_TPLGY_CTRL
12:28:15   8  WHERE AVL_IND = 'Y'
12:28:15   9  AND SNPSHT_DTM >
12:28:15  10  (SELECT MAX (DAY_KEY)
12:28:15  11  FROM RCM_CTRL
12:28:15  12  WHERE TBL_NAME = 'DUMMY_DIM' AND AVL_IND = 'Y'));

  COUNT(1)
----------
   3574152

Elapsed: 00:02:07.34
12:30:23 SQL>

Almost 2 minutes.

What are these tables and why it takes so long even if its running PARALLEL.
Culprit is Table DUMMY_TPLGY, which is going for FULL TABLE SCAN and is very big.

SQL> SELECT OWNER,TABLE_NAME,DEGREE,INSTANCES,PARTITIONED,NUM_ROWS FROM DBA_TABLES WHERE TABLE_NAME IN ('DUMMY_TPLGY','RCM_CTRL','DUMMY_TPLGY_CTRL');

OWNER      TABLE_NAME           DEGREE     INSTANCES  PARTITIONED       NUM_ROWS
---------- -------------------- ---------- ---------- --------------- ----------
RCM        RCM_CTRL                DEFAULT    DEFAULT NO                    4597
TNC        DUMMY_TPLGY                    1          1 YES              277101036
TNC        DUMMY_TPLGY_CTRL               1          1 NO                      59

----------Now Tables are in PARALLEL but degree is one Only.
So lets change it default and let oracle decide how to execute it.

SQL> ALTER TABLE TNC.DUMMY_TPLGY PARALLEL (DEGREE DEFAULT INSTANCES DEFAULT);

Table altered.

SQL> ALTER TABLE TNC.DUMMY_TPLGY_CTRL PARALLEL (DEGREE DEFAULT INSTANCES DEFAULT);

Table altered.

SQL> SELECT OWNER,TABLE_NAME,DEGREE,INSTANCES,PARTITIONED,NUM_ROWS FROM DBA_TABLES WHERE TABLE_NAME IN ('DUMMY_TPLGY','RCM_CTRL','DUMMY_TPLGY_CTRL');

OWNER      TABLE_NAME           DEGREE     INSTANCES  PARTITIONED       NUM_ROWS
---------- -------------------- ---------- ---------- --------------- ----------
RCM        RCM_CTRL                DEFAULT    DEFAULT NO                    4597
TNC        DUMMY_TPLGY              DEFAULT    DEFAULT YES              277101036
TNC        DUMMY_TPLGY_CTRL         DEFAULT    DEFAULT NO                      59

---New execution plan has low cost as compare to previous plan even Execution time also dropped drastically .

12:34:38 SQL> select * from table(dbms_xplan.display);

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Plan hash value: 3873636908

--------------------------------------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name                | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |    TQ  |IN-OUT| PQ Distrib|
-------------------------------------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                     |     1 |    26 |  1746   (1)| 00:00:21 |       |       |        |      |           |
|   1 |  SORT AGGREGATE                     |                     |     1 |    26 |            |          |       |       |        |      |           |
|   2 |   PX COORDINATOR                    |                     |       |       |            |          |       |       |        |      |           |
|   3 |    PX SEND QC (RANDOM)              | :TQ20001            |     1 |    26 |            |          |       |       |  Q2,01 | P->S | QC (RAND) |
|   4 |     SORT AGGREGATE                  |                     |     1 |    26 |            |          |       |       |  Q2,01 | PCWP |           |
|   5 |      NESTED LOOPS                   |                     |  3434K|    85M|  1723   (1)| 00:00:21 |       |       |  Q2,01 | PCWP |           |
|   6 |       BUFFER SORT                   |                     |       |       |            |          |       |       |  Q2,01 | PCWC |           |
|   7 |        PX RECEIVE                   |                     |       |       |            |          |       |       |  Q2,01 | PCWP |           |
|   8 |         PX SEND BROADCAST           | :TQ20000            |       |       |            |          |       |       |        | S->P | BROADCAST |
|*  9 |          TABLE ACCESS BY INDEX ROWID| DUMMY_TPLGY_CTRL     |     3 |    30 |     1   (0)| 00:00:01 |       |       |        |      |           |
|* 10 |           INDEX RANGE SCAN          | XPK_DUMMY_TPLGY_CTRL |     1 |       |     1   (0)| 00:00:01 |       |       |        |      |           |
|  11 |            SORT AGGREGATE           |                     |     1 |    26 |            |          |       |       |        |      |           |
|  12 |             PX COORDINATOR          |                     |       |       |            |          |       |       |        |      |           |
|  13 |              PX SEND QC (RANDOM)    | :TQ10000            |     1 |    26 |            |          |       |       |Q1,00   | P->S | QC (RAND) |
|  14 |               SORT AGGREGATE        |                     |     1 |    26 |            |          |       |       |Q1,00   | PCWP |           |
|  15 |                PX BLOCK ITERATOR    |                     |   382 |  9932 |    24   (0)| 00:00:01 |       |       |Q1,00   | PCWC |           |
|* 16 |                 TABLE ACCESS FULL   | RCM_CTRL            |   382 |  9932 |    24   (0)| 00:00:01 |       |       |Q1,00   | PCWP |           |
|  17 |       PX BLOCK ITERATOR             |                     |  1174K|    17M|   574   (1)| 00:00:07 |   KEY |   KEY |Q2,01   | PCWC |           |
|* 18 |        TABLE ACCESS FULL            | DUMMY_TPLGY          |  1174K|    17M|   574   (1)| 00:00:07 |   KEY |   KEY |Q2,01   | PCWP |           |
-------------------------------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   9 - filter("AVL_IND"='Y')
  10 - access("SNPSHT_DTM"> (SELECT MAX(SYS_OP_CSR(SYS_OP_MSR(MAX("DAY_KEY")),0)) FROM "RCM"."RCM_CTRL" "RCM_CTRL" WHERE "TBL_NAME"='DUMMY_DIM'
              AND "AVL_IND"='Y'))
  16 - filter("TBL_NAME"='DUMMY_DIM' AND "AVL_IND"='Y')
  18 - filter(("DUMMY_TPLGY"."DVIC_TYPE"='amplifier' OR "DUMMY_TPLGY"."DVIC_TYPE"='optical_node') AND "DUMMY_TPLGY"."SNPSHT_DTM"="SNPSHT_DTM")

---Now Lets run and Measure execution time.

SQL> ALTER SESSION ENABLE PARALLEL QUERY;

Session altered.

SQL> SET TIME ON
12:51:00 SQL> SET TIMING ON;
12:51:03 SQL>
12:51:15 SQL> SELECT count(1)
12:51:16   2  FROM DUMMY_TPLGY
12:51:16   3  WHERE (DUMMY_TPLGY.DVIC_TYPE = 'optical_node'
12:51:16   4  OR DUMMY_TPLGY.DVIC_TYPE = 'amplifier')
12:51:16   5  AND DUMMY_TPLGY.SNPSHT_DTM in
12:51:16   6  (SELECT SNPSHT_DTM
12:51:16   7  FROM DUMMY_TPLGY_CTRL
12:51:16   8  WHERE AVL_IND = 'Y'
12:51:16   9  AND SNPSHT_DTM >
12:51:16  10  (SELECT MAX (DAY_KEY)
12:51:16  11  FROM RCM_CTRL
12:51:16  12  WHERE TBL_NAME = 'DUMMY_DIM' AND AVL_IND = 'Y'));

  COUNT(1)
----------
   3574152

Elapsed: 00:00:20.89
12:51:37 SQL>

Here we go only 20 seconds from 2 minutes.

Thursday, July 26, 2012

Install 11.2.0.2 Grid Infrastructure as Role separation.

Labels: Install 11.2.0.2 GI, Install 11.2.0.2 Grid Infrastructure on Linux. Role separation for Grid infrastructure, Role separation

This is Demo of Installation of 11.2.0.2 Grid Infrastructure / database home & DB creation ,with Role separation on Linux x86_64.

Description	OS Group Name	OS Users Assigned to this Group	Oracle Privilege	Oracle Group Name
Oracle Inventory and Software Owner	oinstall	grid, oracle
Oracle Automatic Storage Management Group	asmadmin	grid	SYSASM	OSASM
ASM Database Administrator Group	asmdba	grid, oracle	SYSDBA for ASM	OSDBA for ASM
ASM Operator Group	asmoper	grid	SYSOPER for ASM	OSOPER for ASM
Database Administrator	dba	oracle	SYSDBA	OSDBA
Database Operator	oper	oracle	SYSOPER	OSOPER

I have skipped few steps and few pre-requirement during this exercise. ssh user equivalence is already configured.

if ASM disks are not visible during DBCA . you can change permission of $ORACLE_HOME/bin/oracle to oracle user and asmadmin group. Make sure you have nosuid in your /etc/fstab for Installer mount point.

/etc/fstab where i have created two seprate mount points.
/u00 for grid and /u01 for oracle. where i have dont have nosuid for any mountpoints.

PNETN1@:/home/grid :+ASM1 $cat /etc/fstab
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/u00              /u00                    ext3    defaults        1 2
LABEL=/u01              /u01                    ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   size=1700m      0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

Create group/users/binary home directories as below.

/usr/sbin/groupadd -g 1000 oinstall
/usr/sbin/groupadd -g 1020 asmadmin
/usr/sbin/groupadd -g 1021 asmdba
/usr/sbin/groupadd -g 1030 asmoper
/usr/sbin/groupadd -g 1031 dba
/usr/sbin/groupadd -g 1032 oper
useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper grid
useradd -u 1101 -g oinstall -G dba,asmdba oracle

---create GI/OH and set ownership/permission. 

--GI home
mkdir -p  /u00/app/11.2.0/grid
chmod 775 /u00/app/11.2.0/grid
chown -R grid:oinstall /u00
chmod -R 775 /u00

--OH home
mkdir -p  /u01/app/oracle/11.2.0/db_1
chmod 775 /u01/app/oracle/11.2.0/db_1
chown -R  oracle:oinstall /u01
chmod -R 775 /u01

Configure ASM DISKS.

Configure ASM driver/ MARK asmdisks/ scandisks

[root@PNETN3 ~]# /etc/init.d/oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]: grid
Default group to own the driver interface [asmdba]: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

MARK ASMDISKS ---Only on NODE1.

[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK1 /dev/sdb1
Marking disk "DISK1" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK2 /dev/sdc1
Marking disk "DISK2" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK3 /dev/sdd1
Marking disk "DISK3" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK4 /dev/sde1
Marking disk "DISK4" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK5 /dev/sdf1
Marking disk "DISK5" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]# /etc/init.d/oracleasm createdisk DISK6 /dev/sdg1
Marking disk "DISK6" as an ASM disk:                       [  OK  ]
[root@PNETN1 ~]#

SCAN/LIST ASMDISKS-----On each NODE.

[root@PNETN2 ~]# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@PNETN2 ~]# /etc/init.d/oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
DISK6
[root@PNETN2 ~]#

SET OWNERSHIP/PERMISSION ON ASMDISKS-----on each NODE.

you can either set permission on /etc/rc.d/rc.local OR /etc/udev/rules.d/99-sd.rules

I have used /etc/rc.d/rc.local which resulted in PRVF-5148 , which can be ignored.

[root@PNETN1 ~]# vi /etc/rc.d/rc.local
#!/bin/sh
touch /var/lock/subsys/local
chown grid:asmadmin /dev/sdb1
chown grid:asmadmin /dev/sdc1
chown grid:asmadmin /dev/sdd1
chown grid:asmadmin /dev/sde1
chown grid:asmadmin /dev/sdf1
chown grid:asmadmin /dev/sdg1
chown grid:asmadmin /dev/sdh1
chown grid:asmadmin /dev/sdi1
chown oracle:oinstall /dev/sdj1
chmod 660 /dev/sdb1
chmod 660 /dev/sdc1
chmod 660 /dev/sdd1
chmod 660 /dev/sde1
chmod 660 /dev/sdf1
chmod 660 /dev/sdg1
chmod 660 /dev/sdh1
chmod 660 /dev/sdi1
chmod 777 /dev/sdj1

*OR

You can also do it as below which will result in PRVF-5149.but cause will be unknown.

[root@PNETN3 rules.d]# vi /etc/udev/rules.d/99-sd.rules
KERNEL=="sdb[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdc[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdd[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sde[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdf[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdg[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdh[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdi[1]", OWNER="grid", GROUP="asmadmin", MODE="660"
KERNEL=="sdj[1]", OWNER="oracle", GROUP="oinstall", MODE="777"

---copy to each node. 
[root@PNETN3 rules.d]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules
[root@PNETN3 rules.d]# scp /etc/udev/rules.d/99-oracle-asmdevices.rules root@PNETN1:/etc/udev/rules.d/
The authenticity of host 'pnetn1 (192.168.100.126)' can't be established.
RSA key fingerprint is 27:ac:4b:9a:e3:d2:ae:6d:2b:71:99:8d:b9:c0:b1:a7.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'pnetn1,192.168.100.126' (RSA) to the list of known hosts.
root@pnetn1's password:
99-oracle-asmdevices.rules                                                                                               100%  551     0.5KB/s   00:00
[root@PNETN3 rules.d]# scp /etc/udev/rules.d/99-oracle-asmdevices.rules root@PNETN2:/etc/udev/rules.d/
root@pnetn2's password:
99-oracle-asmdevices.rules                                                                                               100%  551     0.5KB/s   00:00
[root@PNETN3 rules.d]#

---execute on each node. 

/sbin/partprobe /dev/sdb1
/sbin/partprobe /dev/sdc1
/sbin/partprobe /dev/sdd1
/sbin/partprobe /dev/sde1
/sbin/partprobe /dev/sdf1
/sbin/partprobe /dev/sdg1
/sbin/partprobe /dev/sdh1
/sbin/partprobe /dev/sdi1     
/sbin/partprobe /dev/sdj1

----execute on each node. 
/sbin/udevcontrol reload_rules
/sbin/start_udev

[root@PNETN1 rules.d]# /sbin/udevcontrol reload_rules
[root@PNETN1 rules.d]# /sbin/start_udev
Starting udev:                                             [  OK  ]

CHECK PERMISSION.

[root@PNETN1 rules.d]# ls -lart /dev/sd*
brw-r----- 1 root   disk     8, 144 Jul 25 07:43 /dev/sdj
brw-r----- 1 root   disk     8, 128 Jul 25 07:43 /dev/sdi
brw-r----- 1 root   disk     8, 112 Jul 25 07:43 /dev/sdh
brw-r----- 1 root   disk     8,  96 Jul 25 07:43 /dev/sdg
brw-r----- 1 root   disk     8,  80 Jul 25 07:43 /dev/sdf
brw-r----- 1 root   disk     8,  64 Jul 25 07:43 /dev/sde
brw-r----- 1 root   disk     8,  48 Jul 25 07:43 /dev/sdd
brw-r----- 1 root   disk     8,  32 Jul 25 07:43 /dev/sdc
brw-r----- 1 root   disk     8,  16 Jul 25 07:43 /dev/sdb
brw-r----- 1 root   disk     8,   5 Jul 25 07:43 /dev/sda5
brw-r----- 1 root   disk     8,   4 Jul 25 07:43 /dev/sda4
brw-r----- 1 root   disk     8,   0 Jul 25 07:43 /dev/sda
brw-r----- 1 root   disk     8,   3 Jul 25 07:44 /dev/sda3
brw-r----- 1 root   disk     8,   2 Jul 25 07:44 /dev/sda2
brw-r----- 1 root   disk     8,   1 Jul 25 07:44 /dev/sda1
brw-rw---- 1 grid   asmadmin 8,  17 Jul 25 07:44 /dev/sdb1
brw-rw---- 1 grid   asmadmin 8,  33 Jul 25 07:44 /dev/sdc1
brw-rw---- 1 grid   asmadmin 8,  49 Jul 25 07:44 /dev/sdd1
brw-rw---- 1 grid   asmadmin 8,  65 Jul 25 07:44 /dev/sde1
brw-rw---- 1 grid   asmadmin 8,  81 Jul 25 07:44 /dev/sdf1
brw-rw---- 1 grid   asmadmin 8,  97 Jul 25 07:44 /dev/sdg1
brw-rw---- 1 grid   asmadmin 8, 113 Jul 25 07:44 /dev/sdh1
brw-rw---- 1 grid   asmadmin 8, 129 Jul 25 07:44 /dev/sdi1
brwxrwxrwx 1 oracle oinstall 8, 145 Jul 25 07:44 /dev/sdj1

----runcluvfy.sh

PNETN1@:/u01/patch/p10098816_112020_Linux-x86-64_grid : $./runcluvfy.sh stage -pre crsinst -n PNETN1,PNETN2,PNETN3 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "PNETN1"
  Destination Node                      Reachable?
  ------------------------------------  ------------------------
  PNETN1                                yes
  PNETN2                                yes
  PNETN3                                yes
Result: Node reachability check passed from node "PNETN1"


Checking user equivalence...

Check: User equivalence for user "grid"
  Node Name                             Comment
  ------------------------------------  ------------------------
  PNETN3                                passed
  PNETN2                                passed
  PNETN1                                passed
Result: User equivalence check passed for user "grid"

Checking node connectivity...

Checking hosts config file...
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  PNETN3        passed
  PNETN2        passed
  PNETN1        passed

Verification of the hosts config file successful


Interface information for node "PNETN3"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.100.128 192.168.100.0   0.0.0.0         UNKNOWN         00:0C:29:F5:6F:F0 1500
 eth1   192.168.152.128 192.168.152.0   0.0.0.0         UNKNOWN         00:0C:29:F5:6F:FA 1500


Interface information for node "PNETN2"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.100.127 192.168.100.0   0.0.0.0         UNKNOWN         00:0C:29:BB:4A:C2 1500
 eth1   192.168.152.127 192.168.152.0   0.0.0.0         UNKNOWN         00:0C:29:BB:4A:CC 1500


Interface information for node "PNETN1"
 Name   IP Address      Subnet          Gateway         Def. Gateway    HW Address        MTU
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0   192.168.100.126 192.168.100.0   0.0.0.0         192.168.100.1   00:0C:29:01:0F:8B 1500
 eth1   192.168.152.126 192.168.152.0   0.0.0.0         192.168.100.1   00:0C:29:01:0F:95 1500


Check: Node connectivity of subnet "192.168.100.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  PNETN3[192.168.100.128]         PNETN2[192.168.100.127]         yes
  PNETN3[192.168.100.128]         PNETN1[192.168.100.126]         yes
  PNETN2[192.168.100.127]         PNETN1[192.168.100.126]         yes
Result: Node connectivity passed for subnet "192.168.100.0" with node(s) PNETN3,PNETN2,PNETN1


Check: TCP connectivity of subnet "192.168.100.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  PNETN1:192.168.100.126          PNETN3:192.168.100.128          passed
  PNETN1:192.168.100.126          PNETN2:192.168.100.127          passed
Result: TCP connectivity check passed for subnet "192.168.100.0"


Check: Node connectivity of subnet "192.168.152.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  PNETN3[192.168.152.128]         PNETN2[192.168.152.127]         yes
  PNETN3[192.168.152.128]         PNETN1[192.168.152.126]         yes
  PNETN2[192.168.152.127]         PNETN1[192.168.152.126]         yes
Result: Node connectivity passed for subnet "192.168.152.0" with node(s) PNETN3,PNETN2,PNETN1


Check: TCP connectivity of subnet "192.168.152.0"
  Source                          Destination                     Connected?
  ------------------------------  ------------------------------  ----------------
  PNETN1:192.168.152.126          PNETN3:192.168.152.128          passed
  PNETN1:192.168.152.126          PNETN2:192.168.152.127          passed
Result: TCP connectivity check passed for subnet "192.168.152.0"


Interfaces found on subnet "192.168.152.0" that are likely candidates for a private interconnect are:
PNETN3 eth1:192.168.152.128
PNETN2 eth1:192.168.152.127
PNETN1 eth1:192.168.152.126

WARNING:
Could not find a suitable set of interfaces for VIPs

Result: Node connectivity check passed


Checking ASMLib configuration.
  Node Name                             Comment
  ------------------------------------  ------------------------
  PNETN3                                passed
  PNETN2                                passed
  PNETN1                                passed
Result: Check for ASMLib configuration passed.

Check: Total memory
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        1.6625GB (1743280.0KB)    1.5GB (1572864.0KB)       passed
  PNETN2        1.6625GB (1743280.0KB)    1.5GB (1572864.0KB)       passed
  PNETN1        1.6625GB (1743280.0KB)    1.5GB (1572864.0KB)       passed
Result: Total memory check passed

Check: Available memory
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        1.5201GB (1593932.0KB)    50MB (51200.0KB)          passed
  PNETN2        1.5202GB (1594000.0KB)    50MB (51200.0KB)          passed
  PNETN1        1.4477GB (1518024.0KB)    50MB (51200.0KB)          passed
Result: Available memory check passed

Check: Swap space
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        3.8437GB (4030456.0KB)    2.4938GB (2614920.0KB)    passed
  PNETN2        3.8437GB (4030456.0KB)    2.4938GB (2614920.0KB)    passed
  PNETN1        3.8437GB (4030456.0KB)    2.4938GB (2614920.0KB)    passed
Result: Swap space check passed

Check: Free disk space for "PNETN3:/tmp"
  Path              Node Name     Mount point   Available     Required      Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              PNETN3        /             2.7852GB      1GB           passed
Result: Free disk space check passed for "PNETN3:/tmp"

Check: Free disk space for "PNETN2:/tmp"
  Path              Node Name     Mount point   Available     Required      Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              PNETN2        /             2.7852GB      1GB           passed
Result: Free disk space check passed for "PNETN2:/tmp"

Check: Free disk space for "PNETN1:/tmp"
  Path              Node Name     Mount point   Available     Required      Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  /tmp              PNETN1        /             2.5402GB      1GB           passed
Result: Free disk space check passed for "PNETN1:/tmp"

Check: User existence for "grid"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  PNETN3        exists(1100)              passed
  PNETN2        exists(1100)              passed
  PNETN1        exists(1100)              passed

Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  PNETN3        exists                    passed
  PNETN2        exists                    passed
  PNETN1        exists                    passed
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  PNETN3        exists                    passed
  PNETN2        exists                    passed
  PNETN1        exists                    passed
Result: Group existence check passed for "dba"

Check: Membership of user "grid" in group "oinstall" [as Primary]
  Node Name         User Exists   Group Exists  User in Group  Primary       Comment
  ----------------  ------------  ------------  ------------  ------------  ------------
  PNETN3            yes           yes           yes           yes           passed
  PNETN2            yes           yes           yes           yes           passed
  PNETN1            yes           yes           yes           yes           passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Membership of user "grid" in group "dba"
  Node Name         User Exists   Group Exists  User in Group  Comment
  ----------------  ------------  ------------  ------------  ----------------
  PNETN3            yes           yes           no            failed
  PNETN2            yes           yes           no            failed
  PNETN1            yes           yes           no            failed
Result: Membership check for user "grid" in group "dba" failed

Check: Run level
  Node Name     run level                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        5                         3,5                       passed
  PNETN2        5                         3,5                       passed
  PNETN1        5                         3,5                       passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Comment
  ----------------  ------------  ------------  ------------  ----------------
  PNETN3            hard          131072        65536         passed
  PNETN2            hard          131072        65536         passed
  PNETN1            hard          131072        65536         passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
  Node Name         Type          Available     Required      Comment
  ----------------  ------------  ------------  ------------  ----------------
  PNETN3            soft          131072        1024          passed
  PNETN2            soft          131072        1024          passed
  PNETN1            soft          131072        1024          passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
  Node Name         Type          Available     Required      Comment
  ----------------  ------------  ------------  ------------  ----------------
  PNETN3            hard          131072        16384         passed
  PNETN2            hard          131072        16384         passed
  PNETN1            hard          131072        16384         passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
  Node Name         Type          Available     Required      Comment
  ----------------  ------------  ------------  ------------  ----------------
  PNETN3            soft          131072        2047          passed
  PNETN2            soft          131072        2047          passed
  PNETN1            soft          131072        2047          passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        x86_64                    x86_64                    passed
  PNETN2        x86_64                    x86_64                    passed
  PNETN1        x86_64                    x86_64                    passed
Result: System architecture check passed

Check: Kernel version
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        2.6.18-164.el5xen         2.6.18                    passed
  PNETN2        2.6.18-164.el5xen         2.6.18                    passed
  PNETN1        2.6.18-164.el5xen         2.6.18                    passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        250                       250                       passed
  PNETN2        250                       250                       passed
  PNETN1        250                       250                       passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        32000                     32000                     passed
  PNETN2        32000                     32000                     passed
  PNETN1        32000                     32000                     passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        100                       100                       passed
  PNETN2        100                       100                       passed
  PNETN1        100                       100                       passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        142                       128                       passed
  PNETN2        142                       128                       passed
  PNETN1        142                       128                       passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        4398046511104             892559360                 passed
  PNETN2        4398046511104             892559360                 passed
  PNETN1        4398046511104             892559360                 passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        4096                      4096                      passed
  PNETN2        4096                      4096                      passed
  PNETN1        4096                      4096                      passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        1073741824                2097152                   passed
  PNETN2        1073741824                2097152                   passed
  PNETN1        1073741824                2097152                   passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        6815744                   6815744                   passed
  PNETN2        6815744                   6815744                   passed
  PNETN1        6815744                   6815744                   passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        between 9000 & 65500      between 9000 & 65500      passed
  PNETN2        between 9000 & 65500      between 9000 & 65500      passed
  PNETN1        between 9000 & 65500      between 9000 & 65500      passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        262144                    262144                    passed
  PNETN2        262144                    262144                    passed
  PNETN1        262144                    262144                    passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        4194304                   4194304                   passed
  PNETN2        4194304                   4194304                   passed
  PNETN1        4194304                   4194304                   passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        262144                    262144                    passed
  PNETN2        262144                    262144                    passed
  PNETN1        262144                    262144                    passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        1048576                   1048576                   passed
  PNETN2        1048576                   1048576                   passed
  PNETN1        1048576                   1048576                   passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
  Node Name     Configured                Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        3145728                   1048576                   passed
  PNETN2        3145728                   1048576                   passed
  PNETN1        3145728                   1048576                   passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.81( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        make-3.81-3.el5           make-3.81( x86_64)        passed
  PNETN2        make-3.81-3.el5           make-3.81( x86_64)        passed
  PNETN1        make-3.81-3.el5           make-3.81( x86_64)        passed
Result: Package existence check passed for "make-3.81( x86_64)"

Check: Package existence for "binutils-2.17.50.0.6( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6( x86_64)  passed
  PNETN2        binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6( x86_64)  passed
  PNETN1        binutils-2.17.50.0.6-12.el5  binutils-2.17.50.0.6( x86_64)  passed
Result: Package existence check passed for "binutils-2.17.50.0.6( x86_64)"

Check: Package existence for "gcc-4.1.2 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        gcc-4.1.2-46.el5 (x86_64)  gcc-4.1.2 (x86_64)( x86_64)  passed
  PNETN2        gcc-4.1.2-46.el5 (x86_64)  gcc-4.1.2 (x86_64)( x86_64)  passed
  PNETN1        gcc-4.1.2-46.el5 (x86_64)  gcc-4.1.2 (x86_64)( x86_64)  passed
Result: Package existence check passed for "gcc-4.1.2 (x86_64)( x86_64)"

Check: Package existence for "libaio-0.3.106 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        libaio-0.3.106-3.2 (x86_64)  libaio-0.3.106 (x86_64)( x86_64)  passed
  PNETN2        libaio-0.3.106-3.2 (x86_64)  libaio-0.3.106 (x86_64)( x86_64)  passed
  PNETN1        libaio-0.3.106-3.2 (x86_64)  libaio-0.3.106 (x86_64)( x86_64)  passed
Result: Package existence check passed for "libaio-0.3.106 (x86_64)( x86_64)"

Check: Package existence for "glibc-2.5-24 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        glibc-2.5-42 (x86_64)     glibc-2.5-24 (x86_64)( x86_64)  passed
  PNETN2        glibc-2.5-42 (x86_64)     glibc-2.5-24 (x86_64)( x86_64)  passed
  PNETN1        glibc-2.5-42 (x86_64)     glibc-2.5-24 (x86_64)( x86_64)  passed
Result: Package existence check passed for "glibc-2.5-24 (x86_64)( x86_64)"

Check: Package existence for "compat-libstdc++-33-3.2.3 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        compat-libstdc++-33-3.2.3-61 (x86_64)  compat-libstdc++-33-3.2.3 (x86_64)( x86_64)  passed
  PNETN2        compat-libstdc++-33-3.2.3-61 (x86_64)  compat-libstdc++-33-3.2.3 (x86_64)( x86_64)  passed
  PNETN1        compat-libstdc++-33-3.2.3-61 (x86_64)  compat-libstdc++-33-3.2.3 (x86_64)( x86_64)  passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)( x86_64)"

Check: Package existence for "elfutils-libelf-0.125 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        elfutils-libelf-0.137-3.el5 (x86_64)  elfutils-libelf-0.125 (x86_64)( x86_64)  passed
  PNETN2        elfutils-libelf-0.137-3.el5 (x86_64)  elfutils-libelf-0.125 (x86_64)( x86_64)  passed
  PNETN1        elfutils-libelf-0.137-3.el5 (x86_64)  elfutils-libelf-0.125 (x86_64)( x86_64)  passed
Result: Package existence check passed for "elfutils-libelf-0.125 (x86_64)( x86_64)"

Check: Package existence for "elfutils-libelf-devel-0.125( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125( x86_64)  passed
  PNETN2        elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125( x86_64)  passed
  PNETN1        elfutils-libelf-devel-0.137-3.el5  elfutils-libelf-devel-0.125( x86_64)  passed
Result: Package existence check passed for "elfutils-libelf-devel-0.125( x86_64)"

Check: Package existence for "glibc-common-2.5( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        glibc-common-2.5-42       glibc-common-2.5( x86_64)  passed
  PNETN2        glibc-common-2.5-42       glibc-common-2.5( x86_64)  passed
  PNETN1        glibc-common-2.5-42       glibc-common-2.5( x86_64)  passed
Result: Package existence check passed for "glibc-common-2.5( x86_64)"

Check: Package existence for "glibc-devel-2.5 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        glibc-devel-2.5-42 (x86_64)  glibc-devel-2.5 (x86_64)( x86_64)  passed
  PNETN2        glibc-devel-2.5-42 (x86_64)  glibc-devel-2.5 (x86_64)( x86_64)  passed
  PNETN1        glibc-devel-2.5-42 (x86_64)  glibc-devel-2.5 (x86_64)( x86_64)  passed
Result: Package existence check passed for "glibc-devel-2.5 (x86_64)( x86_64)"

Check: Package existence for "glibc-headers-2.5( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        glibc-headers-2.5-42      glibc-headers-2.5( x86_64)  passed
  PNETN2        glibc-headers-2.5-42      glibc-headers-2.5( x86_64)  passed
  PNETN1        glibc-headers-2.5-42      glibc-headers-2.5( x86_64)  passed
Result: Package existence check passed for "glibc-headers-2.5( x86_64)"

Check: Package existence for "gcc-c++-4.1.2 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        gcc-c++-4.1.2-46.el5 (x86_64)  gcc-c++-4.1.2 (x86_64)( x86_64)  passed
  PNETN2        gcc-c++-4.1.2-46.el5 (x86_64)  gcc-c++-4.1.2 (x86_64)( x86_64)  passed
  PNETN1        gcc-c++-4.1.2-46.el5 (x86_64)  gcc-c++-4.1.2 (x86_64)( x86_64)  passed
Result: Package existence check passed for "gcc-c++-4.1.2 (x86_64)( x86_64)"

Check: Package existence for "libaio-devel-0.3.106 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        libaio-devel-0.3.106-3.2 (x86_64)  libaio-devel-0.3.106 (x86_64)( x86_64)  passed
  PNETN2        libaio-devel-0.3.106-3.2 (x86_64)  libaio-devel-0.3.106 (x86_64)( x86_64)  passed
  PNETN1        libaio-devel-0.3.106-3.2 (x86_64)  libaio-devel-0.3.106 (x86_64)( x86_64)  passed
Result: Package existence check passed for "libaio-devel-0.3.106 (x86_64)( x86_64)"

Check: Package existence for "libgcc-4.1.2 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        libgcc-4.1.2-46.el5 (x86_64)  libgcc-4.1.2 (x86_64)( x86_64)  passed
  PNETN2        libgcc-4.1.2-46.el5 (x86_64)  libgcc-4.1.2 (x86_64)( x86_64)  passed
  PNETN1        libgcc-4.1.2-46.el5 (x86_64)  libgcc-4.1.2 (x86_64)( x86_64)  passed
Result: Package existence check passed for "libgcc-4.1.2 (x86_64)( x86_64)"

Check: Package existence for "libstdc++-4.1.2 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        libstdc++-4.1.2-46.el5 (x86_64)  libstdc++-4.1.2 (x86_64)( x86_64)  passed
  PNETN2        libstdc++-4.1.2-46.el5 (x86_64)  libstdc++-4.1.2 (x86_64)( x86_64)  passed
  PNETN1        libstdc++-4.1.2-46.el5 (x86_64)  libstdc++-4.1.2 (x86_64)( x86_64)  passed
Result: Package existence check passed for "libstdc++-4.1.2 (x86_64)( x86_64)"

Check: Package existence for "libstdc++-devel-4.1.2 (x86_64)( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        libstdc++-devel-4.1.2-46.el5 (x86_64)  libstdc++-devel-4.1.2 (x86_64)( x86_64)  passed
  PNETN2        libstdc++-devel-4.1.2-46.el5 (x86_64)  libstdc++-devel-4.1.2 (x86_64)( x86_64)  passed
  PNETN1        libstdc++-devel-4.1.2-46.el5 (x86_64)  libstdc++-devel-4.1.2 (x86_64)( x86_64)  passed
Result: Package existence check passed for "libstdc++-devel-4.1.2 (x86_64)( x86_64)"

Check: Package existence for "sysstat-7.0.2( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        sysstat-7.0.2-3.el5       sysstat-7.0.2( x86_64)    passed
  PNETN2        sysstat-7.0.2-3.el5       sysstat-7.0.2( x86_64)    passed
  PNETN1        sysstat-7.0.2-3.el5       sysstat-7.0.2( x86_64)    passed
Result: Package existence check passed for "sysstat-7.0.2( x86_64)"

Check: Package existence for "ksh-20060214( x86_64)"
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        ksh-20080202-14.el5       ksh-20060214( x86_64)     passed
  PNETN2        ksh-20080202-14.el5       ksh-20060214( x86_64)     passed
  PNETN1        ksh-20080202-14.el5       ksh-20060214( x86_64)     passed
Result: Package existence check passed for "ksh-20060214( x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes
No NTP Daemons or Services were found to be running

Result: Clock synchronization check using Network Time Protocol(NTP) passed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
  Node Name     Status                    Comment
  ------------  ------------------------  ------------------------
  PNETN3        does not exist            passed
  PNETN2        does not exist            passed
  PNETN1        does not exist            passed
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
  Node Name     Available                 Required                  Comment
  ------------  ------------------------  ------------------------  ----------
  PNETN3        0022                      0022                      passed
  PNETN2        0022                      0022                      passed
  PNETN1        unknown                   0022                      failed
Result: Default user file creation mask check failed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "localdomain.com" as found on node "PNETN3"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
  Node Name                             Status
  ------------------------------------  ------------------------
  PNETN3                                failed
  PNETN2                                failed
  PNETN1                                failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: PNETN3,PNETN2,PNETN1

File "/etc/resolv.conf" is not consistent across nodes

Check: Time zone consistency
Result: Time zone consistency check passed

Starting check for Huge Pages Existence ...

Check for Huge Pages Existence passed

Starting check for Hardware Clock synchronization at shutdown ...

Check for Hardware Clock synchronization at shutdown passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

===============================
I have skipped DNS response time/ user mask in my pre-requirement.
Check DNS response time using
time nslookup
===============================

---INSTALL GI.

Below error is result you can get if you set disk permission using /etc/rc.d/rc.local

At end this will fail at Oracle cluster verification Utility. This is due to umask were not set properly on PNETN1 node.

INFO: Default user file creation mask check failed

INFO: Check failed on nodes:

INFO: PNETN1

INFO: Checking cluster integrity...

INFO: Checks did not pass for the following node(s):

INFO: PNETN1

INFO:

WARNING:

INFO: Completed Plugin named: Oracle Cluster Verification Utility

But over-all Installation will be successfully.

---INSTALL ORACLE

---RUN ROOT.SH on ALL NODES.

[root@PNETN1 grid]# ./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u00/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u00/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding daemon to inittab
ACFS-9200: Supported
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9309: ADVM/ACFS installation correctness verified.
CRS-2672: Attempting to start 'ora.mdnsd' on 'pnetn1'
CRS-2676: Start of 'ora.mdnsd' on 'pnetn1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'pnetn1'
CRS-2676: Start of 'ora.gpnpd' on 'pnetn1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'pnetn1'
CRS-2672: Attempting to start 'ora.gipcd' on 'pnetn1'
CRS-2676: Start of 'ora.gipcd' on 'pnetn1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'pnetn1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'pnetn1'
CRS-2672: Attempting to start 'ora.diskmon' on 'pnetn1'
CRS-2676: Start of 'ora.diskmon' on 'pnetn1' succeeded
CRS-2676: Start of 'ora.cssd' on 'pnetn1' succeeded

ASM created and started successfully.

Disk Group CRS created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 3e59c3d836244f36bf834c0c3c9f5484.
Successfully replaced voting disk group with +CRS.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   3e59c3d836244f36bf834c0c3c9f5484 (/dev/sdb1) [CRS]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'pnetn1'
CRS-2676: Start of 'ora.asm' on 'pnetn1' succeeded
CRS-2672: Attempting to start 'ora.CRS.dg' on 'pnetn1'
CRS-2676: Start of 'ora.CRS.dg' on 'pnetn1' succeeded
ACFS-9200: Supported
ACFS-9200: Supported
CRS-2672: Attempting to start 'ora.registry.acfs' on 'pnetn1'
CRS-2676: Start of 'ora.registry.acfs' on 'pnetn1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

CHECK ALL RESOURCES.

---------------------------------------
[root@PNETN1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
ora.LISTENER.lsnr
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
ora.asm
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
ora.gsd
               OFFLINE OFFLINE      pnetn1
               OFFLINE OFFLINE      pnetn2
               OFFLINE OFFLINE      pnetn3
ora.net1.network
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
ora.ons
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
ora.registry.acfs
               ONLINE  ONLINE       pnetn1
               ONLINE  ONLINE       pnetn2
               ONLINE  ONLINE       pnetn3
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       pnetn3
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       pnetn1
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       pnetn2
ora.cvu
      1        ONLINE  ONLINE       pnetn2
ora.oc4j
      1        ONLINE  ONLINE       pnetn2
ora.pnetn1.vip
      1        ONLINE  ONLINE       pnetn1
ora.pnetn2.vip
      1        ONLINE  ONLINE       pnetn2
ora.pnetn3.vip
      1        ONLINE  ONLINE       pnetn3
ora.scan1.vip
      1        ONLINE  ONLINE       pnetn3
ora.scan2.vip
      1        ONLINE  ONLINE       pnetn1
ora.scan3.vip
      1        ONLINE  ONLINE       pnetn2
[root@PNETN1 ~]#

RUN DATABASE root.sh

[root@PNETN3 db_1]# ./root.sh
Running Oracle 11g root script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/11.2.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

CREATE DISKGROUP For DB.

PNETN1@:/home/grid : $. oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u00/app/grid
PNETN1@:/home/grid :+ASM1 $sqlplus / as sysasm
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production

SQL> CREATE DISKGROUP EDW_DATA EXTERNAL REDUNDANCY DISK '/dev/sdc1','/dev/sdd1','/dev/sde1';

Diskgroup created.

SQL>

----MOUNT DISKGROUP ON OTHER NODES. 

PNETN2@:/home/grid :+ASM2 $sqlplus / as sysasm

SQL> ALTER DISKGROUP EDW_DATA MOUNT;

Diskgroup altered.

---------------RUN DBCA... SELECT EDW_DATA DISKGROUP for DB CREATION.

---------------LOGIN AS ORACLE and TEST READ/WRITE.

PNETN1@:/home/oracle :EDWP1 $. oraenv
ORACLE_SID = [EDWP1] ? EDWP1
The Oracle base has been set to /u01/app/oracle

PNETN1@:/home/oracle :EDWP1 $sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Jul 26 06:47:10 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> CREATE TABLESPACE TEST DATAFILE '+EDW_DATA' size 50m;

Tablespace created.

SQL> DROP TABLESPACE TEST INCLUDING CONTENTS;

Tablespace dropped.