What does root.sh perform on RAC Cluster Installation

I thought, it may be useful for others to understand what actions are being performed while running root.sh on RAC cluster installation. I spend some time today and update this topic on my blog for every one benefit :-)

At the end of the RAC cluster installation,  we run root.sh and orainstRoot.sh file.  

You will see this screen right before the installation completion.  Please note, i captured this screen on 11.2.0 installation. But oracle12c. you may not see this screen and GUI tool will run the root,.sh. 

What does orainstRoot.sh perform?

1. It creates the inventory pointer file (/etc/oraInst.loc)
2. It Changes groupname of the oraInventory directory to oinstall group.

The log is from first node.

[root@RAC1 grid]# sh /grid/app/oraInventory/orainstRoot.sh Changing permissions of /grid/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /grid/app/oraInventory to oinstall. The execution of the script is complete. [root@RAC1 grid]#

 What does Root.sh perform?  

1. It will set the environment variable for ORACLE_HOME and ORACLE_OWNER
2. Copy dbhome,oraenv,coraenv to /usr/local/bin
3. Creates the /etc/oratab file
4. Verify the super user privileges for the user.
5. Creating the trace directory
6. Creates  OCR keys for the 'root' user
7. Adding clusterware Daemon information  to inittab file
8. Starts Oracle High Availability service Daemon(OHASD) service
9. Start the clusterware daemons(gipcd,mdnsd,gpnpd,cssmonitor,cssd,diskmon,ctssd etc)
10. Creates and configures an ASM instance and ASM disk group will be created.
11. Creates required ASM disk groups, if ASM is being used to put OCR and voting files.
12. It Configures the OCR and voting disks (only on the first node)
13. Starts up the Cluster Ready Service Daemon (CRSD) process
14. updating the profile by adding voting disk
15. It Puts the voting disk on the ASM disk group ,if ASM type is selected.
16. It Displays voting disk details
17. Attempting to stop and start the cluster resources(crsd,asm, ctssd,cssmonitor, cssd,gpnpd, gipcd etc)
18. updates the OLR files
19. Installs the cvuqdisk-1.0.7-1 package
20. Configure Oracle Grid Infrastructure for Cluster
21. Updating inventory properties for clusterware
22. Starts the Oracle universal Installer
23. Check the swapspace and make sure we have enough space
24. Completes with the UpdateNodeList success operation. 

The log is from first node.

[root@RAC1 grid]# pwd /grid/app/11.2.0/grid [root@RAC1 grid]# sh root.sh Running Oracle 11g root.sh script... The following environment variables are set as:     ORACLE_OWNER= oracle     ORACLE_HOME=  /grid/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]:    Copying dbhome to /usr/local/bin ...    Copying oraenv to /usr/local/bin ...    Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2015-10-04 17:08:18: Parsing the host name 2015-10-04 17:08:18: Checking for super user privileges 2015-10-04 17:08:18: User has super user privileges Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful.   root wallet   root wallet cert   root cert export   peer wallet   profile reader wallet   pa wallet   peer wallet keys   pa wallet keys   peer cert request   pa cert request   peer cert   pa cert   peer root cert TP   profile reader root cert TP   pa root cert TP   peer pa cert TP   pa peer cert TP   profile reader pa cert TP   profile reader peer cert TP   peer user cert   pa user cert Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64 CRS-2672: Attempting to start 'ora.gipcd' on 'rac1' CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1' CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1' CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1' CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rac1' CRS-2672: Attempting to start 'ora.diskmon' on 'rac1' CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'rac1' CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded ASM created and started successfully. DiskGroup DATA created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-2672: Attempting to start 'ora.crsd' on 'rac1' CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded CRS-4256: Updating the profile Successful addition of voting disk 5df60344ce984fd3bf6a36825778db82. Successfully replaced voting disk group with +DATA. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ##  STATE    File Universal Id                File Name Disk group --  -----    -----------------                --------- ---------  1. ONLINE   5df60344ce984fd3bf6a36825778db82 (ORCL:DISK1) [DATA] Located 1 voting disk(s). CRS-2673: Attempting to stop 'ora.crsd' on 'rac1' CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1' CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1' CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac1' CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1' CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1' CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1' CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1' CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'rac1' CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1' CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1' CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rac1' CRS-2672: Attempting to start 'ora.diskmon' on 'rac1' CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'rac1' CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rac1' CRS-2676: Start of 'ora.asm' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'rac1' CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'rac1' CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rac1' CRS-2676: Start of 'ora.asm' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1' CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded rac1     2015/10/04 17:13:29     /grid/app/11.2.0/grid/cdata/rac1/backup_20151004_171329.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 3998 MB    Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /grid/app/oraInventory 'UpdateNodeList' was successful. [root@RAC1 grid]#

The log for root.sh from second node.

[root@RAC2 grid]# sh root.sh Running Oracle 11g root.sh script... The following environment variables are set as:     ORACLE_OWNER= oracle     ORACLE_HOME=  /grid/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n) [n]: Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2015-10-04 19:46:01: Parsing the host name 2015-10-04 19:46:01: Checking for super user privileges 2015-10-04 19:46:01: User has super user privileges Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_                                                                                                                                                             params LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64 CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on                                                                                                                                                      node rac1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the cluster CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2' CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'rac2' CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2' CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2' CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'rac2' CRS-2672: Attempting to start 'ora.diskmon' on 'rac2' CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'rac2' CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.asm' on 'rac2' CRS-2676: Start of 'ora.asm' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'rac2' CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'rac2' CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded rac2     2015/10/04 19:48:17     /grid/app/11.2.0/grid/cdata/rac2/backup_20151004_194817                                                                                                                                                     .olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /grid/app/oraInventory [root@RAC2 grid]#

Hope this helps! Please let me know if there is any correction or suggestion on this topic.

Enjoy reading!

CRS-4124: Oracle High Availability Services startup failed

While  installing Oracle11g(11.2.0.1) clusterware on RedHat Linux 6, i got the below error at time of running root.sh file. It is two node RAC and node names are RACNODE1 and RACNODE2.

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

Here is the complete log.

[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 11:17:19: Parsing the host name
2015-09-28 11:17:19: Checking for super user privileges
2015-09-28 11:17:19: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_           params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding daemon to inittab
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
ohasd failed to start: Inappropriate ioctl for device
ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

After googling, i found that 11.2.0.1 does not support RHL 6.0.

Here is workaround to fix the issue.

Step 1

Login to racnode1 and open the file  $GRID_HOME/crs/install/s_crsconfig_lib.pm
Add the following command before # Start OHASD

my $UPSTART_OHASD_SERVICE = "oracle-ohasd";
my $INITCTL = "/sbin/initctl";

($status, @output) = system_cmd_capture ("$INITCTL start $UPSTART_OHASD_SERVICE");
if (0 != $status)
{
error ("Failed to start $UPSTART_OHASD_SERVICE, error: $!");
return $FAILED;
}

Please see the screenshot.

Step 2

Create a file /etc/init/oracle-ohasd.conf with below content.

# Oracle OHASD startup
start on runlevel [35]
stop on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1

Step 3

Rollback the root.sh changes on racnode1. 

cd $GRID_HOME/crs/install

[root@racnode1 install]# ./roothas.pl -deconfig -force -verbose
2015-09-28 13:03:50: Checking for super user privileges
2015-09-28 13:03:50: User has super user privileges
2015-09-28 13:03:50: Parsing the host name
Using configuration parameter file: ./crsconfig_params
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Delete failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: /lib/acfstoolsdriver.sh: No such file or directory
/grid/app/11.2.0/grid/bin/acfsdriverstate: line 51: exec: /lib/acfstoolsdriver.sh: cannot execute: No such file or directory

Successfully deconfigured Oracle Restart stack

Step 4

Rerun the root.sh on racnode1

[root@racnode1 grid]# sh root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying dbhome to /usr/local/bin ...
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2015-09-28 13:04:35: Parsing the host name
2015-09-28 13:04:35: Checking for super user privileges
2015-09-28 13:04:35: User has super user privileges
Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded

ASM created and started successfully.

DiskGroup DATA created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 2c2253ff0c544fdabf043b4752630d9e.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   2c2253ff0c544fdabf043b4752630d9e (ORCL:DISK1) [DATA]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'racnode1'
CRS-2677: Stop of 'ora.crsd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racnode1'
CRS-2677: Stop of 'ora.asm' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode1'
CRS-2677: Stop of 'ora.ctssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racnode1'
CRS-2677: Stop of 'ora.cssd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode1'
CRS-2677: Stop of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode1'
CRS-2677: Stop of 'ora.gipcd' on 'racnode1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode1'
CRS-2677: Stop of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode1'
CRS-2676: Start of 'ora.mdnsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'racnode1'
CRS-2676: Start of 'ora.gipcd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode1'
CRS-2676: Start of 'ora.gpnpd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode1'
CRS-2676: Start of 'ora.cssdmonitor' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'racnode1'
CRS-2672: Attempting to start 'ora.diskmon' on 'racnode1'
CRS-2676: Start of 'ora.diskmon' on 'racnode1' succeeded
CRS-2676: Start of 'ora.cssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'racnode1'
CRS-2676: Start of 'ora.ctssd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'racnode1'
CRS-2676: Start of 'ora.crsd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'racnode1'
CRS-2676: Start of 'ora.evmd' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-2676: Start of 'ora.asm' on 'racnode1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'racnode1'
CRS-2676: Start of 'ora.DATA.dg' on 'racnode1' succeeded

racnode1     2015/09/28 13:08:57     /grid/app/11.2.0/grid/cdata/racnode1/backup_20150928_130857.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /grid/app/oraInventory
'UpdateNodeList' was successful.

[root@racnode1 grid]#

Now racnode1 is successfully started OHASD service. Let us move on to racnode2 and run the root.sh

Step 5

[root@racnode2 oraInventory]#  sh orainstRoot.sh

Creating the Oracle inventory pointer file (/etc/oraInst.loc)

Changing permissions of /grid/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /grid/app/oraInventory to oinstall.

The execution of the script is complete.

[root@racnode2 oraInventory]#  cd /grid/app/11.2.0/grid

[root@racnode2 grid]# sh root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

   Copying dbhome to /usr/local/bin ...

   Copying oraenv to /usr/local/bin ...

   Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2015-09-28 11:30:55: Parsing the host name

2015-09-28 11:30:55: Checking for super user privileges

2015-09-28 11:30:55: User has super user privileges

Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_                                   params

Creating trace directory

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

  root wallet

  root wallet cert

  root cert export

  peer wallet

  profile reader wallet

  pa wallet

  peer wallet keys

  pa wallet keys

  peer cert request

  pa cert request

  peer cert

  pa cert

  peer root cert TP

  profile reader root cert TP

  pa root cert TP

  peer pa cert TP

  pa peer cert TP

  profile reader pa cert TP

  profile reader peer cert TP

  peer user cert

  pa user cert

Adding daemon to inittab

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

ohasd failed to start: Inappropriate ioctl for device

ohasd failed to start at /grid/app/11.2.0/grid/crs/install/rootcrs.pl line 443.

[root@racnode2 grid]#

[root@racnode2 grid]#

Opps.. RACNODE2 also has similar issue as we had in RACNODE1.

Step 6

Repeat the step1, step2 & step3 on RACNODE2. 

Run root.sh on racnode2.

The OHASD service again failed  and found the below info on log

2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 338904345, wrtcnt, 4437, LATS 10349194, lastSeqNo 4437, uniqueness 1443489200, timestamp 1443490040/4294892050

2015-09-28 18:23:38.934: [    CSSD][1588156160]clssnmvDHBValidateNCopy: node 2, , has a disk HB, but no network HB, DHB has rcfg 338904344, wrtcnt, 733, LATS 10349194, lastSeqNo 733, uniqueness 0, timestamp 1443478983/10313614

2015-09-28 18:23:38.954: [    CLSF][1588156160]Closing handle:0x1a6ed70

2

Realized that firewall issue on private network which is blocking between racnode1 and racnode. 

Found the above info in $GRID_HOME/log/racnode2/cssd/ocssd.log

Execute the below command on  both node to fix the firewall issue on private network

service iptables stop

service ip6tables stop

chkconfig iptables off

chkconfig ip6tables off

Step 7

Run root.sh on racnode2 as below

[root@racnode2 ~]# pwd

/root

[root@racnode2 ~]# cd /grid/app/11.2.0/grid

[root@racnode2 grid]# pwd

/grid/app/11.2.0/grid

[root@racnode2 grid]# cd crs

[root@racnode2 crs]# cd install

[root@racnode2 install]# ./roothas.pl -deconfig -force -verbose

2015-09-28 18:41:44: Checking for super user privileges

2015-09-28 18:41:44: User has super user privileges

2015-09-28 18:41:44: Parsing the host name

Using configuration parameter file: ./crsconfig_params

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Stop failed, or completed with errors.

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4000: Command Delete failed, or completed with errors.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racnode2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'racnode2'

CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'racnode2'

CRS-2673: Attempting to stop 'ora.ctssd' on 'racnode2'

CRS-2677: Stop of 'ora.cssdmonitor' on 'racnode2' succeeded

cd ..

cd CRS-2677: Stop of 'ora.ctssd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'racnode2'

..

CRS-2677: Stop of 'ora.mdnsd' on 'racnode2' succeeded

CRS-2677: Stop of 'ora.cssd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'racnode2'

CRS-2673: Attempting to stop 'ora.diskmon' on 'racnode2'

pwd

CRS-2677: Stop of 'ora.gpnpd' on 'racnode2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'racnode2'

CRS-2677: Stop of 'ora.gipcd' on 'racnode2' succeeded

CRS-2677: Stop of 'ora.diskmon' on 'racnode2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racnode2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

ACFS-9201: Not Supported

Successfully deconfigured Oracle Restart stack

[root@racnode2 install]# cd ..

[root@racnode2 crs]# cd ..

[root@racnode2 grid]# pwd

/grid/app/11.2.0/grid

[root@racnode2 grid]# sh root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:

    ORACLE_OWNER= oracle

    ORACLE_HOME=  /grid/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)

[n]:

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root.sh script.

Now product-specific root actions will be performed.

2015-09-28 18:42:17: Parsing the host name

2015-09-28 18:42:17: Checking for super user privileges

2015-09-28 18:42:17: User has super user privileges

Using configuration parameter file: /grid/app/11.2.0/grid/crs/install/crsconfig_params

LOCAL ADD MODE

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

Adding daemon to inittab

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

ADVM/ACFS is not supported on oraclelinux-release-6Server-0.0.5.x86_64

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node racnode1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

CRS-2672: Attempting to start 'ora.mdnsd' on 'racnode2'

CRS-2676: Start of 'ora.mdnsd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.gipcd' on 'racnode2'

CRS-2676: Start of 'ora.gipcd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'racnode2'

CRS-2676: Start of 'ora.gpnpd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racnode2'

CRS-2676: Start of 'ora.cssdmonitor' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'racnode2'

CRS-2672: Attempting to start 'ora.diskmon' on 'racnode2'

CRS-2676: Start of 'ora.diskmon' on 'racnode2' succeeded

CRS-2676: Start of 'ora.cssd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.ctssd' on 'racnode2'

CRS-2676: Start of 'ora.ctssd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.asm' on 'racnode2'

CRS-2676: Start of 'ora.asm' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.crsd' on 'racnode2'

CRS-2676: Start of 'ora.crsd' on 'racnode2' succeeded

CRS-2672: Attempting to start 'ora.evmd' on 'racnode2'

CRS-2676: Start of 'ora.evmd' on 'racnode2' succeeded

racnode2     2015/09/28 18:44:22     /grid/app/11.2.0/grid/cdata/racnode2/backup_20150928_184422.olr

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Updating inventory properties for clusterware

Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 3999 MB    Passed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /grid/app/oraInventory

[root@racnode2 grid]# ps -ef|grep pmon

oracle   10130     1  0 18:43 ?        00:00:00 asm_pmon_+ASM2

root     10689  8438  0 18:44 pts/0    00:00:00 grep pmon

[root@racnode2 grid]#

Step 8

Validate both node and make sure all looks good.

Tablespace missing in DBA_FREE_SPACE

Today i came across one of the interesting issue in my development database. Developer was loading the data and he got the ORA-01653: unable to extend table XXX.YYYYY by 128 in tablespace ZZZZ  error and data load got failed.  Immediately, he called me and reported this error.  I went and checked the DBA_FREE_SPACE and not able to find the tablespace... 

hmmm... interesting!

I found that tablespace will disappear in DBA_FREE_SPACE for the following scenario.

1. The tablespace has no free space. 
2. Tablespace is offline
3. Tablespace is temporary tablespace

Let us test the first case.  

 Create the tablespace.

SQL> create tablespace test

datafile  '/u02/oradata/test.dbf' size 10M;    2

Tablespace created.

Verify if the tablespace available in DBA_FREE_SPACE

SQL>

select tablespace_name from dba_free_space

where tablespace_name='TEST';  SQL>   2

TABLESPACE_NAME

------------------------------

TEST

Load the data  until it runs out of space

SQL>

create table gt_objects as select *

from dba_objects;

SQL>   2

Table created.

SQL>

SQL> alter table gt_objects move tablespace test;

Table altered.

SQL>

begin

for i in 1..1000000 loop

insert into gt_objects select * from dba_objects;

commit;

end  loop;

end;

/

SQL>   2    3    4    5    6    7  begin

*

ERROR at line 1:

ORA-01653: unable to extend table SYS.GT_OBJECTS by 128 in tablespace TEST

ORA-06512: at line 3

 Check if the tablespace disappeard in DBA_FREE_SPACE

SQL> SQL> select tablespace_name from dba_free_space

  2  where tablespace_name='TEST';

no rows selected

SQL>

Drop the table and see if it appears in DBA_FREE_SPACE

SQL> drop table gt_objects;

Table dropped.

SQL> select tablespace_name from dba_free_space

  2   where tablespace_name='TEST';

TABLESPACE_NAME

------------------------------

TEST

SQL>

Drop the tablespace

SQL> drop tablespace test including contents and datafiles;

Tablespace dropped.

SQL>

Conclusion :  The minimum space required for the tablespace to be reported in the DBA_FREE_SPACE is that of the extent size of the tablespace. We have to be careful if we use DBA_FREE_SPACE for tablespace monitoring query.