RAC Off – Relinking the RAC Option Worked

Starting RAC database with srvctl resulted with error below.
ORA-00439: feature not enabled: Real Application Clusters

Rac On / Rac Off – Relinking the RAC Option (Doc ID 211177.1)

RAC OFF
1. Login as the Oracle software owner and shutdown all database instances on all nodes in the cluster.
2. cd $ORACLE_HOME/rdbms/lib
3. make -f ins_rdbms.mk rac_off
If this step did not fail with fatal errors, proceed to step 4.
4. make -f ins_rdbms.mk ioracle
Note:  If these steps fail with a Fatal Error, then you will need to open a Service Request and submit these errors to Oracle Support for analysis. 

================================================================================
### STOP Database 
================================================================================
[oracle@ol7-121-rac1 ~]$ srvctl stop database -d cdbrac
[oracle@ol7-121-rac1 ~]$ srvctl status database -d cdbrac -v
Instance cdbrac1 is not running on node ol7-121-rac1
Instance cdbrac2 is not running on node ol7-121-rac2
[oracle@ol7-121-rac1 ~]$

================================================================================
### make -f ins_rdbms.mk rac_off
================================================================================
[oracle@ol7-121-rac1 ~]$ cd $ORACLE_HOME/rdbms/lib
[oracle@ol7-121-rac1 lib]$ make -f ins_rdbms.mk rac_off
rm -f /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libskgxp12.so
cp /u01/app/oracle/product/12.1.0.2/dbhome_1/lib//libskgxpg.so /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libskgxp12.so
rm -f /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libskgxn2.so
cp /u01/app/oracle/product/12.1.0.2/dbhome_1/lib//libskgxns.so \
      /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libskgxn2.so
/usr/bin/ar d /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/libknlopt.a kcsm.o
/usr/bin/ar cr /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/libknlopt.a /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/ksnkcs.o
[oracle@ol7-121-rac1 lib]$

================================================================================
### make -f ins_rdbms.mk ioracle
================================================================================
[oracle@ol7-121-rac1 lib]$ make -f ins_rdbms.mk ioracle
chmod 755 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin
- Linking Oracle
rm -f /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/oracle
/u01/app/oracle/product/12.1.0.2/dbhome_1/bin/orald  -o /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/oracle -m64 -z noexecstack -Wl,--disable-new-dtags -L/u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/ -L/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/ -L/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/stubs/   -Wl,-E /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/opimai.o /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/ssoraed.o /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv12 -Wl,--no-whole-archive /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/nautab.o /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/naeet.o /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/naect.o /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/naedhs.o /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/config.o  -lserver12 -lodm12 -lcell12 -lnnet12 -lskgxp12 -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lclient12  -lvsn12 -lcommon12 -lgeneric12 -lknlopt `if /usr/bin/ar tv /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap12" ; fi` -lskjcx12 -lslax12 -lpls12  -lrt -lplp12 -lserver12 -lclient12  -lvsn12 -lcommon12 -lgeneric12 `if [ -f /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libavserver12.a ] ; then echo "-lavserver12" ; else echo "-lavstub12"; fi` `if [ -f /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libavclient12.a ] ; then echo "-lavclient12" ; fi` -lknlopt -lslax12 -lpls12  -lrt -lplp12 -ljavavm12 -lserver12  -lwwg  `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnro12 `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnnzst12 -lzt12 -lztkg12 -lmm -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lztkg12 `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnro12 `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/ldflags`    -lncrypt12 -lnsgr12 -lnzjs12 -ln12 -lnl12 -lnnzst12 -lzt12 -lztkg12   -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 `if /usr/bin/ar tv /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo12 -lserver12"; fi` -L/u01/app/oracle/product/12.1.0.2/dbhome_1/ctx/lib/ -lctxc12 -lctx12 -lzx12 -lgx12 -lctx12 -lzx12 -lgx12 -lordimt12 -lclsra12 -ldbcfg12 -lhasgen12 -lskgxn2 -lnnzst12 -lzt12 -lxml12 -locr12 -locrb12 -locrutl12 -lhasgen12 -lskgxn2 -lnnzst12 -lzt12 -lxml12  -lgeneric12 -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lsnls12 -lunls12  -lsnls12 -lnls12  -lcore12 -lsnls12 -lnls12 -lcore12 -lsnls12 -lnls12 -lxml12 -lcore12 -lunls12 -lsnls12 -lnls12 -lcore12 -lnls12 -lasmclnt12 -lcommon12 -lcore12  -laio -lons    `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/sysliblist` -Wl,-rpath,/u01/app/oracle/product/12.1.0.2/dbhome_1/lib -lm    `cat /u01/app/oracle/product/12.1.0.2/dbhome_1/lib/sysliblist` -ldl -lm   -L/u01/app/oracle/product/12.1.0.2/dbhome_1/lib
test ! -f /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle ||\
mv -f /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracleO
mv /u01/app/oracle/product/12.1.0.2/dbhome_1/rdbms/lib/oracle /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle
chmod 6751 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle
[oracle@ol7-121-rac1 lib]$

================================================================================
### srvctl config database -d cdbrac -v
================================================================================
[oracle@ol7-121-rac1 lib]$ srvctl config database -d cdbrac -v
Database unique name: cdbrac
Database name: cdbrac
Oracle home: /u01/app/oracle/product/12.1.0.2/dbhome_1
Oracle user: oracle
Spfile: +DATA/CDBRAC/PARAMETERFILE/spfile.277.1155927597
Password file: +DATA/CDBRAC/PASSWORD/pwdcdbrac.257.1155927083
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group:
Database instances: cdbrac1,cdbrac2
Configured nodes: ol7-121-rac1,ol7-121-rac2
Database is administrator managed
[oracle@ol7-121-rac1 lib]$

================================================================================
### srvctl start database -d cdbrac
================================================================================
[oracle@ol7-121-rac1 lib]$ srvctl start database -d cdbrac
PRCR-1079 : Failed to start resource ora.cdbrac.db

CRS-5017: The resource action "ora.cdbrac.db start" encountered the following error:
ORA-00439: feature not enabled: Real Application Clusters. 

For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/ol7-121-rac1/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.cdbrac.db' on 'ol7-121-rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.cdbrac.db' on that would satisfy its placement policy
[oracle@ol7-121-rac1 lib]$