After applying Oracle GoldenGate V12.2.0.1.170919 for Oracle Database 12c OPTIMIZER Patch for Bug# 26849949, starting GoldenGate extract failed with OGG-00303: Unable to open credential store. Error code 43,490.
Here is what the report looks like.
$ head -50 E_LAX6.rpt
***********************************************************************
Oracle GoldenGate Capture for Oracle
Version 12.2.0.1.170919 OGGCORE_12.2.0.1.0OGGBP_PLATFORMS_171030.0908_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Oct 30 2017 20:59:41
Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.
Starting at 2018-10-25 15:08:33
***********************************************************************
Operating System Version:
Linux
Version #2 SMP Wed Jul 11 12:11:36 PDT 2018, Release 4.1.12-94.8.5.el7uek.x86_64
Node: localhost
Machine: x86_64
soft limit hard limit
Address Space Size : unlimited unlimited
Heap Size : unlimited unlimited
File Size : unlimited unlimited
CPU Time : unlimited unlimited
Process id: 23154
Description:
***********************************************************************
** Running with the following parameters **
***********************************************************************
2018-10-25 15:08:33 INFO OGG-03059 Operating system character set identified as UTF-8.
2018-10-25 15:08:33 INFO OGG-02695 ANSI SQL parameter syntax is used for parameter parsing.
EXTRACT e_lax
USERIDALIAS gguser
Source Context :
SourceModule : [er.init]
SourceID : [/scratch/aime/adestore/views/aime_adc4150431/oggcore/OpenSys/src/app/er/init.cpp]
SourceFunction : [get_infile_params]
SourceLine : [5554]
ThreadBacktrace : [11] elements
: [/u01/gg/12.2.0/libgglog.so(CMessageContext::AddThreadContext()+0x1b) [0x7f714024709b]]
: [/u01/gg/12.2.0/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x135) [0x7f7140241165]]
: [/u01/gg/12.2.0/libgglog.so(_MSG_ERR_STARTUP_PARAMERROR_ERRORTEXT(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x30) [0x7f71402308d0]]
: [/u01/gg/12.2.0/extract(get_infile_params(time_elt_def*, time_elt_def*, char**, ggs::gglib::ggdatasource::DataSourceParams&, ggs::Heartbeat::MapGeneratorParams&)+0x5da1) [0x5c4c91]]
: [/u01/gg/12.2.0/extract() [0x5f036a]]
: [/u01/gg/12.2.0/extract(ggs::gglib::MultiThreading::MainThread::ExecMain()+0x60) [0x6cea60]]
: [/u01/gg/12.2.0/extract(ggs::gglib::MultiThreading::Thread::RunThread(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x14d) [0x6cfcdd]]
Since I did not have the password for database gguser, I modified the password from the database and recreated credentialstore using ggsci.
info credentialstore
delete credentialstore
create wallet.
add credentialstore
alter credentialstore add user gguser alias gguser.
alter credentialstore add user GGUSER alias GGUSER
info credentialstore
Reason check credentialstore before deleting is to determine aliases.
GGSCI 1> info credentialstore Reading from ./dircrd/: Default domain: OracleGoldenGate Alias: gguser Userid: gguser Alias: GGUSER Userid: GGUSER
Here are the steps from ggserr.log.
2018-10-25 15:44:00 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): create wallet. 2018-10-25 15:44:00 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Created wallet at location 'dirwlt'. 2018-10-25 15:44:00 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Opened wallet at location 'dirwlt'. 2018-10-25 15:44:10 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): add credentialstore. 2018-10-25 15:44:10 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store created in ./dircrd/. 2018-10-25 15:44:22 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): alter credentialstore add user gguser alias gguser. 2018-10-25 15:44:31 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store in ./dircrd/ altered. 2018-10-25 15:44:55 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): alter credentialstore add user GGUSER alias GGUSER. 2018-10-25 15:45:06 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Credential store in ./dircrd/ altered. 2018-10-25 15:45:15 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): info credentialstore. 2018-10-25 15:45:15 INFO OGG-02096 Oracle GoldenGate Command Interpreter for Oracle: Reading from ./dircrd/:.
Here are the permissions for the files.
$ chmod 775 -R dirwlt/ dircrd/ $ ls -l dirwlt/ dircrd/ dircrd/: total 4 -rwxrwxr-x 1 gguser oinstall 701 Oct 25 15:45 cwallet.sso dirwlt/: total 4 -rwxrwxr-x 1 gguser oinstall 290 Oct 25 15:44 cwallet.sso
Starting extract failed again!
2018-10-25 15:45:30 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): dblogin useridalias gguser.
2018-10-25 15:45:35 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): start e*.
2018-10-25 15:45:35 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 10.80.27.191:39060 (START EXTRACT E_LAX ).
2018-10-25 15:45:35 INFO OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #5).
2018-10-25 15:45:35 INFO OGG-00975 Oracle GoldenGate Manager for Oracle, mgr.prm: EXTRACT E_LAX starting.
2018-10-25 15:45:35 INFO OGG-00992 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX starting.
2018-10-25 15:45:35 INFO OGG-03059 Oracle GoldenGate Capture for Oracle, e_lax.prm: Operating system character set identified as UTF-8.
2018-10-25 15:45:35 INFO OGG-02695 Oracle GoldenGate Capture for Oracle, e_lax.prm: ANSI SQL parameter syntax is used for parameter parsing.
2018-10-25 15:45:35 ERROR OGG-00303 Oracle GoldenGate Capture for Oracle, e_lax.prm: Unable to open credential store. Error code 43,490.
2018-10-25 15:45:35 ERROR OGG-01668 Oracle GoldenGate Capture for Oracle, e_lax.prm: PROCESS ABENDING.
Set environment variables for ORACLE_HOME and ORACLE_SID for extract, and restart extract solved the issue.
FYI – environment variables already exist from OS.
2018-10-25 15:45:38 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (gguser): info all.
2018-10-25 15:52:44 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (ggsuser): start e*.
2018-10-25 15:52:44 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 10.80.27.191:39169 (START EXTRACT E_LAX ).
2018-10-25 15:52:44 INFO OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #5).
2018-10-25 15:52:44 INFO OGG-00975 Oracle GoldenGate Manager for Oracle, mgr.prm: EXTRACT E_LAX starting.
2018-10-25 15:52:44 INFO OGG-00992 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX starting.
2018-10-25 15:52:44 INFO OGG-03059 Oracle GoldenGate Capture for Oracle, e_lax.prm: Operating system character set identified as UTF-8.
2018-10-25 15:52:44 INFO OGG-02695 Oracle GoldenGate Capture for Oracle, e_lax.prm: ANSI SQL parameter syntax is used for parameter parsing.
2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_2.
2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_SID=sourcedb.
2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_2.
2018-10-25 15:52:44 INFO OGG-02095 Oracle GoldenGate Capture for Oracle, e_lax.prm: Successfully set environment variable ORACLE_SID=sourcedb.
2018-10-25 15:52:56 INFO OGG-00993 Oracle GoldenGate Capture for Oracle, e_lax.prm: EXTRACT E_LAX started.
Currently, I don’t know if patching caused the issue or if it is a pre-existing condition.
What’s interesting is the same issue happened for another system where environment variables for ORACLE_HOME and ORACLE_SID were not set for extract.
Instead of restarting just the extract, all the processes were stopped, and restarted.
It would be a good idea to stop and start Goldengate processes before patching in order to identify any pre-existing conditions.
Thinking Out Loud 