Thinking Out Loud

May 22, 2022

Systemd Persistent Logging

Filed under: 19c,linux,OEL7,oracle — mdinh @ 3:05 pm

Remembrance from an old friend: Curiosity killed the cat, but satisfaction brought it back.

Finally found how to keep systemd logging persistent.

==================================================
### Number of reboots:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# last reboot
reboot   system boot  5.4.17-2136.307. Sat May 21 17:53 - 18:01  (00:07)
reboot   system boot  5.4.17-2136.307. Sat May 21 17:48 - 17:51  (00:03)
reboot   system boot  5.4.17-2136.307. Sat May 21 11:01 - 17:48  (06:46)
reboot   system boot  5.4.17-2136.307. Fri May 20 23:12 - 11:01  (11:48)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:54 - 23:11  (00:17)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:49 - 22:54  (00:04)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:21 - 22:49  (00:27)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:04 - 22:21  (00:16)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:35 - 22:03  (22:28)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:19 - 23:34  (00:15)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:07 - 23:18  (00:11)
reboot   system boot  5.4.17-2136.307. Thu May 19 22:54 - 23:05  (00:11)
reboot   system boot  5.4.17-2136.307. Thu May 19 22:47 - 22:53  (00:06)
reboot   system boot  5.4.17-2136.307. Thu May 19 21:08 - 22:45  (01:37)
reboot   system boot  5.4.17-2136.307. Thu May 19 20:40 - 21:07  (00:27)
reboot   system boot  5.4.17-2136.307. Thu May 19 07:28 - 10:59  (03:31)
reboot   system boot  5.4.17-2136.307. Thu May 19 07:22 - 07:27  (00:05)
reboot   system boot  5.4.17-2036.100. Wed May 18 22:24 - 07:22  (08:57)

wtmp begins Wed May 18 22:24:31 2022
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Only 1 reboot recorded from journalctl:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# journalctl --list-boots
 0 016de73a7e81411bbd7c0b7657863852 Sat 2022-05-21 17:53:18 -04—Sat 2022-05-21 18:01:01 -04
[root@oracle-goldengate-1910-vagrant ~]#

[root@oracle-goldengate-1910-vagrant ~]# journalctl | head -1
-- Logs begin at Sat 2022-05-21 17:53:18 -04, end at Sat 2022-05-21 18:01:01 -04. --
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Enable persistent systemd journal log:
==================================================

Enable persistent storage for the systemd journal log
https://gist.github.com/JPvRiel/b7c185833da32631fa6ce65b40836887

[root@oracle-goldengate-1910-vagrant ~]# mkdir /var/log/journal
[root@oracle-goldengate-1910-vagrant ~]# systemd-tmpfiles --create --prefix /var/log/journal
[root@oracle-goldengate-1910-vagrant ~]# echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf
[root@oracle-goldengate-1910-vagrant ~]# grep -i SystemMaxUse /etc/systemd/journald.conf
#SystemMaxUse=
SystemMaxUse=100M
[root@oracle-goldengate-1910-vagrant ~]# systemctl restart systemd-journald.service

==================================================
### Moment of truth:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# last reboot
reboot   system boot  5.4.17-2136.307. Sat May 21 18:31 - 18:33  (00:01)
reboot   system boot  5.4.17-2136.307. Sat May 21 18:12 - 18:29  (00:16)
reboot   system boot  5.4.17-2136.307. Sat May 21 17:53 - 18:09  (00:15)

reboot   system boot  5.4.17-2136.307. Sat May 21 17:48 - 17:51  (00:03)
reboot   system boot  5.4.17-2136.307. Sat May 21 11:01 - 17:48  (06:46)
reboot   system boot  5.4.17-2136.307. Fri May 20 23:12 - 11:01  (11:48)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:54 - 23:11  (00:17)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:49 - 22:54  (00:04)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:21 - 22:49  (00:27)
reboot   system boot  5.4.17-2136.307. Fri May 20 22:04 - 22:21  (00:16)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:35 - 22:03  (22:28)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:19 - 23:34  (00:15)
reboot   system boot  5.4.17-2136.307. Thu May 19 23:07 - 23:18  (00:11)
reboot   system boot  5.4.17-2136.307. Thu May 19 22:54 - 23:05  (00:11)
reboot   system boot  5.4.17-2136.307. Thu May 19 22:47 - 22:53  (00:06)
reboot   system boot  5.4.17-2136.307. Thu May 19 21:08 - 22:45  (01:37)
reboot   system boot  5.4.17-2136.307. Thu May 19 20:40 - 21:07  (00:27)
reboot   system boot  5.4.17-2136.307. Thu May 19 07:28 - 10:59  (03:31)
reboot   system boot  5.4.17-2136.307. Thu May 19 07:22 - 07:27  (00:05)
reboot   system boot  5.4.17-2036.100. Wed May 18 22:24 - 07:22  (08:57)

wtmp begins Wed May 18 22:24:31 2022
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### More than 1 reboot recorded from journalctl:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# journalctl --list-boots|sort
 0 beedb88ca40c4c6187ae9d5c7ff1e96e Sat 2022-05-21 18:31:53 -04—Sat 2022-05-21 23:30:01 -04
-1 e12a9e02fd374350a5fcd3d4360b09fc Sat 2022-05-21 18:12:48 -04—Sat 2022-05-21 18:29:34 -04
-2 016de73a7e81411bbd7c0b7657863852 Sat 2022-05-21 17:53:18 -04—Sat 2022-05-21 18:09:16 -04
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Review GoldengGate Start and Stop:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# last reboot|head -5|sort -n
reboot   system boot  5.4.17-2136.307. Sat May 21 11:01 - 17:48  (06:46)
reboot   system boot  5.4.17-2136.307. Sat May 21 17:48 - 17:51  (00:03)

reboot   system boot  5.4.17-2136.307. Sat May 21 17:53 - 18:09  (00:15)

reboot   system boot  5.4.17-2136.307. Sat May 21 18:12 - 18:29  (00:16)
reboot   system boot  5.4.17-2136.307. Sat May 21 18:31 - 19:11  (00:39)
[root@oracle-goldengate-1910-vagrant ~]#

[root@oracle-goldengate-1910-vagrant ~]# journalctl --no-pager|grep "Manager st"
May 21 17:53:44 oracle-goldengate-1910-vagrant start_goldengate.sh[3115]: Manager started.

May 21 18:09:04 oracle-goldengate-1910-vagrant stop_goldengate.sh[6177]: Manager stopped.
May 21 18:13:15 oracle-goldengate-1910-vagrant start_goldengate.sh[3383]: Manager started.

May 21 18:29:21 oracle-goldengate-1910-vagrant stop_goldengate.sh[6290]: Manager stopped.
May 21 18:32:17 oracle-goldengate-1910-vagrant start_goldengate.sh[4134]: Manager started.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================

reboot system boot  5.4.17-2136.307. Sat May 21 17:48 - 17:51  (00:03) 
journalctl May 21 17:53:44 oracle-goldengate-1910-vagrant start_goldengate.sh[3115]: Manager started.

reboot system boot  5.4.17-2136.307. Sat May 21 17:53 - 18:09  (00:15)
journalctl May 21 18:09:04 oracle-goldengate-1910-vagrant stop_goldengate.sh[6177]: Manager stopped.
journalctl May 21 18:13:15 oracle-goldengate-1910-vagrant start_goldengate.sh[3383]: Manager started.

reboot system boot  5.4.17-2136.307. Sat May 21 18:12 - 18:29  (00:16)
journalctl May 21 18:13:15 oracle-goldengate-1910-vagrant start_goldengate.sh[3383]: Manager started.
journalctl May 21 18:29:21 oracle-goldengate-1910-vagrant stop_goldengate.sh[6290]: Manager stopped.

reboot system boot  5.4.17-2136.307. Sat May 21 18:31 - 10:57  (16:25)
journalctl May 21 18:32:17 oracle-goldengate-1910-vagrant start_goldengate.sh[4134]: Manager started.

May 21, 2022

Systemd Database GoldenGate Service Start Stop

Filed under: 19c,GoldenGate,linux,OEL7,oracle — mdinh @ 6:13 pm

A complicated solution for a simple process.

If you can’t beat them, join them.

==================================================
### Restart vagrant box:
==================================================

resetlogs@ghost MINGW64 /e/oracle/vagrant-projects/OracleGoldenGate/19.3.0 (main)
$ vagrant halt

resetlogs@ghost MINGW64 /e/oracle/vagrant-projects/OracleGoldenGate/19.3.0 (main)
$ vagrant up

resetlogs@ghost MINGW64 /e/oracle/vagrant-projects/OracleGoldenGate/19.3.0 (main)
$ vagrant ssh

==================================================
### System info and last reboot:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# cat /etc/system-release
Oracle Linux Server release 7.9
[root@oracle-goldengate-1910-vagrant ~]#

[root@oracle-goldengate-1910-vagrant ~]# last reboot|head -1
reboot   system boot  5.4.17-2136.307. Sat May 21 11:01 - 11:02  (00:00)
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Verify goldengate.service started on reboot:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# systemctl status goldengate.service
● goldengate.service - Oracle GoldenGate Manager
   Loaded: loaded (/etc/systemd/system/goldengate.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2022-05-21 11:02:14 -04; 1min 9s ago
  Process: 4485 ExecStart=/home/oracle/scripts/start_goldengate.sh (code=exited, status=0/SUCCESS)
 Main PID: 4504 (mgr)
   CGroup: /system.slice/goldengate.service
           ├─4504 ./mgr PARAMFILE /u01/ogg/dirprm/mgr.prm REPORTFILE /u01/ogg/dirrpt/MGR.rpt PROCESSID MGR
           └─4522 /u01/ogg/extract PARAMFILE /u01/ogg/dirprm/ext.prm REPORTFILE /u01/ogg/dirrpt/EXT.rpt PROCESSID EXT

May 21 11:02:06 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Manager started.
May 21 11:02:06 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 3> START ER *
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Sending START request to MANAGER ...
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: EXTRACT EXT starting
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 11:02:13 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: MANAGER     RUNNING
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: EXTRACT     RUNNING     EXT         00:01:25      00:00:00
May 21 11:02:14 oracle-goldengate-1910-vagrant systemd[1]: Started Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Check logs for goldengate.service:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# journalctl -u goldengate.service
-- Logs begin at Sat 2022-05-21 11:01:41 -04, end at Sat 2022-05-21 11:03:24 -04. --
May 21 11:02:04 oracle-goldengate-1910-vagrant systemd[1]: Starting Oracle GoldenGate Manager...
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: The Oracle base has been set to /opt/oracle
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Oracle GoldenGate Command Interpreter for Oracle
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Operating system character set identified as UTF-8.
May 21 11:02:04 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
May 21 11:02:05 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 1>
May 21 11:02:05 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 2> START MGR
May 21 11:02:06 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Manager started.
May 21 11:02:06 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 3> START ER *
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Sending START request to MANAGER ...
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: EXTRACT EXT starting
May 21 11:02:07 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 11:02:13 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: MANAGER     RUNNING
May 21 11:02:14 oracle-goldengate-1910-vagrant start_goldengate.sh[4485]: EXTRACT     RUNNING     EXT         00:01:25      00:00:00
May 21 11:02:14 oracle-goldengate-1910-vagrant systemd[1]: Started Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### There is no log from journalctl before reboot:
==================================================

[root@oracle-goldengate-1910-vagrant ~]#  last reboot|head -1
reboot   system boot  5.4.17-2136.307. Sat May 21 11:01 - 11:32  (00:30)
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### NOTICE: Logs begin at Sat 2022-05-21 11:01:41
==================================================

[root@oracle-goldengate-1910-vagrant ~]# journalctl --no-pager|head -2
-- Logs begin at Sat 2022-05-21 11:01:41 -04, end at Sat 2022-05-21 11:30:01 -04. --
May 21 11:01:41 oracle-goldengate-1910-vagrant systemd-journal[151]: Runtime journal is using 8.0M (max allowed 96.8M, trying to leave 145.3M free of 960.7M available → current limit 96.8M).
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### stop_goldengate.sh attempts to gather info:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# ls -alrt /u01/ogg/ggse*
-rw-r-----. 1 oracle oinstall  7835 May 21 11:00 /u01/ogg/ggserr.log.20220521.1100
-rw-r--r--. 1 oracle oinstall   955 May 21 11:00 /u01/ogg/ggserr.log.stop
-rw-r-----. 1 oracle oinstall 10384 May 21 11:02 /u01/ogg/ggserr.log
[root@oracle-goldengate-1910-vagrant ~]#

[root@oracle-goldengate-1910-vagrant ~]# cat /u01/ogg/ggserr.log.stop
2022-05-21T11:00:47.915-0400  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): STOP MGR.
2022-05-21T11:00:47.930-0400  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from GGSCI on host [127.0.0.1]:58056 (STOP).
2022-05-21T11:00:47.930-0400  WARNING OGG-00938  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager is stopping at user request.
2022-05-21T11:00:51.934-0400  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): STOP ER *.
2022-05-21T11:00:52.175-0400  INFO    OGG-01021  Oracle GoldenGate Capture for Oracle, ext.prm:  Command received from GGSCI: STOP.
2022-05-21T11:00:52.179-0400  INFO    OGG-00991  Oracle GoldenGate Capture for Oracle, ext.prm:  EXTRACT EXT stopped normally.
2022-05-21T11:00:59.213-0400  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): INFO ALL.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Continue with demo:
==================================================

==================================================
### Stop goldengate.service using systemctl stop:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# systemctl stop goldengate.service
[root@oracle-goldengate-1910-vagrant ~]# systemctl status goldengate.service
● goldengate.service - Oracle GoldenGate Manager
   Loaded: loaded (/etc/systemd/system/goldengate.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sat 2022-05-21 11:16:37 -04; 13s ago
  Process: 5432 ExecStop=/home/oracle/scripts/stop_goldengate.sh (code=exited, status=0/SUCCESS)
  Process: 4485 ExecStart=/home/oracle/scripts/start_goldengate.sh (code=exited, status=0/SUCCESS)
 Main PID: 4504 (code=exited, status=0/SUCCESS)

May 21 11:16:28 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Manager stopped.
May 21 11:16:28 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 3> STOP ER *
May 21 11:16:29 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Sending STOP request to EXTRACT EXT ...
May 21 11:16:30 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Request processed.
May 21 11:16:30 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 11:16:36 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: MANAGER     STOPPED
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: EXTRACT     STOPPED     EXT         00:00:00      00:00:07
May 21 11:16:37 oracle-goldengate-1910-vagrant systemd[1]: Stopped Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Check journalctl -u goldengate.service:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# journalctl -u goldengate.service|grep -i stop
May 21 11:16:24 oracle-goldengate-1910-vagrant systemd[1]: Stopping Oracle GoldenGate Manager...
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: The Oracle base has been set to /opt/oracle
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: ‘/u01/ogg/ggserr.log’ -> ‘/u01/ogg/ggserr.log.20220521.1116’
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Oracle GoldenGate Command Interpreter for Oracle
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Operating system character set identified as UTF-8.
May 21 11:16:24 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
May 21 11:16:25 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 1>
May 21 11:16:25 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 2> STOP MGR !
May 21 11:16:26 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Sending STOP request to MANAGER ...
May 21 11:16:26 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Request processed.
May 21 11:16:28 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Manager stopped.
May 21 11:16:28 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 3> STOP ER *
May 21 11:16:29 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Sending STOP request to EXTRACT EXT ...
May 21 11:16:30 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Request processed.
May 21 11:16:30 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 11:16:36 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: MANAGER     STOPPED
May 21 11:16:37 oracle-goldengate-1910-vagrant stop_goldengate.sh[5432]: EXTRACT     STOPPED     EXT         00:00:00      00:00:07
May 21 11:16:37 oracle-goldengate-1910-vagrant systemd[1]: Stopped Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### List ggserr logs:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# ls -alrt /u01/ogg/ggse*
-rw-r-----. 1 oracle oinstall  7835 May 21 11:00 /u01/ogg/ggserr.log.20220521.1100
-rw-r-----. 1 oracle oinstall 10384 May 21 11:16 /u01/ogg/ggserr.log.20220521.1116
-rw-r-----. 1 oracle oinstall   955 May 21 11:16 /u01/ogg/ggserr.log
-rw-r--r--. 1 oracle oinstall   955 May 21 11:16 /u01/ogg/ggserr.log.stop
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Services configured for rdbms and goldengate:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# systemctl list-unit-files|egrep -i 'oracle|gate'
goldengate.service                            enabled
oracle-rdbms.service                          enabled
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### Configuration for goldengate.service:
==================================================

[root@oracle-goldengate-1910-vagrant ~]# ls -l /etc/systemd/system/goldengate.service
-rw-r--r--. 1 root root 713 May 19 23:16 /etc/systemd/system/goldengate.service
[root@oracle-goldengate-1910-vagrant ~]#

[root@oracle-goldengate-1910-vagrant ~]# cat /etc/systemd/system/goldengate.service
[Unit]
Description=Oracle GoldenGate Manager
After=syslog.target network.target
# systemctl list-unit-files|grep -i oracle
After=oracle-rdbms.service

[Service]
Type=forking
RemainAfterExit=yes
KillMode=none
User=oracle
Group=oinstall
TimeoutStopSec=60
TimeoutSec=300
Restart=no

# systemd ignores PAM limits, so set any necessary limits in the service.
# Not really a bug, but a feature.
# https://bugzilla.redhat.com/show_bug.cgi?id=754285
# /etc/security/limits.d/oracle-database-preinstall-19c.conf

LimitMEMLOCK=infinity
LimitNOFILE=131072
LimitNPROC=131072

ExecStart=/home/oracle/scripts/start_goldengate.sh
ExecStop=/home/oracle/scripts/stop_goldengate.sh
Restart=no

[Install]
WantedBy=multi-user.target
[root@oracle-goldengate-1910-vagrant ~]#

==================================================
### GoldenGate processes:
==================================================

oracle@oracle-goldengate-1910-vagrant ogg]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.



GGSCI (oracle-goldengate-1910-vagrant) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXT         00:00:00      00:00:00


GGSCI (oracle-goldengate-1910-vagrant) 2> info mgr detail

Manager is running (IP port TCP:oracle-goldengate-1910-vagrant.7809, Process ID 5539).


GGSCI (oracle-goldengate-1910-vagrant) 3> info extract *

EXTRACT    EXT       Last Started 2022-05-20 23:31   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:03 ago)
Process ID           5546
Log Read Checkpoint  Oracle Integrated Redo Logs
                     2022-05-20 23:35:08
                     SCN 0.3722791 (3722791)


GGSCI (oracle-goldengate-1910-vagrant) 4> exit
[oracle@oracle-goldengate-1910-vagrant ogg]$

==================================================
### start_goldengate.sh & start_goldengate.obey:
==================================================

[oracle@oracle-goldengate-1910-vagrant ~]$ cat /home/oracle/scripts/start_goldengate.sh
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

ORAENV_ASK=NO; ORACLE_SID=ORCLCDB; OGG_HOME=/u01/ogg
. /usr/local/bin/oraenv

$OGG_HOME/ggsci << EOF
  OBEY /home/oracle/scripts/start_goldengate.obey
  exit
EOF
exit
[oracle@oracle-goldengate-1910-vagrant ~]$

[oracle@oracle-goldengate-1910-vagrant ~]$ cat /home/oracle/scripts/start_goldengate.obey
START MGR
START ER *
shell sleep 5
INFO ALL
[oracle@oracle-goldengate-1910-vagrant ~]$

==================================================
### stop_goldengate.sh & stop_goldengate.obey:
==================================================

[oracle@oracle-goldengate-1910-vagrant ~]$ cat /home/oracle/scripts/stop_goldengate.sh
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

ORAENV_ASK=NO; ORACLE_SID=ORCLCDB; OGG_HOME=/u01/ogg
. /usr/local/bin/oraenv

# Rotate existing GGSERR_LOG with append date
GGSERR_LOG=$OGG_HOME/ggserr.log
cp -fv ${GGSERR_LOG%%.*}.{${GGSERR_LOG#*.},${GGSERR_LOG#*.}."$(date +%Y%m%d.%H%M)"}
cat /dev/null > $GGSERR_LOG

$OGG_HOME/ggsci << EOF
  OBEY /home/oracle/scripts/stop_goldengate.obey
  exit
EOF
tail -9 $GGSERR_LOG > $GGSERR_LOG.stop
exit
[oracle@oracle-goldengate-1910-vagrant ~]$

[oracle@oracle-goldengate-1910-vagrant ~]$ cat /home/oracle/scripts/stop_goldengate.obey
STOP MGR !
STOP ER *
shell sleep 5
INFO ALL
[oracle@oracle-goldengate-1910-vagrant ~]$

==================================================
### GoldenGate credential store:
==================================================

GGSCI (oracle-goldengate-1910-vagrant) 1> INFO CREDENTIALSTORE

Reading from credential store:

Default domain: OracleGoldenGate

  Alias: c##ggadmin
  Userid: c##ggadmin@ORCLCDB

GGSCI (oracle-goldengate-1910-vagrant) 2>

==================================================
### GoldenGate mgr.prm:
==================================================

[oracle@oracle-goldengate-1910-vagrant ogg]$ cat dirprm/mgr.prm
PORT 7809
USERIDALIAS c##ggadmin
PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 3
AUTOSTART ER *
AUTORESTART ER *, RETRIES 5, WAITMINUTES 2, RESETMINUTES 60
CHECKMINUTES 5
LAGINFOMINUTES 0
LAGCRITICALMINUTES 5
[oracle@oracle-goldengate-1910-vagrant ogg]$

==================================================
### GoldenGate ext.prm:
==================================================

[oracle@oracle-goldengate-1910-vagrant ogg]$ cat dirprm/ext.prm
EXTRACT ext
SETENV (ORACLE_SID = "ORCLCDB")
SETENV (ORACLE_HOME = "/opt/oracle/product/19c/dbhome_1")
USERIDALIAS c##ggadmin
EXTTRAIL ./dirdat/et
GETUPDATEBEFORES
UPDATERECORDFORMAT COMPACT

SOURCECATALOG ORCLPDB1
TABLE ogguser.*;
SEQUENCE ogguser.*;
[oracle@oracle-goldengate-1910-vagrant ogg]$

==================================================
### tnsnames.ora requires (UR=A):
==================================================

ORCLPDB1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 0.0.0.0)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = ORCLPDB1)(UR=A)
    )
  )
  
==================================================
!!! IMPORTANT !!!
==================================================

systemctl is NOT aware when goldengate start/stopp from ggsci!

GGSCI (oracle-goldengate-1910-vagrant) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXT         00:00:00      00:00:09


GGSCI (oracle-goldengate-1910-vagrant) 2> stop mgr !

Sending STOP request to MANAGER ...
Request processed.
Manager stopped.


GGSCI (oracle-goldengate-1910-vagrant) 3> stop er *

Sending STOP request to EXTRACT EXT ...
Request processed.


GGSCI (oracle-goldengate-1910-vagrant) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED
EXTRACT     STOPPED     EXT         00:00:00      00:00:04


GGSCI (oracle-goldengate-1910-vagrant) 5>

==============================

[root@oracle-goldengate-1910-vagrant ~]# systemctl status goldengate.service
● goldengate.service - Oracle GoldenGate Manager
   Loaded: loaded (/etc/systemd/system/goldengate.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sat 2022-05-21 12:00:46 -04; 1min 40s ago
  Process: 5432 ExecStop=/home/oracle/scripts/stop_goldengate.sh (code=exited, status=0/SUCCESS)
  Process: 7921 ExecStart=/home/oracle/scripts/start_goldengate.sh (code=exited, status=0/SUCCESS)
 Main PID: 4504 (code=exited, status=0/SUCCESS)

May 21 12:00:37 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 2> START MGR
May 21 12:00:38 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: MGR is already running.
May 21 12:00:38 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 3> START ER *
May 21 12:00:39 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: EXTRACT EXT is already running.
May 21 12:00:39 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 12:00:45 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: MANAGER     RUNNING
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: EXTRACT     RUNNING     EXT         00:00:00      00:00:00
May 21 12:00:46 oracle-goldengate-1910-vagrant systemd[1]: Started Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

==============================

GGSCI (oracle-goldengate-1910-vagrant) 1> sh date

Sat May 21 12:07:40 -04 2022


GGSCI (oracle-goldengate-1910-vagrant) 2> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED
EXTRACT     STOPPED     EXT         00:00:00      00:05:45


GGSCI (oracle-goldengate-1910-vagrant) 3> start mgr
Manager started.


GGSCI (oracle-goldengate-1910-vagrant) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
EXTRACT     RUNNING     EXT         00:00:00      00:00:03


GGSCI (oracle-goldengate-1910-vagrant) 5>

==============================

[root@oracle-goldengate-1910-vagrant ~]# systemctl status goldengate.service
● goldengate.service - Oracle GoldenGate Manager
   Loaded: loaded (/etc/systemd/system/goldengate.service; enabled; vendor preset: disabled)
   Active: active (exited) since Sat 2022-05-21 12:00:46 -04; 7min ago
  Process: 5432 ExecStop=/home/oracle/scripts/stop_goldengate.sh (code=exited, status=0/SUCCESS)
  Process: 7921 ExecStart=/home/oracle/scripts/start_goldengate.sh (code=exited, status=0/SUCCESS)
 Main PID: 4504 (code=exited, status=0/SUCCESS)

May 21 12:00:37 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 2> START MGR
May 21 12:00:38 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: MGR is already running.
May 21 12:00:38 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 3> START ER *
May 21 12:00:39 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: EXTRACT EXT is already running.
May 21 12:00:39 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 4> shell sleep 5
May 21 12:00:45 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: GGSCI (oracle-goldengate-1910-vagrant) 5> INFO ALL
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: Program     Status      Group       Lag at Chkpt  Time Since Chkpt
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: MANAGER     RUNNING
May 21 12:00:46 oracle-goldengate-1910-vagrant start_goldengate.sh[7921]: EXTRACT     RUNNING     EXT         00:00:00      00:00:00
May 21 12:00:46 oracle-goldengate-1910-vagrant systemd[1]: Started Oracle GoldenGate Manager.
[root@oracle-goldengate-1910-vagrant ~]#

February 17, 2022

DataGuard : ORA-16809: multiple warnings detected – Solution

Filed under: 19c,Dataguard,dgmgrl — mdinh @ 6:31 pm

This is the solution for the issues identified from post above.

--- Use SYS password vs OS authenitication.
--- Note this is RAC.

$ dgmgrl       
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Feb 15 11:55:28 2022
Version 19.13.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@EAGLE
Password:
Connected to "EAGLE"
Connected as SYSDBA.
DGMGRL> show configuration

Configuration - EAGLE

  Protection Mode: MaxPerformance
  Members:
  EAGLE     - Primary database
    EAGLE_PHO - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 49 seconds ago)

DGMGRL> validate network configuration for all;
Connecting to instance "EAGLE1" on database "EAGLE" ...
Connected to "EAGLE"
Checking connectivity from instance "EAGLE1" on database "EAGLE to instance "EAGLE1" on database "EAGLE_PHO"...
Succeeded.
Checking connectivity from instance "EAGLE1" on database "EAGLE to instance "EAGLE2" on database "EAGLE_PHO"...
Succeeded.
Connecting to instance "EAGLE2" on database "EAGLE" ...
Connected to "EAGLE"
Checking connectivity from instance "EAGLE2" on database "EAGLE to instance "EAGLE1" on database "EAGLE_PHO"...
Succeeded.
Checking connectivity from instance "EAGLE2" on database "EAGLE to instance "EAGLE2" on database "EAGLE_PHO"...
Succeeded.
Connecting to instance "EAGLE1" on database "EAGLE_PHO" ...
Connected to "EAGLE_PHO"
Checking connectivity from instance "EAGLE1" on database "EAGLE_PHO to instance "EAGLE1" on database "EAGLE"...
Succeeded.
Checking connectivity from instance "EAGLE1" on database "EAGLE_PHO to instance "EAGLE2" on database "EAGLE"...
Succeeded.
Connecting to instance "EAGLE2" on database "EAGLE_PHO" ...
Connected to "EAGLE_PHO"
Checking connectivity from instance "EAGLE2" on database "EAGLE_PHO to instance "EAGLE1" on database "EAGLE"...
Succeeded.
Checking connectivity from instance "EAGLE2" on database "EAGLE_PHO to instance "EAGLE2" on database "EAGLE"...
Succeeded.

Oracle Clusterware on database "EAGLE" is available for database restart.

Oracle Clusterware on database "EAGLE_PHO" is available for database restart.

DGMGRL> validate static connect identifier for all;
Oracle Clusterware on database "EAGLE" is available for database restart.

Oracle Clusterware on database "EAGLE_PHO" is available for database restart.

DGMGRL> exit
$

February 15, 2022

DataGuard : ORA-16809: multiple warnings detected

Filed under: 19c,Dataguard,dgmgrl — mdinh @ 10:10 pm

EM13c Daily Report was report contained the errors as shown below:

The Data Guard status of EAGLE_PHO is

Warning ORA-16809: multiple warnings detected for the member.

Started to investigate as shown below and what do you think is wrong?

Will give you a chance to guess and post solution later.

$ oerr ora 16809
16809, 00000, "multiple warnings detected for the member"
// *Cause:  The broker detected multiple warnings for the member.
// *Action: To get a detailed status report, check the status of the member
//          specified using either Enterprise Manager or the DGMGRL CLI SHOW
//          command.


$ dgmgrl /
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Tue Feb 15 11:33:54 2022
Version 19.13.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "EAGLE"
Connected as SYSDG.
DGMGRL> show configuration

Configuration - EAGLE

  Protection Mode: MaxPerformance
  Members:
  EAGLE     - Primary database
    EAGLE_PHO - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 29 seconds ago)

DGMGRL> validate network configuration for all;
Connecting to instance "EAGLE1" on database "EAGLE" ...
ORA-01017: invalid username/password; logon denied

Warning: Cannot connect to instance "EAGLE1" on database "EAGLE".
Connecting to instance "EAGLE2" on database "EAGLE" ...
ORA-01017: invalid username/password; logon denied

Warning: Cannot connect to instance "EAGLE2" on database "EAGLE".
Connecting to instance "EAGLE2" on database "EAGLE_PHO" ...
ORA-01017: invalid username/password; logon denied

Warning: Cannot connect to instance "EAGLE2" on database "EAGLE_PHO".
Connecting to instance "EAGLE1" on database "EAGLE_PHO" ...
ORA-01017: invalid username/password; logon denied

Warning: Cannot connect to instance "EAGLE1" on database "EAGLE_PHO".

Oracle Clusterware on database "EAGLE" is available for database restart.

Oracle Clusterware on database "EAGLE_PHO" is available for database restart.

DGMGRL> validate static connect identifier for all;
Oracle Clusterware on database "EAGLE" is available for database restart.

Oracle Clusterware on database "EAGLE_PHO" is available for database restart.

DGMGRL> exit

January 28, 2022

Silent Install Oracle DB 19.3

Filed under: 19c,linux,response_file,runInstaller — mdinh @ 10:16 pm

I was asked to Install Oracle 19c EE with the provided information:

DB: /app/oracle/product/19.3.0
SW: /backup/media/oracle19.3_database/LINUX.X64_193000_db_home.zip

Not knowing anything about the environment, more due diligence is required and here are the steps taken:

--- Find OS info:

oracle@oradev001: ~ $ uname -an
Linux oradev001 3.10.0-1160.31.1.el7.x86_64 #1 SMP Wed May 26 20:18:08 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

--- Create new directory:

oracle@oradev001: ~ $ ls -ld /app/oracle/product/19.3.0
ls: cannot access /app/oracle/product/19.3.0: No such file or directory

oracle@oradev001: ~ $ mkdir -p /app/oracle/product/19.3.0
oracle@oradev001: ~ $ ls -ld /app/oracle/product/19.3.0
drwxr-xr-x 2 oracle dba 4096 Jan 26 12:44 /app/oracle/product/19.3.0

--- Extract software:

oracle@oradev001: ~ $ cd /app/oracle/product/19.3.0
oracle@oradev001: /app/oracle/product/19.3.0 $ unzip -oq /backup/media/oracle19.3_database/LINUX.X64_193000_db_home.zip; echo $?
0

--- Extract rspfmt_dbinstall_response_schema_v19.0.0 from db_install.rsp:

oracle@oradev001: /app/oracle/product/19.3.0 $ ls -l install/response/db_install.rsp
-rw-r--r-- 1 oracle dba 19932 Feb  6  2019 install/response/db_install.rsp

oracle@oradev001: /app/oracle/product/19.3.0 $ grep responseFileVersion install/response/db_install.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0

--- Find UNIX_GROUP_NAME (oinstall or dba):

oracle@oradev001: /app/oracle/product/19.3.0 $ id oracle
uid=3000(oracle) gid=3000(dba) groups=3000(dba),54329(asmadmin),54327(asmdba)

--- Find ORACLE_BASE:

oracle@oradev001: /app/oracle/product/19.3.0 $ env|grep ORACLE_BASE
ORACLE_BASE=/app/oracle

--- Find INVENTORY_LOCATION:

oracle@oradev001: /app/oracle/product/19.3.0 $ ls -l /etc/oraInst.loc
-rw-r--r-- 1 root root 47 Nov  7  2018 /etc/oraInst.loc

--- Create and copy response file to /tmp:

oracle@oradev001: /app/oracle/product/19.3.0 $ cat db_install.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_dbinstall_response_schema_v19.0.0
oracle.install.option=INSTALL_DB_SWONLY
UNIX_GROUP_NAME=dba
INVENTORY_LOCATION=/etc/oraInst.loc
SELECTED_LANGUAGES=en
ORACLE_HOME=/app/oracle/product/19.3.0
ORACLE_BASE=/app/oracle
oracle.install.db.InstallEdition=EE
oracle.install.db.OSDBA_GROUP=dba
oracle.install.db.OSBACKUPDBA_GROUP=dba
oracle.install.db.OSDGDBA_GROUP=dba
oracle.install.db.OSKMDBA_GROUP=dba
oracle.install.db.OSRACDBA_GROUP=dba
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false
DECLINE_SECURITY_UPDATES=true

--- Perform executePrereqs and Ignore:

oracle@oradev001: /app/oracle/product/19.3.0 $ ./runInstaller -silent -executePrereqs -waitforcompletion -force -responseFile /tmp/db_install.rsp
Launching Oracle Database Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /app/oraInventory/logs/InstallActions2022-01-26_01-19-52PM/installActions2022-01-26_01-19-52PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /app/oraInventory/logs/InstallActions2022-01-26_01-19-52PM/installActions2022-01-26_01-19-52PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

--- Install software only;

oracle@oradev001: /app/oracle/product/19.3.0 $ ./runInstaller -silent -waitforcompletion -responseFile /tmp/db_install.rsp
Launching Oracle Database Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /app/oracle/product/19.3.0/install/response/db_2022-01-26_01-28-45PM.rsp

You can find the log of this install session at:
 /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log

As a root user, execute the following script(s):
        1. /app/oracle/product/19.3.0/root.sh

Execute /app/oracle/product/19.3.0/root.sh on the following nodes:
[oradev001]


Successfully Setup Software with warning(s).
oracle@oradev001: /app/oracle/product/19.3.0 $

--- Review ignorable errors: insufficient swap and OS kernel parameter "shmall"

oracle@oradev001: /app/oracle/product/19.3.0 $ grep -C 10 INS-13014 /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log
INFO:  [Jan 26, 2022 1:29:03 PM] ------------------List of failed Tasks------------------
INFO:  [Jan 26, 2022 1:29:03 PM] *********************************************
INFO:  [Jan 26, 2022 1:29:03 PM] Swap Size: This is a prerequisite condition to test whether sufficient total swap space is available on the system.
INFO:  [Jan 26, 2022 1:29:03 PM] Severity:IGNORABLE
INFO:  [Jan 26, 2022 1:29:03 PM] OverallStatus:VERIFICATION_FAILED
INFO:  [Jan 26, 2022 1:29:03 PM] *********************************************
INFO:  [Jan 26, 2022 1:29:03 PM] OS Kernel Parameter: shmall: This is a prerequisite condition to test whether the OS kernel parameter "shmall" is properly set.
INFO:  [Jan 26, 2022 1:29:03 PM] Severity:IGNORABLE
INFO:  [Jan 26, 2022 1:29:03 PM] OverallStatus:VERIFICATION_FAILED
INFO:  [Jan 26, 2022 1:29:03 PM] -----------------End of failed Tasks List----------------

WARNING:  [Jan 26, 2022 1:29:03 PM] [WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /app/oraInventory/logs/InstallActions2022-01-26_01-28-45PM/installActions2022-01-26_01-28-45PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

INFO:  [Jan 26, 2022 1:29:03 PM] Advice is CONTINUE
INFO:  [Jan 26, 2022 1:29:03 PM] Completed validating state <performChecks>
INFO:  [Jan 26, 2022 1:29:03 PM] Verifying route success
INFO:  [Jan 26, 2022 1:29:03 PM] Executing initialize action at state showSummary
INFO:  [Jan 26, 2022 1:29:03 PM] Completed executing initialize action at state <showSummary>
INFO:  [Jan 26, 2022 1:29:03 PM] Waiting for completion of background operations
INFO:  [Jan 26, 2022 1:29:03 PM] Completed background operations
INFO:  [Jan 26, 2022 1:29:03 PM] Waiting for completion of background operations
oracle@oradev001: /app/oracle/product/19.3.0 $

August 20, 2021

AutoUpgrade 21.2.210721 Data Guard ORA-16532

Filed under: 19c,Dataguard,upgrade — mdinh @ 8:25 pm

Basically, I was somewhat purposely reckless as I wanted to see how much the new AutoUpgrade will do.

After Data Guard System Upgrade, Error reported ORA-16532 (Doc ID 2186456.1)

Environment is single instance and no Grid installed.

Learned something new :=)

Copy dg_broker_config_file and network files from old home to new home.

$ORACLE_HOME = /opt/oracle/product/19c/dbhome_1

cp -v /u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/dr*.dat $ORACLE_HOME/dbs
cp -v /u01/app/oracle/product/11.2.0.4/dbhome_1/network/admin/* $ORACLE_HOME/network/admin


[oracle@ol7-112-dg1 admin]$ dgmgrl /
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Fri Aug 20 19:40:23 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "hawk"
Connected as SYSDG.
DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
      Error: ORA-16525: The Oracle Data Guard broker is not yet available.

Fast-Start Failover:  Disabled

Configuration Status:
ERROR   (status updated 11 seconds ago)

DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
      Warning: ORA-16853: apply lag has exceeded specified threshold

Fast-Start Failover:  Disabled

Configuration Status:
WARNING   (status updated 39 seconds ago)

DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 55 seconds ago)

DGMGRL>

July 17, 2021

RMAN Backup Tuning – MAXOPENFILES

Filed under: 19c,RMAN — mdinh @ 5:03 pm

Currently, database backup is performed using RMAN from shell script.

The shell script has has 2,347 lines. Does anyone KISS anymore?

Anyhow, I recalled very early in my career when someone coined the phrased, “Doing It The Dinh’s Way”.

Basically, it’s not effective nor efficient to debug and troubleshoot shell script.

Client provided the following info and off I went.

Database backup changed from filesperset 4 to filesperset 1.

Rollback change and still facing performance issues.

Level 1 increment backup with filesperset = 4 still running > 24 hours – killed.

Can you try a few things not using the rman script in crontab, but using your own rman command?

Request was to disable block change tracking – DONE.

Database size is 272G with the following datafiles.

SQL> select file_id, file_name, round(bytes/1024/1024/1024) gb from dba_data_files order by 1 asc;

 FILE_ID FILE_NAME                                                          GB
-------- ------------------------------------------------------------ --------
       1 +DATA_XXXDWP01/XXXdwp01/datafile/system.256.964878417               1
       2 +DATA_XXXDWP01/XXXdwp01/datafile/sysaux.257.964878419               4
       3 +DATA_XXXDWP01/XXXdwp01/datafile/undotbs1.258.964878421            21
       4 +DATA_XXXDWP01/XXXdwp01/datafile/gg_tbs.261.964878453               5
       5 +DATA_XXXDWP01/XXXdwp01/datafile/undotbs2.262.964878457            19
       6 +DATA_XXXDWP01/XXXdwp01/datafile/users.264.965655143               31
       7 +DATA_XXXDWP01/XXXdwp01/datafile/users.267.1020339295              32
       8 +DATA_XXXDWP01/XXXdwp01/datafile/dev_users.268.1027263011           0
       9 +DATA_XXXDWP01/XXXdwp01/datafile/dev_users.269.1027263013           0
      10 +DATA_XXXDWP01/XXXdwp01/datafile/dev_users.270.1027263013           0
      11 +DATA_XXXDWP01/XXXdwp01/datafile/dev_users.271.1027263013           0
      12 +DATA_XXXDWP01/XXXdwp01/datafile/dev_users.272.1027263015           0
      13 +DATA_XXXDWP01/XXXdwp01/datafile/users.274.1046508647              32
      14 +DATA_XXXDWP01/XXXdwp01/datafile/users.275.1056870315              32
      15 +DATA_XXXDWP01/XXXdwp01/datafile/users.276.1066387693              32
      16 +DATA_XXXDWP01/XXXdwp01/datafile/users.277.1066387695              32
      17 +DATA_XXXDWP01/XXXdwp01/datafile/users.278.1066387697              32

17 rows selected.

SQL>

Backup completed in 3.25 hours using MAXPIECESIZE 2G MAXOPENFILES 1.

Believe further improvements by increasing MAXPIECESIZE to 4G; however, client wanted to leave as is.

Backupset size is 24G.

I believe performance may be related to Pure Storage Deduplication; however, there’s no more time nor different destination to test with.

MAXOPENFILES ≤ 4
Each buffer = 1MB, total buffer size for channel is up to 16MB

MAXOPENFILES = 1 => 16 buffers/file, 1 MB/buffer = 16 MB/file
Optimal for ASM or striped system

Recovery Manager: Release 19.0.0.0.0 - Production on Fri Jul 16 11:08:18 2021 Version 19.8.0.0.0

RMAN> connect target;
2> set echo on
3> spool log to backup.log
4> show all;
5> run {
6> allocate channel c1 device type disk format '/backup/XXXDWP01/%d_%I_%T_%U_%s' MAXPIECESIZE 2G MAXOPENFILES 1;
7> allocate channel c2 device type disk format '/backup/XXXDWP01/%d_%I_%T_%U_%s' MAXPIECESIZE 2G MAXOPENFILES 1;
8> allocate channel c3 device type disk format '/backup/XXXDWP01/%d_%I_%T_%U_%s' MAXPIECESIZE 2G MAXOPENFILES 1;
9> allocate channel c4 device type disk format '/backup/XXXDWP01/%d_%I_%T_%U_%s' MAXPIECESIZE 2G MAXOPENFILES 1;
10> backup incremental level 1 filesperset 1 database;
11> }
12> exit

June 8, 2021

Create Delete RAC DB Using dbca silent

Filed under: 19c,dbca — mdinh @ 10:18 pm

Tested version.

[oracle@ol7-19-lax1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@ol7-19-lax1 ~]$

Create database using dbca silent.

--- Set environment variables to be used by dbca.
export ORACLE_UNQNAME=owl
export PDB_NAME=mice
export NODE1=ol7-19-lax1
export NODE2=ol7-19-lax2
export SYS_PASSWORD=Oracle_4U
export PDB_PASSWORD=Oracle_4U

Note: -gdbName global_database_name (-gdbname oradb.example.com)

dbca -silent -createDatabase \
  -templateName General_Purpose.dbc \
  -gdbname ${ORACLE_UNQNAME} -responseFile NO_VALUE \
  -characterSet AL32UTF8 \
  -sysPassword ${SYS_PASSWORD} \
  -systemPassword ${SYS_PASSWORD} \
  -createAsContainerDatabase true \
  -numberOfPDBs 1 \
  -pdbName ${PDB_NAME} \
  -pdbAdminPassword ${PDB_PASSWORD} \
  -databaseType MULTIPURPOSE \
  -automaticMemoryManagement false \
  -totalMemory 1024 \
  -redoLogFileSize 50 \
  -emConfiguration NONE \
  -ignorePreReqs \
  -nodelist ${NODE1},${NODE2} \
  -storageType ASM \
  -diskGroupName +DATA \
  -recoveryGroupName +RECO \
  -useOMF true \
  -asmsnmpPassword ${SYS_PASSWORD}

[oracle@ol7-19-lax1 ~]$ dbca -silent -createDatabase \
>   -templateName General_Purpose.dbc \
>   -gdbname ${ORACLE_UNQNAME} -responseFile NO_VALUE \
>   -characterSet AL32UTF8 \
>   -sysPassword ${SYS_PASSWORD} \
>   -systemPassword ${SYS_PASSWORD} \
>   -createAsContainerDatabase true \
>   -numberOfPDBs 1 \
>   -pdbName ${PDB_NAME} \
>   -pdbAdminPassword ${PDB_PASSWORD} \
>   -databaseType MULTIPURPOSE \
>   -automaticMemoryManagement false \
>   -totalMemory 1024 \
>   -redoLogFileSize 50 \
>   -emConfiguration NONE \
>   -ignorePreReqs \
>   -nodelist ${NODE1},${NODE2} \
>   -storageType ASM \
>   -diskGroupName +DATA \
>   -recoveryGroupName +RECO \
>   -useOMF true \
>   -asmsnmpPassword ${SYS_PASSWORD}
Prepare for db operation
7% complete
Copying database files
27% complete
Creating and starting Oracle instance
28% complete
31% complete
35% complete
37% complete
40% complete
Creating cluster database views
41% complete
53% complete
Completing Database Creation
57% complete
59% complete
60% complete
Creating Pluggable Databases
64% complete
80% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
 /u01/app/oracle/cfgtoollogs/dbca/owl.
Database Information:
Global Database Name:owl
System Identifier(SID) Prefix:owl
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/owl/owl.log" for further details.
[oracle@ol7-19-lax1 ~]$

Log files are located at ORACLE_BASE/cfgtoollogs/dbca/${ORACLE_UNQNAME}

[oracle@ol7-19-lax1 ~]$ cd $ORACLE_BASE/cfgtoollogs/dbca/${ORACLE_UNQNAME}
[oracle@ol7-19-lax1 owl]$ pwd
/u01/app/oracle/cfgtoollogs/dbca/owl

[oracle@ol7-19-lax1 owl]$ ls -l
total 23576
-rw-r-----. 1 oracle oinstall    10422 Jun  8 21:20 catclust0.log
-rw-------. 1 oracle oinstall   201621 Jun  8 21:20 catclust_catcon_31776.lst
-rw-r-----. 1 oracle oinstall     2450 Jun  8 21:14 cloneDBCreation.log
-rw-r-----. 1 oracle oinstall      380 Jun  8 20:57 CloneRmanRestore.log
-rw-r-----. 1 oracle oinstall    44272 Jun  8 21:20 CreateClustDBViews.log
-rw-r-----. 1 oracle oinstall     1711 Jun  8 21:49 DBDetails.log
-rw-r-----. 1 oracle oinstall     9948 Jun  8 21:19 execemx0.log
-rw-------. 1 oracle oinstall   200759 Jun  8 21:19 execemx_catcon_31544.lst
-rw-r-----. 1 oracle oinstall      910 Jun  8 21:20 lockAccount.log
-rw-r-----. 1 oracle oinstall     9560 Jun  8 21:18 ordlib0.log
-rw-------. 1 oracle oinstall   200561 Jun  8 21:18 ordlib_catcon_31269.lst
-rw-r-----. 1 oracle oinstall      796 Jun  8 21:51 owl0.log
-rw-r-----. 1 oracle oinstall      952 Jun  8 21:34 owl.log
-rw-r-----. 1 oracle oinstall        0 Jun  8 21:33 PDBCreation.log
-rw-r-----. 1 oracle oinstall       28 Jun  8 21:34 plugDatabase1R.log
-rw-r-----. 1 oracle oinstall     4105 Jun  8 21:18 plugDatabase.log
-rw-r-----. 1 oracle oinstall    46082 Jun  8 21:33 postDBCreation.log
-rw-r-----. 1 oracle oinstall       24 Jun  8 21:34 postPDBCreation.log
-rw-r-----. 1 oracle oinstall    88296 Jun  8 21:19 postScripts.log
-rw-r-----. 1 oracle oinstall        0 Jun  8 21:50 rmanUtil
-rw-r-----. 1 oracle oinstall     1479 Jun  8 21:49 ShutdownInst.log
-rw-r-----. 1 oracle oinstall 18726912 Jun  8 20:57 tempControl.ctl
-rw-r-----. 1 oracle oinstall  2670547 Jun  8 21:34 trace.log_2021-06-08_08-55-55PM
-rw-r-----. 1 oracle oinstall  1649779 Jun  8 21:51 trace.log_2021-06-08_09-49-34PM
-rw-r-----. 1 oracle oinstall    15420 Jun  8 21:32 utlrp0.log
-rw-------. 1 oracle oinstall   200463 Jun  8 21:32 utlrp_catcon_3004.lst
[oracle@ol7-19-lax1 owl]$

Here is the created database.

[oracle@ol7-19-lax1 owl]$ srvctl config database -d owl
Database unique name: owl
Database name: owl
Oracle home: /u01/app/oracle/product/19.0.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/OWL/PARAMETERFILE/spfile.311.1074720717
Password file: +DATA/OWL/PASSWORD/pwdowl.298.1074718605
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: RECO,DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: owl1,owl2
Configured nodes: ol7-19-lax1,ol7-19-lax2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managed

[oracle@ol7-19-lax1 owl]$ srvctl status database -d owl
Instance owl1 is running on node ol7-19-lax1
Instance owl2 is running on node ol7-19-lax2

--- Newly created database was not added to oratab.
[oracle@ol7-19-lax1 owl]$ cat /etc/oratab
#Backup file is  /u01/app/oracle/product/19.0.0/dbhome_1/srvm/admin/oratab.bak.ol7-19-lax1.oracle line added by Agent
+ASM1:/u01/app/19.0.0/grid:N
hawk1:/u01/app/oracle/product/19.0.0/dbhome_1:N
[oracle@ol7-19-lax1 owl]$ 

Delete database using dbca silent.

dbca -silent -deleteDatabase -sourceDB ${ORACLE_UNQNAME} -sysDBAUserName sys -sysDBAPassword ${SYS_PASSWORD}

[oracle@ol7-19-lax1 ~]$ dbca -silent -deleteDatabase -sourceDB ${ORACLE_UNQNAME} -sysDBAUserName sys -
sysDBAPassword ${SYS_PASSWORD}

[WARNING] [DBT-19202] The Database Configuration Assistant will delete the Oracle instances and datafiles for your database. All information in the database will be destroyed.
Prepare for db operation
32% complete
Connecting to database
39% complete
42% complete
45% complete
48% complete
52% complete
55% complete
58% complete
65% complete
Updating network configuration files
68% complete
Deleting instances and datafiles
77% complete
87% complete
97% complete
100% complete
Database deletion completed.
Look at the log file "/u01/app/oracle/cfgtoollogs/dbca/owl/owl0.log" for further details.
[oracle@ol7-19-lax1 ~]$

May 1, 2021

Extract DB User Password

Filed under: 19c,oracle — mdinh @ 12:40 am

For some reason, I had a mental block in trying to extract password for database users.

Here are some options.

SQL> select banner from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

SQL>

-- From notes:
SQL> set echo off head off verify off feedb off pages 0 long 10000 longchunk 10000 trimspool on lines 2000 timing off term off
SQL> exec dbms_metadata.set_transform_param(dbms_metadata.SESSION_TRANSFORM,'SQLTERMINATOR',TRUE);
SQL> exec dbms_metadata.set_transform_param(dbms_metadata.SESSION_TRANSFORM,'PRETTY',TRUE);

SQL> r
  1  select 'alter user '||username||' identified by values '||REGEXP_SUBSTR(DBMS_METADATA.get_ddl ('USER',USERNAME), '''[^'']+''')||';' ddl
  2  from dba_users where username='CTXSYS'
  3*
alter user CTXSYS identified by values 'S:29174843DD6989AA921A152BF37264659F042C2D8C216D97F6176AF13E4F;T:7B311EEA53E028F937CFABC4D6F6142E3027195E099798CD9E67910ABE6C621E9C23780121F208451B95AB558A4862F206A917C93B50D96E8F573FE7E1A4B2E98D77B9504BC2EBB457B63600127E34D';

SQL>


-- From colleague
SQL> select name, spare4 from sys.user$ where name='CTXSYS';
CTXSYS
S:29174843DD6989AA921A152BF37264659F042C2D8C216D97F6176AF13E4F;T:7B311EEA53E028F937CFABC4D6F6142E3027195E099798CD9E67910ABE6C621E9C23780121F208451B95AB558A4862F206A917C93B50D96E8F573FE7E1A34B2E98D77B9504BC2EBB457B63600127E34D

SQL>

-- From asktom
SQL> r
  1  with t as
  2  (select TO_CHAR(dbms_metadata.get_ddl('USER','CTXSYS')) ddl from dual )
  3  select replace(substr(ddl,1,instr(ddl,'DEFAULT')-1),'CREATE','ALTER')||';' sql
  4* from t

ALTER USER "CTXSYS" IDENTIFIED BY VALUES 'S:29174843DD6989AA921A152BF37264659F042C2D8C216D97F6176AF13E4F;T:7B311EEA53E028F937CFABC4D6F6142E3027195E099798CD9E67910ABE6C621E9C23780121F208451B95AB558A4862F206A917C93B50D96E8F573FE7E1A34B2E98D77B9504BC2EBB457B63600127E34D';

SQL>

December 4, 2020

DBSAT Error: Data collection was not successful.

Filed under: 19c — mdinh @ 6:29 pm
Tags:

First time running DBSAT for 19.8 RAC CDB and it failed miserably.

Create TNS entry for PDB and connect as system@PDB_service failed.

Many have tried and failed and thanks to Roy Salazar who came up with a work around.

Here is a demo for error and work around.

### Error: Data collection was not successful.
$ export TNS_ADMIN=/u01/app/grid/19.0/network/admin
$ ./dbsat collect  "/ as sysdba" dbsat_$ORACLE_SID
Setup complete.
SQL queries complete.
OS commands complete.
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.8.0.0.0
Error: Data collection was not successful.

### There should have been dbsat_$ORACLE_SID.json; however, nowrap.json exist due to error.
$ ls -l *.json
-rw-------. 1 oracle oracle 486835 Dec  5 04:34 nowrap.json

### Create report using nowrap which will create nowrap_report.zip and rename nowrap_report.zip accordingly. Unfortunately, the report contents are still named nowrap.
$ ./dbsat report nowrap
 Database Security Assessment Tool version 2.2.1 (May 2020)
 This tool is intended to assist you in securing your Oracle database
 system. You are solely responsible for your system and the effect and
 results of the execution of this tool (including, without limitation,
 any damage or data loss). Further, the output generated by this tool may
 include potentially sensitive system configuration data and information
 that could be used by a skilled attacker to penetrate your system. You
 are solely responsible for ensuring that the output of this tool,
 including any generated reports, is handled in accordance with your
 company's policies.
 DBSAT Reporter ran successfully.
 Calling /usr/bin/zip to encrypt the generated reports…
 Enter password:
 Verify password:
   zip warning: nowrap_report.zip not found or empty
   adding: nowrap_report.txt (deflated 77%)
   adding: nowrap_report.html (deflated 83%)
   adding: nowrap_report.xlsx (deflated 3%)
   adding: nowrap_report.json (deflated 81%)
 zip completed successfully.

$ ls -ltrh
 total 6.1M
 -r-xr-xr-x. 1 oracle oracle  14K May  6  2020 dbsat
 -rw-rw-r--. 1 oracle oracle 290K May  6  2020 sat_reporter.py
 -rw-rw-r--. 1 oracle oracle  61K May  6  2020 sat_collector.sql
 -rw-rw-r--. 1 oracle oracle  26K May  6  2020 sat_analysis.py
 -r-xr-xr-x. 1 oracle oracle  14K May  6  2020 dbsat.bat
 -rw-r--r--. 1 oracle oracle 4.5M Dec  3 04:33 dbsat.zip
 drwxr-xr-x. 5 oracle oracle   40 Dec  3 04:33 Discover
 -rw-------. 1 oracle oracle  26K Dec  4 09:40 sat_analysis.pyc
 drwxr-xr-x. 2 oracle oracle 4.0K Dec  4 09:40 xlsxwriter
 -rw-------. 1 oracle oracle 476K Dec  5 04:34 nowrap.json
 -rw-------. 1 oracle oracle 105K Dec  5 04:36 nowrap_report.zip
Next Page »

Create a free website or blog at WordPress.com.