Thinking Out Loud

January 7, 2017

Not Another Post Configuring HugePages for Oracle on Linux (x86-64)

Filed under: 11g,linux,oracle — mdinh @ 6:46 am

USE_LARGE_PAGES (TRUE/FALSE/ONLY)

Test case is only for one database instance on server.

DB is using memory_target.

SQL> show parameter use_large_pages

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
use_large_pages                      string      TRUE

SQL> show parameter sga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
lock_sga                             boolean     FALSE
pre_page_sga                         boolean     FALSE
sga_max_size                         big integer 768M
sga_target                           big integer 0

SQL> show parameter pga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target                 big integer 0

SQL> show parameter target

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
archive_lag_target                   integer     900
db_flashback_retention_target        integer     1440
fast_start_io_target                 integer     0
fast_start_mttr_target               integer     0
memory_max_target                    big integer 1G
memory_target                        big integer 1G
parallel_servers_target              integer     16
pga_aggregate_target                 big integer 0
sga_target                           big integer 0

SQL> show sga

Total System Global Area  801701888 bytes
Fixed Size                  2257520 bytes
Variable Size             276827536 bytes
Database Buffers          520093696 bytes
Redo Buffers                2523136 bytes
SQL> exit

Gather memory configuration.

$ grep Huge /proc/meminfo
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

++++++++++

$ grep PageTables /proc/meminfo
PageTables:        24428 kB

++++++++++

$ free
             total       used       free     shared    buffers     cached
Mem:       1534604    1484476      50128       1512      20352     335916
-/+ buffers/cache:    1128208     406396
Swap:      4194300          0    4194300

Calculate memlock.

SQL> select round(1534604*.875) from dual;

ROUND(1534604*.875)
-------------------
            1342779

Edit /etc/security/limits.conf to add memlock as shown below.
Logout, login, check ulimit -l

$ grep memlock /etc/security/limits.conf
#        - memlock - max locked-in-memory address space (KB)
oracle   soft   memlock    1342779
oracle   hard   memlock    1342779

++++++++++

$ ulimit -l
134217728

Run hugepages_settings.sh ERROR due to memory target being used.

$ ./hugepages_settings.sh

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating SGA size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

***********
** ERROR **
***********
Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured

Remove memory target configuration from DB.
Note reset was used as shown:
alter system reset memory_target scope=spfile sid=’*’;
alter system reset memory_max_target scope=spfile sid=’*’;

oracle@arrow1:HAWKA:/home/oracle
$ sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jan 6 21:33:31 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning and Real Application Testing options

SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /u01/app/oracle/product/11.2.0
                                                 .4/db_1/dbs/spfileHAWKA.ora
SQL> alter system reset memory_target scope=spfile sid='*';

System altered.

SQL> alter system reset memory_max_target scope=spfile sid='*';

System altered.

SQL> alter system set sga_max_size=768M scope=spfile sid='*';

System altered.

SQL> alter system set sga_target=768M scope=spfile sid='*';

System altered.

SQL> alter system set pga_aggregate_target=256M scope=spfile sid='*';

System altered.

SQL> shu immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.

Total System Global Area  801701888 bytes
Fixed Size                  2257520 bytes
Variable Size             276827536 bytes
Database Buffers          520093696 bytes
Redo Buffers                2523136 bytes
Database mounted.
Database opened.
SQL> @show.sql
SQL> show parameter use_large_pages

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
use_large_pages                      string      TRUE
SQL> show parameter sga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
lock_sga                             boolean     FALSE
pre_page_sga                         boolean     FALSE
sga_max_size                         big integer 768M
sga_target                           big integer 768M
SQL> show parameter pga

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_target                 big integer 256M
SQL> show parameter target

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
archive_lag_target                   integer     900
db_flashback_retention_target        integer     1440
fast_start_io_target                 integer     0
fast_start_mttr_target               integer     0
memory_max_target                    big integer 0
memory_target                        big integer 0
parallel_servers_target              integer     16
pga_aggregate_target                 big integer 256M
sga_target                           big integer 768M
SQL> show sga

Total System Global Area  801701888 bytes
Fixed Size                  2257520 bytes
Variable Size             276827536 bytes
Database Buffers          520093696 bytes
Redo Buffers                2523136 bytes
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning and Real Application Testing options
oracle@arrow1:HAWKA:/home/oracle
$

Run hugepages_settings.sh – Recommended setting: vm.nr_hugepages = 388

$ ./hugepages_settings.sh

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating SGA size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 388

Manually calculate vm.nr_hugepages in KB using [sga_max_size(768M) * 1024 * Hugepagesize(2048 kB)]

SQL> select round(768*1024/2048)+1 from dual;

ROUND(768*1024/2048)+1
----------------------
                   385

SQL>

From alert log – vm.nr_hugepages=385

Fri Jan 06 21:34:33 2017
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = 128 GB

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
  Total System Global Area size is 770 MB. For optimal performance,
  prior to the next instance restart:
  1. Increase the number of unused large pages by at least 385 
     (page size 2048 KB, total size 770 MB) system wide to get 
     100% of the System Global Area allocated with large pages
********************************************************************

Configure Dynamic vm.nr_hugepages=385

[root@arrow1 ~]# sysctl -w vm.nr_hugepages=385
vm.nr_hugepages = 385
[root@arrow1 ~]# grep Huge /proc/meminfo
HugePages_Total:     353
HugePages_Free:      353
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

Configure Static vm.nr_hugepages=385 and restart server – Oracle recommended.

[root@arrow1 ~]# grep vm.nr_hugepages /etc/sysctl.conf
[root@arrow1 ~]# vi /etc/sysctl.conf
[root@arrow1 ~]# grep vm.nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=385
[root@arrow1 ~]# cat /etc/system-release
Oracle Linux Server release 6.6
[root@arrow1 ~]#

HugePages on Oracle Linux 64-bit (Doc ID 361468.1)
Modified:Mar 7, 2016

Step 6: Stop all the database instances and reboot the server
(Although settings could have been done dynamically they would not be effective to the extent we require before a reboot.
The best practice is to do a persistent system configuration and perform a reboot to complete the configuration as presented through the steps above)

ALERT: Disable Transparent HugePages on SLES11, RHEL6, RHEL7, OL6, OL7, and UEK2 and above (Doc ID 1557478.1)
cat /boot/grub/grub.conf

Actual error example.

$ free 
              total        used        free      shared  buff/cache   available
Mem:      263760440    34016732   173314152      786468    56429556   228457668
Swap:      16777212           0    16777212

++++++++++

ARROW1:(SYS@HAWKA):PRIMARY> select round(263760440*.875) memlock from dual;

 MEMLOCK
----------
 230790385

ARROW1:(SYS@HAWKA):PRIMARY>

++++++++++

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:   15400
HugePages_Free:    15023
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

$ grep memlock /etc/security/limits.conf
oracle	soft    memlock 	unlimited
oracle	hard    memlock 	unlimited

*.sga_max_size=32212254720/1024 = KB

SYS@HAWKA> select round(32212254720/1024/2048)+1 from dual;

ROUND(32212254720/1024/2048)+1
------------------------------
                         15361

select 'vm.nr_hugepages='||to_char(round(32212254720/1024/2048,-2)+100) hugepages
from dual
 3 ;

HUGEPAGES
---------------------
vm.nr_hugepages=15500

ARROW1:(SYS@HAWKA):PRIMARY>

From alert.log
    Thu Feb 09 15:16:55 2017
      PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
    Thu Feb 09 15:16:55 2017
         2048K            15023           15362           15012        NONE
Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: