OIM / OID : Referential Integrity for referenced group memberships on change username

Use Case 

Change Username (typically via OIM APIs) doesn't update group memberships. When we change username, the cn of the user changes. Since the dn comprises of cn and the dc, the referenced group memberships are orphaned.

This is because, by default, "Referential Integrity" is turned OFF in OID and OIM. 

Refer Section : C.2.2 Common Name Generation for Modify User Operation

"Because the CN changed, Oracle Identity Manager attempts a modify operation (modrdn) in the directory resulting in DN change. Because of this unintended DN change, the group membership DN becomes stale resulting in the user loosing membership in that group. This subsequently results in authorization failure. This happens when referential integrity is turned off in the LDAP server, and therefore, the referenced groups are not updated when the RDN of the user changes. Therefore, referential integrity must be turned on in the target LDAP server. Otherwise, the group memberships become stale. The referential integrity issue is also applicable to roles. Groups are also members of other groups and any RDN changes must be reflected as well."

Please see table in link for scenarios when this property is not consistent in OIM and OID.

• Turn on Referential Integrity Check in OID : 

Change orclRIEnabled to 2 within cn=dsaconfig,cn=configsets,cn=oracle internet directory [Default value is 0]
http://docs.oracle.com/cd/E15586_01/oid.1111/e10029/ref_integ.htm

• In OIM, modify System property : XL.IsReferentialIntegrityEnabled to TRUE

The above will make sure change username is seamless and doesn't cause any data corruption issues. Obviously, we need to have LDAP Sync turned on between OIM and OID to make sure that this works as expected.

OHS : Periodic OHS/Web Server/WebGate Crash due to cron job incorrectly deleting *.lck files/httpd.pid files

Issue

We faced a very unique issue for one of our OAM Single Sign On implementations wherein , all the OHS Nodes in a cluster setup used to crash every 7th day generating core dumps running into dozens of GBs which potentially used to crash the OHS in addition to the downtime on Production systems.

Stack Trace on OHS/Webgates :

Loaded symbols for /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/libxmlengine.so
Core was generated by `/u01/app/orasec/middleware/Oracle_WT1/ohs/bin/httpd.worker -DSSL'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fd717f1161f in ObLockFileRelease(void*, bool) ()
  from /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/webgate.so
(gdb) (gdb)

Detailed Analysis of Root Cause 

On detailed debugging and some guidance from Oracle Support, we discovered that this was being caused by a Cron Job which was written to ensure oam_server.out files as well as oblog.log files get deleted every 7 days. This was due to the fact that Oracle doesn't provided log retention policies for these files OOTB.

The path that was used by the Cron Job was <MiddlewareHome>/<Oracle_WebTier>/instances/<instance_name>/diagnostics/logs/OHS/ohs1 which incidentally also hosted the important .lck files (polltracking.lck, oblog.log.lck, ObAccessClient.xml.lck)  and http.pid files [Why Oracle, Why ??!!]

Remember : Removing PID and *.lck files caused instability an is not supported by OAM or OHS.

Solution 

It is not supported to remove httpd.pid and *.lck or log files that are created by a running instance while it is running - 

1.  Setup up logging to another location where lock file and httpd.pid and other process files do not exist, if it's a cron job or something else is used to remove those files. In our case we explicitly called out the files which needed to be deleted instead of running the cronjob on a folder.

2.  Use documented log rotation methods as much as possible ( The files in question though don't have OOTB options)

References  - 

OHS Segfault 11 Core Dumps ObLockFileRelease Webgate.so 5-7 Days (Doc ID 1985491.1)
http://oracleoam.blogspot.com/2014/07/lock-files-in-oam-11g-r2ps2_5.html

OAM : Oracle Traffic Director Licensing for Oracle Access Portal

Starting with Oracle Access Manager (OAM) 11.1.2.2.x , the license includes Oracle Access Portal Service.

What is Oracle Access Portal (OAP) ?
The Access Portal Service is a hosted single sign-on proxy service that enables intranet and extranet applications with Oracle's form-fill single sign-on technology. Web Logon Manager, available as a standalone download from Oracle Support, provides end-users with the ability to create, modify, and delete application credentials as well as log on to provisioned applications through both desktop and mobile browsers. Available from 11gR2 PS2.

What is Oracle Traffic Director(OTD)  ? 
Oracle Traffic Director is a fast, reliable, and scalable layer-7 software load balancer. The architecture of Oracle Traffic Director enables it to handle large volumes of application traffic with low latency. The product is optimized for use in Oracle Exalogic Elastic Cloud and Oracle SuperCluster.

OTD and OAP
For enabling the Oracle Access Portal Service, Oracle traffic Director (OTD) is mandatory as it intercepts user connections to the target application and provides path-proxy and DNS-proxy functionality, allowing for path and DNS rewriting.  It also hosts the WebGate plugin.

OTD Licensing for OAP
Though OTD is primarily licensed only for Exalogic , following are exceptions -

• The Oracle Traffic Director portion of the Oracle Access Portal is restricted to the following features: High Availability Virtual IP, Access Manager WebGate, and Origin Server Load Balancing to WebLogic Server.
• IDM Oracle Access Portal (OAP) license entitlement now includes OTD to be a front-end on Oracle Enterprise Linux 5.6+, Redhat Enterprise Linux 5.6+ and Solaris (SPARC, x64) 11.1+.
  Access Management Licensing - http://docs.oracle.com/cd/E29542_01/doc.1111/e14860/im_options.htm#FMWLC240

Reference :

http://docs.oracle.com/cd/E28280_01/doc.1111/e14860/im_options.htm#FMWLC239

Is Oracle Traffic Director (OTD) Available/Licensed to use for a Non Exalogic setup? (Doc ID 1500970.1)

OID : Deleting OID/OPMN instances

One can face weird issues due to OID Instances not being deleted the right way. Simply deleting them from the file system or from OID might not be enough as there are entries that stay on the ODS Schema of the Database.
This can cause unforseen issues like delay in OID restarts (ldapbind fails for few mins even though instance shows up as ALIVE) or weird issues like referential integrity working intermittenly in OID/OIM.

It is highly recommended to remove an Oracle Internet Directory component by using opmnctl deletecomponent. This also unregisters the component with the WebLogic server.

Syntax :

$ORACLE_INSTANCE/bin/opmnctl deletecomponent
  -adminHost webLogicHostName
  -adminPort webLogicPort
  -adminUsername weblogicAdminUsername
  -adminPasswordFile text_file_containing_admin_password
  -componentType OID
  -componentName componentName

Sample for removing an instance named oid4
export ORACLE_INSTANCE=/u01/app/oracle/admin/config/oid_inst_3/
$ORACLE_INSTANCE/bin/opmnctl deletecomponent \
  -adminHost <wl_server_Name> \
  -adminPort 7001 \
  -adminUsername weblogic \
  -adminPasswordFile adminpass.txt \
  -componentType OID \
  -componentName oid4

Reference - https://docs.oracle.com/cd/E16764_01/oid.1111/e10029/oid_server_instances.htm#BABDJABF