OHS : Periodic OHS/Web Server/WebGate Crash due to cron job incorrectly deleting *.lck files/httpd.pid files

Issue

We faced a very unique issue for one of our OAM Single Sign On implementations wherein , all the OHS Nodes in a cluster setup used to crash every 7th day generating core dumps running into dozens of GBs which potentially used to crash the OHS in addition to the downtime on Production systems.

Stack Trace on OHS/Webgates :

Loaded symbols for /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/libxmlengine.so
Core was generated by `/u01/app/orasec/middleware/Oracle_WT1/ohs/bin/httpd.worker -DSSL'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fd717f1161f in ObLockFileRelease(void*, bool) ()
  from /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/webgate.so
(gdb) (gdb)

Detailed Analysis of Root Cause 

On detailed debugging and some guidance from Oracle Support, we discovered that this was being caused by a Cron Job which was written to ensure oam_server.out files as well as oblog.log files get deleted every 7 days. This was due to the fact that Oracle doesn't provided log retention policies for these files OOTB.

The path that was used by the Cron Job was <MiddlewareHome>/<Oracle_WebTier>/instances/<instance_name>/diagnostics/logs/OHS/ohs1 which incidentally also hosted the important .lck files (polltracking.lck, oblog.log.lck, ObAccessClient.xml.lck)  and http.pid files [Why Oracle, Why ??!!]

Remember : Removing PID and *.lck files caused instability an is not supported by OAM or OHS.

Solution 

It is not supported to remove httpd.pid and *.lck or log files that are created by a running instance while it is running - 

1.  Setup up logging to another location where lock file and httpd.pid and other process files do not exist, if it's a cron job or something else is used to remove those files. In our case we explicitly called out the files which needed to be deleted instead of running the cronjob on a folder.

2.  Use documented log rotation methods as much as possible ( The files in question though don't have OOTB options)

References  - 

OHS Segfault 11 Core Dumps ObLockFileRelease Webgate.so 5-7 Days (Doc ID 1985491.1)
http://oracleoam.blogspot.com/2014/07/lock-files-in-oam-11g-r2ps2_5.html

OAM Single-Sign-On (SSO) Deployment Architecture : Best Practice

Recently I came across couple of OAM Deployment Architectures which have been implemented and can potentially cause multiple issues - 

• Using the same OHS Instance which has a webgate deployed on it for reverse proxy to OAM Servers in addition to the target application which needs to be protected (e.g. WebCenter)
• Front-ending OAM Servers directly with an external Load Balancer(LBR) skipping the Web Server layer altogether

Ideally, OAM should be front-ended by a web server/OHS instance of it's own to 

• Allow separate streams of HTTP traffic(in addition to one for Application) 
• Scale the SSO architecture to other target applications - 
• In case the same OAM Server is used for a new application which needs to be SSO enabled as well, the standalone OHS which just services requests to OAM (and doesn't have any webgate on it) is a must!
• This would also ensure that any files needed to be cached (like javascript, css etc) for any OAM related applications can be cached at the OHS layer

(Image Courtesy : A-Team Blog) 

If we have to use a Load Balancer(LBR) to directly front-end the OAM Server instead of an intermittent OHS(probably due to cost constraints), we should have this LBR within the corporate network (in addition to an external LBR which front-ends the other OHS instance(s) for applications) and not in the DMZ to prevent the security risk of an external LBR based in the DMZ exposing the OAM located in the Application Tier directly.

Courtesy  :
1) Forum Post which was logged as few items in the A-Team blog mentioned below were not crystal clear
2) A-Team Blog

OAM 11gR2 / 11.1.2.2 : Redirection to favicon.ico screen after authentication to a Portal

Scenario - After login to the Portal using OAM 11gR2PS2 login to access a protected resource for the first time (page not cached in browser), the end user is redirected to the favicon.ico url instead of the resource url. The browser needs to be refreshed with the Portal URL to get this to navigate to the Portal home page as expected.

Also if the favicon.ico is not present in the OHS /htdocs folder a 404 may also appear.

It was initially tricky to figure out that this issue was related to permissions in OAM. But if we remove the webgate entry from httpd.conf (i.e. Security layer is bypassed), we can confirm that this issue doesnot occur which means OAM is playing up here.

Cause -
This is caused by the favicon.ico being protected by OAM. 
If it's not in the browser cache, the client browser will fetch the favicon.ico resource on the server. It will get the favicon before the page, setting incorrectly the end_url parameter, redirecting to the favicon url instead of the resource url.

Solution -
Make sure that favicon.ico is created in <OHS_INSTANCE>/config/OHS/ohs1/htdocs/  folder so that there is no 404 ever. This is the icon which shows besides the website address.

Then, create a resource definition in OAM as documented in http://docs.oracle.com/cd/E27559_01/admin.1112/e27239/app_domn.htm#CACJJGCA 
1) set type as HTTP
2) define resource URL to /favicon.ico
3) set protection level to "excluded"  (More about excluded resources here )

Reference - Oracle Support Document 

OAM 11gR2/Weblogic : The important of parameters in mod_wl_ohs.conf(Web Server plugins)

Configurations of various parameters in web server plugins plays in a major part in ensuring that Single-Sign-On works fine using OAM.

Oracle Documentation -
http://docs.oracle.com/cd/E23943_01/web.1111/e14395/plugin_params.htm

This post is intended to share my experiences with certain parameters and the repercussions if you don't include them :)

WLProxyPassThrough
WLProxySSl works great if webserver is doing the SSL work. But if SSL being terminated by a load balancer then mod_wl will remove any incoming WL-Proxy-SSL and the request will reach OHS over HTTP this means that the WebLogic server won't ever get that header and so request.isSecure() will always return false. If you add that directive and set it to ON then the WebLogic plug-in will not remove any incoming WL-Proxy-SSL header. This lets WebLogic Server know that the original request was initiated over SSL.  WL-Proxy-SSL header should not be sent if the inbound traffic to the load balancer was not SSL (HTTPS).

Error Scenario : 
Once I added this parameter for under the <if weblogic_module> tag and set it to true, this issue no longer reccurred .




WLCookieName
If you change the name of the WebLogic Server session cookie in the WebLogic Server Web application, you need to change the WLCookieName parameter in the plug-in to the same value. The name of the WebLogic session cookie is set in the WebLogic-specific deployment descriptor, in the <session-descriptor> element.

Error Scenario :
The Webcenter Portal application I was implementing SSO using OAM for, had changed the weblogic session cookie name to a non-JSESSION ID value for some reason.
This was not giving me any issues until I was configuring "Weblogic Cluster" value(instead of "Weblogic Host") in the OHS layer pointing to the Webcenter managed servers.Once I did so,the Webcenter Portal page would not load and instead would give me a flickering page with consistently changing values of adf_ctrl.state and the page would not load up.
This issue was resolved once I added WLCookieName <cookieName> under the context root tag for the Webcenter Portal app in mod_wl_ohs.conf

This post is also relevant in this regard.

Allowing unauthenticated access to Webcenter Content/UCM public documents via OAM SSO

Recently we had a requirement at a client wherein Public Documents in UCM be accessed via a SSO URL (using the OHS Port) without the user being challenged for credentials.

Sounds pretty straighforward right ? Since anyways using the default managed server port of 16200 of the Content Server , anyways those documents don't popup asking for user credentials.

Well , wasn't that simple really! Took us(myself , Sachin Saxena et al) a few days to exactly figure this out and now we have Oracle's stamp on it as well !

Following were the examples of documents which needed to be publically accessible

1)http://<host>:7778/cs/idcplg?idcService=GET_FILE&dID=1445&dDocName=DEV_COMPLOGO_31364&allowInterrupt=1

(Accessing public document  called 'DEV_COMPLOGO_31364' having Public Security group via IdcService)

2)http://<host>:7778/cs/groups/public/documents/digitalmedia/b2dv/xzmx/~edisp/dev_complogo_31364.jpg

(This has a definite URL Pattern of /cs/groups/public)

3)Also this general service/document search page needs to open up without authentication.

http://<host>:7778/cs/idcplg?IdcService=GET_DOC_PAGE

**Oracle Access Manager (OAM) basically protects URLs or definite URL parameters , it cannot go inside an end-user application and check security assigned to a resource to determine if that should be challenged for credentials.

(Example in this case is that it cannot go and check the authorization / security Group using IdcService URL for the file DEV_COMPLOG_31364)**

Hence we have only two options here -

1)Either the URL pattern /cs/groups/public can be marked as unprotected in OAM Application Domain and hence user won't be challenged when using Pattern 2 as above.

2)Create a mapping folder pattern in UCM , like pretty URL to access even webDav content)

3)You can configure the IdcService url pattern to be public via OAM. (something like http://oamserver.com/cs/idcplg as url and query parameters as IdcService=GET_FILE) .But by exposing that people can still construct url of private documents if they know the dID and bypass OAM as the pattern is public. But once they reach UCM, ucm security will deny them access as they are not authenticated/have required permissions [Courtesy : Shidharth Mishra]

References

1)Oracle SR [Closed]            2) Forums