OHS : Periodic OHS/Web Server/WebGate Crash due to cron job incorrectly deleting *.lck files/httpd.pid files

Issue

We faced a very unique issue for one of our OAM Single Sign On implementations wherein , all the OHS Nodes in a cluster setup used to crash every 7th day generating core dumps running into dozens of GBs which potentially used to crash the OHS in addition to the downtime on Production systems.

Stack Trace on OHS/Webgates :

Loaded symbols for /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/libxmlengine.so
Core was generated by `/u01/app/orasec/middleware/Oracle_WT1/ohs/bin/httpd.worker -DSSL'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fd717f1161f in ObLockFileRelease(void*, bool) ()
  from /u01/app/orasec/middleware/Oracle_OAMWebGate1/webgate/ohs/lib/webgate.so
(gdb) (gdb)

Detailed Analysis of Root Cause 

On detailed debugging and some guidance from Oracle Support, we discovered that this was being caused by a Cron Job which was written to ensure oam_server.out files as well as oblog.log files get deleted every 7 days. This was due to the fact that Oracle doesn't provided log retention policies for these files OOTB.

The path that was used by the Cron Job was <MiddlewareHome>/<Oracle_WebTier>/instances/<instance_name>/diagnostics/logs/OHS/ohs1 which incidentally also hosted the important .lck files (polltracking.lck, oblog.log.lck, ObAccessClient.xml.lck)  and http.pid files [Why Oracle, Why ??!!]

Remember : Removing PID and *.lck files caused instability an is not supported by OAM or OHS.

Solution 

It is not supported to remove httpd.pid and *.lck or log files that are created by a running instance while it is running - 

1.  Setup up logging to another location where lock file and httpd.pid and other process files do not exist, if it's a cron job or something else is used to remove those files. In our case we explicitly called out the files which needed to be deleted instead of running the cronjob on a folder.

2.  Use documented log rotation methods as much as possible ( The files in question though don't have OOTB options)

References  - 

OHS Segfault 11 Core Dumps ObLockFileRelease Webgate.so 5-7 Days (Doc ID 1985491.1)
http://oracleoam.blogspot.com/2014/07/lock-files-in-oam-11g-r2ps2_5.html

OAM Single-Sign-On (SSO) Deployment Architecture : Best Practice

Recently I came across couple of OAM Deployment Architectures which have been implemented and can potentially cause multiple issues - 

• Using the same OHS Instance which has a webgate deployed on it for reverse proxy to OAM Servers in addition to the target application which needs to be protected (e.g. WebCenter)
• Front-ending OAM Servers directly with an external Load Balancer(LBR) skipping the Web Server layer altogether

Ideally, OAM should be front-ended by a web server/OHS instance of it's own to 

• Allow separate streams of HTTP traffic(in addition to one for Application) 
• Scale the SSO architecture to other target applications - 
• In case the same OAM Server is used for a new application which needs to be SSO enabled as well, the standalone OHS which just services requests to OAM (and doesn't have any webgate on it) is a must!
• This would also ensure that any files needed to be cached (like javascript, css etc) for any OAM related applications can be cached at the OHS layer

(Image Courtesy : A-Team Blog) 

If we have to use a Load Balancer(LBR) to directly front-end the OAM Server instead of an intermittent OHS(probably due to cost constraints), we should have this LBR within the corporate network (in addition to an external LBR which front-ends the other OHS instance(s) for applications) and not in the DMZ to prevent the security risk of an external LBR based in the DMZ exposing the OAM located in the Application Tier directly.

Courtesy  :
1) Forum Post which was logged as few items in the A-Team blog mentioned below were not crystal clear
2) A-Team Blog