How I Tracked Down a Crash Sequence Across EBS Application Logs, OPMN, and AIX OS Events
Alhamdulillah.
Around 3 AM, the production EBS application tier was in an unstable state. The application tier was experiencing repeated process crashes and restart failures, and the OPMN logs showed multiple processes entering a recovery cycle.
I start scrolling through logs and find the kind of chaos that makes troubleshooting interesting: OACORE processes that won’t stay alive, Forms instances crashing in sequence, OPMN repeatedly detecting failed JVM processes and attempting to restart them.
This is the story of how a single incident revealed the importance of timeline building, log correlation, and knowing the difference between recovery and root cause.
The Environment
• Oracle E-Business Suite R12.1
• AIX 7.1
• OC4J 10.1.3 with OPMN management
• IBM Java 7
• Multiple OACORE and Forms processes
The First Red Flag
I start with the OACORE process startup logs. The timeline shows a clean start:
20:57 — OACORE Startup 26/08/12 20:57:06 Start process
But three seconds later, something unexpected appears. The JVM is configured with 2560 MB heap and Gencon garbage collection. Then:
26/08/12 20:57:09 WARNING: GC: Unable to access pending References. 26/08/12 20:57:09 WARNING: GC: Unable to access pending Reference lock.
That’s interesting, but not necessarily fatal. GC warnings can be transient. I keep digging.
In the same timeframe, OACORE logs show an RMI failure:
Error: Could not connect to the remote server caused by: oracle.oc4j.security.KeyExchange.getSecretKey caused by: Unsupported secret key algorithm: DES
This is worth noting. An unsupported DES algorithm error in the OC4J RMI layer. But here’s the thing: I have two observations now, and I don’t yet know if they’re related, or if either one caused what happens next.
The Cascade
Fast forward to 01:26 AM on August 13. The OPMN logs suddenly light up:
01:26 — Multiple Forms Processes Crash 26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~2 26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~5 26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~7 26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~8 26/08/13 01:26:07 Process Crashed: default_group~forms-c4ws~default_group~1
Not just one. Not even two. Five processes are crashing in the same second. The simultaneous failure of multiple Forms-related processes suggested that the issue was broader than a single application process failure.
OPMN immediately begins detecting these failures. It does what it’s designed to do: attempt to restart failed processes. But the restart attempts are not succeeding cleanly:
Process Start Error: default_group~forms~default_group~4 Failed to construct stop command Forcefully Terminating Process Could not connect to socket while pinging an OC4J process Operation: internal (Connection refused)
OPMN detects the failed processes and repeatedly attempts to restart them, but several restart attempts encounter process communication, socket, and port-information errors. This is the kind of situation where the logs quickly become a wall of repeated error messages as OPMN continues attempting to recover the affected processes.
The Port Information Mystery
As I dig deeper into the OPMN logs, I find another clue that doesn’t quite fit the picture yet:
Host and Port information for port type jms not sent by the OC4J process WARNING: OC4J did not send the protocol value for port id: jms
OPMN relies on port information to manage processes. OPMN was not receiving the expected JMS port/protocol information from the OC4J process, which complicated process management and restart attempts.
At this point I have:
• GC warnings on OACORE startup
• An RMI/DES algorithm error
• Multiple OC4J processes crashing simultaneously
• OPMN unable to restart them cleanly
• Missing JMS port/protocol information
But which one caused the others? That’s when I make an important decision: instead of trying to decide, I document what I observe and build a timeline.
Build a Timeline, Not a Theory
When several application processes fail around the same period, it is tempting to take the first recognizable error and declare it the root cause. In this incident, the OACORE GC warnings and OC4J DES error were early clues, but the subsequent OPMN logs showed that the failure was broader.
At this point, I have a comprehensive picture of application failures. But I’ve been told the server was rebooted. So I check the AIX logs.
Incident Timeline
| Time | Observation |
|---|---|
| Aug 12, 20:57 | OACORE processes started |
| 20:57:09 | IBM JVM GC warnings observed (Unable to access pending References) |
| 20:57 | OC4J RMI connection failure — Unsupported secret key algorithm: DES |
| Aug 13, 01:26:07 | Multiple Forms / Forms-C4WS processes crashed simultaneously |
| 01:28 onward | OPMN restart attempts encountered process start, socket, and port-information errors |
| 02:36 | AIX system shutdown recorded; system subsequently rebooted |
| After reboot | Application services recovered and remained stable |
The OS Logs Tell a Different Story
Now I verify the OS timeline. First, boot information:
who -b
This tells me the boot time was Aug 13 at 02:36. Next I check the AIX error log with a time-window query around the known failure period:
errpt -s 0813010026 -e 0813030026
This returns:
2BFA76F6 0813023626 T S SYSPROC SYSTEM SHUTDOWN BY USER
Confirmed: SYSTEM SHUTDOWN BY USER at 02:36.
02:36 — AIX Reboot System boot time: Aug 13 02:36
After the reboot, the application services came back up and remained stable.
What the Logs Proved — and What They Didn’t
Proved by the evidence:
• Multiple EBS application processes crashed
• OPMN attempted recovery and restarts
• Several restart attempts encountered communication and port-related errors
• AIX recorded a system shutdown by user at 02:36
• who -b confirmed the system boot time as 02:36
• Application services recovered after the reboot
Not proved by the evidence:
• That the GC warnings caused the incident
• That the DES error caused the process crashes
• That the JMS port issue was the initiating failure
• That the reboot fixed the underlying root cause
• The exact initiating cause of the overall process-crash cascade
The Critical Distinction
Here’s where I have to be careful about my conclusions.
The reboot restored service. That’s a fact. But did the reboot fix the root cause, or did it just clear transient state? These are two different things.
A reboot can:
• Clear unstable JVM runtime state
• Terminate stuck processes
• Clear transient socket/process communication state
• Reinitialize application-tier processes and their associated listeners/ports
None of those tell me what originally triggered the failure.
What I can confirm:
• OACORE: Startup with GC warnings
• OC4J RMI: DES algorithm error
• Forms and Forms-C4WS: Multiple process crashes during the recovery/restart sequence
• OPMN: Multiple restart attempts encountered process communication failures, connection refusals, and missing JMS port/protocol information
• Recovery: Reboot restored stability
What I cannot conclusively determine from logs: the exact initiating cause of the cascade.
When to Raise an Oracle Support SR
In production, your first priority is recovery. But recovery should not be your last action.
If you encounter OC4J/OPMN instability involving multiple process crashes and restart failures, here’s the right approach:
Before you reboot:
• If service impact is acceptable, preserve application logs, OPMN logs, and OS error logs (errpt)
• Consider opening an SR with Oracle Support before recovery actions
• Provide an SR reference for all follow-up communication
If service impact demands immediate recovery:
• Proceed with the reboot
• Collect and preserve log artifacts afterward
• Raise the SR as soon as practical with the preserved log artifacts
• Include full OPMN, OACORE, Forms logs and AIX errpt output
An SR with Oracle Support positions your incident for definitive analysis. It moves you from “we rebooted and it worked” to “here’s what happened, and here’s what Oracle Support says about it.”
The Troubleshooting Checklist
When you see OC4J instability, use these commands to build your timeline:
Grep for key patterns across logs:
grep -n -E "GC: Unable|Unsupported secret|Process Start Error|Process Crashed|Connection refused|Exception|ERROR|FATAL" <oacore_log>
Check OPMN for the restart cycle:
grep -n -E "Process Crashed|Process Start Error|Connection refused|Port information" <opmn.log>
Verify boot timeline and OS events:
who -b && uptime && last reboot | head -5
Extract the error events in a time window (the most useful approach):
errpt -s 0813010026 -e 0813030026 # Adjust dates for your incident
This time-window query is much more powerful than a simple grep because it shows you events in context, within the period your incident occurred.
What I Learned
Alhamdulillah for this incident, because it reinforced several critical principles:
The first error is not the root cause. I started with GC warnings and an RMI error. But the real story emerged from the sequence of crashes, not from the opening symptoms.
Build a timeline, not a theory. A timeline shows sequence and correlation; theories can introduce assumptions. Timestamps tied together the OACORE observations, the Forms crashes, OPMN’s restart failures, and ultimately the reboot recovery.
Recovery and RCA are different questions. The reboot answered “what restored service?” but not “what caused the failure?” It’s important to say the difference out loud.
Correlate application and OS logs. OPMN logs showed application behavior; AIX errpt showed OS timeline. Together they tell the full story.
Open an SR when multiple OC4J components fail. When you have a cascade of crashes involving Forms, OACORE, and OPMN restart failures, that’s beyond the scope of “try restarting the services.” Engage Oracle Support early with an SR. It positions you for definitive analysis and creates a record for future reference.
Closing Thoughts
This incident reinforced something I’ve learned through years of production troubleshooting in EBS and Oracle Database: patience with logs pays off.
The reboot restored application stability, but the available evidence did not establish whether the reboot cleared a transient runtime condition or whether the underlying initiating problem had already resolved.
Instead of manufacturing an RCA that the logs don’t support, I documented: application symptoms → OPMN behavior → OS timeline → reboot → recovery → RCA limitation.
For any DBA handling production incidents, this is the framework:
1. Document what the evidence proves
2. Clearly identify what it does not prove
3. Continue the investigation with Oracle Support if RCA is incomplete
ما شاء الله
Leave a comment