Syed Anwar Ahmed – Oracle DBA Blog

Tag: ebs

  • Oracle EBS R12.2 Clone Troubleshooting: RC-50221 Port Pool Conflict and FRM-92101 Forms Session Failed During Startup

    Disclaimer: This article is based on field experience and publicly shareable troubleshooting steps. Oracle MOS notes are referenced for further reading. Customers with active Oracle Support should review the referenced MOS documents for complete guidance.

    Category: Oracle EBS R12.2 | Cloning | Troubleshooting
    Level: Intermediate to Advanced
    Reference: MOS Doc ID 454427.1

    Introduction

    Cloning an Oracle E-Business Suite R12.2 application tier is a common activity — whether for setting up a DEV, TEST, or UAT environment. While the process is well-documented, real-world scenarios often throw up issues that aren’t immediately obvious from the error messages alone.

    In this post, I walk through two common issues encountered during an EBS R12.2 app tier clone using adcfgclone.pl:

    1. RC-50221: Port Pool Not Free — CloneContext failing due to an already-occupied port pool
    2. FRM-92101: Forms Session Failed During Startup — Forms not launching due to missing library dependencies or invalid symlinks

    Both issues are real, well-known, and worth documenting together since they often appear in sequence during a clone.

    Environment Details

    • Oracle EBS Version: R12.2.x
    • OS: Oracle Linux 7 (OL7) / RHEL 7
    • Clone type: Application Tier only (adcfgclone.pl appsTier)
    • Target server: Fresh clone target (non-production)

    Issue 1: RC-50221 — Port Pool Not Free

    Symptom

    When running adcfgclone.pl appsTier, the CloneContext step prompts for a Target System Port Pool and throws the following warning/error:

    Checking the port pool 0
RC-50221: Warning: Port Pool 0 is not free.
Please check logfile $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log for conflicts.
RC-00201: Error: Not a valid port pool number
ERROR: Context creation not completed successfully.

    CloneContext Log Location

    $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log
# Or typically:
/u01/APPLTOP/EBSapps/comn/clone/bin/CloneContext_<timestamp>.log

    Root Cause

    Port pool 0 is already in use by another EBS instance running on the same server, causing CloneContext validation to fail. Port pool 0 generally corresponds to the initially configured EBS port set for that environment. Actual port assignments vary by release, topology, and previous configuration changes.

    ps -ef | grep weblogic | grep AdminServer
lsof -i -P | grep LISTEN | grep <applcrp_user>

    Diagnosis

    Check which port pools are already in use:

    grep -i "port_pool" $CONTEXT_FILE
cat $INST_TOP/admin/out/portpool.lst

    You can also use the following utility to validate port availability before proceeding with the clone:

    perl $AD_TOP/bin/txkValidateRollupPorts.pl \
-contextfile=$CONTEXT_FILE \
-outfile=/tmp/portcheck.txt

    Review /tmp/portcheck.txt for any conflicts before selecting a port pool number.

    Fix

    Target System Port Pool [0-99] : 1
Checking the port pool 1
done: Port Pool 1 is free

    Important Note: Using a different port pool causes Oracle EBS to calculate a different set of ports based on the predefined port pool mapping. The resulting ports are not always a simple increment of one and should be verified in the generated context file after CloneContext completes.

    Always verify the actual assigned ports:
    grep -i "port" $CONTEXT_FILE | grep -v "#"
    cat $INST_TOP/admin/out/portpool.lst

    If your firewall rules, load balancer, or application configuration depend on specific port numbers, plan accordingly or stop the existing instance first to free pool 0.

    Patch File System Port Pool

    For R12.2 dual file system architecture, the Patch file system must use a different port pool than the Run file system. AutoConfig and cloning utilities will calculate a separate set of ports accordingly.

    Patch file system should have different port pool than Run file system.
Target System Port Pool [0-99] : 2

    Issue 2: FRM-92101 — Forms Session Failed During Startup

    Symptom

    FRM-92101: There was a failure in the Forms Server during startup.
Please look into the web-server log file for details.
Java Exception:
oracle.forms.net.ConnectionException: Forms session <1> failed during startup:
no response from runtime process

    What This Means

    The WebLogic Forms server (forms_server1) is running, but the underlying Forms runtime process (frmweb) is failing to start. WebLogic receives no response from it, and the session times out. FRM-92101 can have more than one root cause — the key is to run frmweb directly to identify which one applies.

    Log Analysis

    tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    This indicates that the Forms managed server is up and processing requests, shifting the investigation toward the underlying frmweb runtime process.

    First Diagnostic Step: Run frmweb Directly

    The most effective way to identify the root cause is to source the 10.1.2 environment and run frmweb manually:

    . /u01/APPLTOP/EBSapps/10.1.2/EBS_appserver.env
$ORACLE_HOME/bin/frmweb

    The output will indicate which root cause applies:

    • Undefined symbol errors (e.g. Symbol nnftboot was referenced...not found) → Root Cause 1: Invalid ldflags symlink
    • Missing shared library error (e.g. libXm.so.2: cannot open shared object file) → Root Cause 2: Missing OpenMotif symlink

    Root Cause 1: Invalid ldflags Symlink

    Symptom

    Running frmweb directly produces undefined symbol errors:

    rtld: 0712-001 Symbol nnftboot was referenced from module frmweb(), but a runtime definition of the symbol was not found.
rtld: 0712-001 Symbol ldap_search_s was referenced from module frmweb(), but a runtime definition of the symbol was not found.

    Root Cause

    Investigation revealed that the ldflags symbolic link under $ORACLE_HOME/lib32 was missing or pointing to an incorrect location, causing the Forms executable to fail during startup. This is a known issue after fresh installations or clones and is referenced in MOS Doc ID 454427.1.

    Verify ldflags

    ls -la $ORACLE_HOME/lib32/ldflags

    If the symlink is missing or pointing to a non-existent path, it needs to be corrected to point to $ORACLE_HOME/lib/ldflags.

    Fix

    After correcting the ldflags symbolic link and relinking the Forms components, frmweb started successfully. The relinking is performed using the Forms makefile under the 10.1.2 Oracle Home. Refer to MOS Doc ID 454427.1 for the complete steps applicable to your environment.

    Root Cause 2: Missing OpenMotif Library Symlink

    Symptom

    Running frmweb directly produces a missing shared library error:

    error while loading shared libraries:
libXm.so.2: cannot open shared object file: No such file or directory

    Root Cause

    The Oracle Forms runtime binary frmweb is compiled against libXm.so.2 (OpenMotif 2.x). On newer Linux versions (OL7/OL8/RHEL7/RHEL8), the installed OpenMotif package provides libXm.so.4, but the compatibility symlink libXm.so.2 → libXm.so.4 is missing.

    Verify with ldd

    ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Before fix:
libXm.so.2 => not found
# After fix:
libXm.so.2 => /usr/lib/libXm.so.4 (0x...)

    Option A: Create the Missing Symlink (Quick Fix)

    📌 Note on 32-bit vs 64-bit Libraries: On some Linux platforms the required library may reside under /usr/lib64 rather than /usr/lib. Always verify the actual location before creating symbolic links.

    # Step 1: Find the actual location of libXm.so.4
ls -l /usr/lib/libXm.so*
ls -l /usr/lib64/libXm.so*
# Step 2: Create the symlink (run as root)
# If library is in /usr/lib:
ln -s /usr/lib/libXm.so.4 /usr/lib/libXm.so.2
# If library is in /usr/lib64:
ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.2
# Step 3: Verify
ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Expected: libXm.so.2 => /usr/lib/libXm.so.4 (or /usr/lib64/...)

    Option B: Install openmotif21 (Recommended Clean Fix)

    # Step 1: Check current state
rpm -qa | grep motif
# Step 2: Enable ol7_addons repo (as root)
yum-config-manager --enable ol7_addons
# Step 3: Remove lesstif if present
yum remove lesstif
# Step 4: Install openmotif21
yum install --setopt=obsoletes=0 openmotif21

    Post-Fix: Bounce EBS Services and Validate

    cd $ADMIN_SCRIPTS_HOME
./adstpall.sh apps/<apps_password>
./adstrtal.sh apps/<apps_password>
# Or start Forms server specifically:
./admanagedsrvctl.sh start forms_server1

    After starting the services, verify that frmweb runtime processes are spawning successfully:

    ps -ef | grep frmweb
ps -ef | grep forms

    Then launch a Forms-based responsibility or function to confirm successful startup.

    Conclusion

    Both RC-50221 and FRM-92101 are common issues that can delay Oracle EBS R12.2 cloning activities. While the error messages may initially appear unrelated, a structured troubleshooting approach focusing on log analysis, port validation, and runtime dependency checks can quickly identify the root cause and reduce downtime.

    Summary

    IssueRoot CauseFix
    RC-50221: Port Pool Not FreeAnother EBS instance occupying port pool 0Use port pool 1 (or stop existing instance first)
    FRM-92101: ldflags issueldflags symlink missing or pointing to wrong locationFix ldflags symlink and relink Forms components (MOS Doc ID 454427.1)
    FRM-92101: OpenMotif issuelibXm.so.2 symlink missing after cloneln -s /usr/lib[64]/libXm.so.4 /usr/lib[64]/libXm.so.2 or install openmotif21

    Key Diagnostic Commands Cheat Sheet

    # Check which port pools are in use
grep -i "port_pool" $CONTEXT_FILE
cat $INST_TOP/admin/out/portpool.lst
# Validate port availability
perl $AD_TOP/bin/txkValidateRollupPorts.pl -contextfile=$CONTEXT_FILE -outfile=/tmp/portcheck.txt
# Check running WebLogic processes
ps -ef | grep weblogic | grep -v grep
# Run frmweb manually to see actual error
$ORACLE_HOME/bin/frmweb
# Check frmweb library dependencies
ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Check ldflags symlink
ls -la $ORACLE_HOME/lib32/ldflags
# Check OpenMotif installation
rpm -qa | grep motif
# Check Forms runtime processes
ps -ef | grep frmweb
# Check Forms server log
tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    Lessons Learned

    • FRM-92101 is often a symptom, not the root cause. The WebLogic log may show the Java exception, but the actual failure is in the native frmweb binary. Always run frmweb directly to see the real error.
    • FRM-92101 has multiple root causes. Check the actual error output from frmweb to distinguish between an ldflags relinking issue and a missing OpenMotif library symlink.
    • Use ldd to diagnose missing shared library dependencies. It quickly identifies unresolved libraries without having to dig through multiple log files.
    • Verify port pool availability before starting a clone. Stop existing instances or choose a different pool to avoid RC-50221 errors.
    • Always validate the resulting ports in the generated context file after CloneContext completes. Port pool mapping is predefined and the assigned ports should be confirmed before proceeding.
    • Newer Linux builds may include libXm.so.4 but not the compatibility symlink required by Oracle Forms. OL7/OL8 installations do not always create libXm.so.2 automatically.
    • Document port assignments after every clone to avoid firewall and load balancer configuration issues down the line.

    Keywords

    • Oracle EBS R12.2 Clone
    • Oracle EBS R12.2 Forms Error
    • RC-50221 Port Pool Not Free
    • RC-50221 CloneContext Failed
    • FRM-92101 Forms Session Failed During Startup
    • FRM-92101 No Response From Runtime Process
    • ldflags symlink Oracle Forms
    • libXm.so.2 not found
    • Oracle Forms Runtime Process
    • adcfgclone.pl appsTier
    • adcfgclone.pl appsTier Error
    • Oracle EBS Clone Troubleshooting
    • Oracle Linux 7 OpenMotif Issue
    • OpenMotif
    • frmweb
    • CloneContext

    References

    • MOS Doc ID 454427.1 — FRM-92101: There was a failure in the Forms Server during startup
    • Oracle EBS R12.2 Cloning Guide

    Related Technologies

    • Oracle E-Business Suite R12.2
    • Oracle Forms
    • Oracle WebLogic Server
    • Oracle Linux 7
    • OpenMotif
    • AutoConfig
    • Rapid Clone

    Syed Anwar Ahmed
    Oracle ACE Associate | Oracle Apps DBA
    Sharing real-world Oracle EBS administration, cloning, patching, and troubleshooting experiences.
    Blog: syedanwarahmedoracle.blog

  • Fixing Oracle EBS 12.2 Forms Not Opening (Step-by-Step Guide)

    Oracle E-Business Suite (EBS) 12.2 uses Java Web Start (JNLP) to launch forms, unlike older versions like 11i which relied on browser plugins. If your forms are not opening, this guide will help you fix the issue completely.

    Common Symptoms

    • Forms not opening after click
    • .jnlp file downloads but does nothing
    • Application Blocked by Java Security error
    • Nothing happens when clicking forms

    Root Causes

    • Using unsupported Java version (Java 11, 17, or 24)
    • Missing Java Web Start (OpenWebStart)
    • Java security blocking self-signed certificates
    • Old EBS 11i configurations applied to 12.2 environment

    Step-by-Step Solution

    Step 1: Remove Unsupported Java Versions

    Uninstall Java 11, 17, 24 or any non-Java 8 version from Control Panel → Programs → Uninstall a Program.

    Verify removal:

    java -version

    Expected output:

    'java' is not recognized as an internal or external command

    Also clean up residual folders if they exist:

    C:\Program Files\Java\
    C:\Program Files (x86)\Java\
    C:\Users\<username>\AppData\LocalLow\Sun\Java\

    Step 2: Install Java 8

    Download and install Java 8 JRE (1.8.0_xxx) from Oracle’s website. Verify after installation:

    java -version

    Expected output:

    java version "1.8.0_xxx"

    Step 3: Install OpenWebStart

    Oracle EBS 12.2 requires Java Web Start to launch .jnlp files. Since modern Java versions removed the built-in Web Start, install OpenWebStart as a replacement. Download from: https://openwebstart.com

    Step 4: Configure Java Security

    Go to Control Panel → Java → Security → Edit Site List and add your EBS URLs:

    http://your-ebs-url
    http://your-ebs-url:port
    https://your-ebs-url:port

    Step 5: Clear Java Cache

    Go to Java Control Panel → General → Delete Files, select all options, and click OK.

    Step 6: Associate JNLP Files with OpenWebStart

    • Right-click any .jnlp file
    • Select Open with → Choose another app
    • Select OpenWebStart
    • Check Always use this app

    Fix for “Application Blocked by Java Security”

    If you see the Application Blocked by Java Security error, add your EBS URL to the Exception Site List as described in Step 4. This allows self-signed certificates to run without being blocked.

    Expected Behavior After Configuration

    1. Login to EBS 12.2
    2. Click a Form-based responsibility
    3. .jnlp file launches via OpenWebStart
    4. Oracle Form opens successfully

    Important Notes

    • Do not use Java 7 — that is only required for EBS 11i
    • IE Mode in Edge is not required for EBS 12.2
    • Always use the Java 8 + OpenWebStart combination
    • Ensure popups and file downloads are allowed in your browser

    Summary

    ComponentRequirement
    JavaJava 8 (required)
    LauncherOpenWebStart (required)
    Browser PluginNot required
    IE ModeNot required

    Conclusion

    Most EBS 12.2 Forms issues come down to incorrect Java setup or missing OpenWebStart. If you previously configured your machine for EBS 11i, fully revert those settings before applying the 12.2 configuration — conflicting setups are one of the most common causes of Forms not opening.

    — Syed Anwar Ahmed | Oracle Apps DBA | LinkedIn

  • Rethinking UNDO Tablespace Monitoring in Multitenant Environments: A Case Study on APPS_UNDOTS1

    Introduction

    A production alert indicating UNDO tablespace usage at ~99% is typically treated as a critical issue. However, in Oracle Multitenant environments with local UNDO enabled, this metric can be misleading when evaluated using traditional tablespace monitoring techniques.

    This article presents a real-world scenario involving the PDB-level UNDO tablespace APPS_UNDOTS1, where utilization reached ~99%, yet no actual resource contention existed.

    The Incident

    An alert was triggered showing:

    Tablespace              TOTAL_MB     USED_MB     FREE_MB PCT_USED
    -------------------- ----------- ----------- ----------- --------
    APPS_UNDOTS1           28,863.25   28,681.56      181.69    99.37

    At first glance, this indicated:

    • UNDO nearly full
    • Minimal free space
    • Risk of ORA-30036 / ORA-01555

    Environment Context

    This issue occurred in a Multitenant environment with Local UNDO enabled:

    • UNDO tablespace: APPS_UNDOTS1
    • Scope: PDB-level
    • Each PDB maintains its own UNDO

    UNDO behavior must always be analyzed at the PDB level, not just at the CDB level.

    Investigation Approach

    Rather than relying on % used, a state-based analysis was performed.

    Step 1: Active Transactions

    SELECT COUNT(*) FROM v$transaction;

    Observation:

    • Minimal active transactions
    • No ongoing workload pressure

    Step 2: UNDO Extent Analysis

    SELECT status, SUM(bytes)/1024/1024 MB
    FROM dba_undo_extents
    GROUP BY status;

    Key Observation

    STATUS        MB         PCT
    -----------   --------   -------
    ACTIVE        ~3 MB      ~0.01%
    UNEXPIRED     ~51 MB     ~0.18%
    EXPIRED       ~28 GB     ~99.81%

    💡 Critical Insight

    ~99.81% of UNDO (APPS_UNDOTS1) was EXPIRED and fully reusable

    This means:

    • No real space pressure
    • No transaction risk
    • Tablespace is effectively available

    Understanding UNDO Behavior

    UNDO extents follow a lifecycle:

    ACTIVE → UNEXPIRED → EXPIRED → REUSED
    StateDescriptionReusable
    ACTIVEUsed by active transactions❌
    UNEXPIREDRetained for consistency⚠️
    EXPIREDNo longer needed✅

    UNDO extents are not physically freed but logically reused by Oracle based on demand.

    Why the Alert Was Misleading

    The alert was based on:

    SELECT * FROM dba_free_space;

    Limitation

    • Shows only physically free space
    • Does not reflect reusable UNDO
    • Ignores Oracle’s internal reuse mechanism

    Root Cause

    A long-running concurrent request:

    • Ran for ~30 hours
    • Performed heavy DELETE operations
    • Generated significant UNDO

    During execution:

    • UNDO reached ~99% → real pressure

    After completion:

    • UNDO became EXPIRED
    • Space became reusable
    • Alert persisted → false positive

    When UNDO Is Actually a Problem

    ACTIVE      > 20–30%
    UNEXPIRED   very high
    EXPIRED     very low

    UNDO becomes a real issue only when both EXPIRED and UNEXPIRED extents are exhausted and no reusable space remains.

    Recommended Monitoring Approach

    WITH undo AS (
        SELECT status, SUM(bytes)/1024/1024 mb
        FROM dba_undo_extents
        GROUP BY status
    ),
    total AS (
        SELECT SUM(mb) total_mb FROM undo
    )
    SELECT 
        u.status,
        ROUND(u.mb,2) AS mb,
        ROUND((u.mb / t.total_mb) * 100, 2) AS pct
    FROM undo u, total t;

    Improved Alert Logic

    • ACTIVE > 20% → Critical
    • UNEXPIRED > 80% → Warning
    • Otherwise → Normal

    Key Takeaways

    • UNDO is state-driven, not space-driven
    • EXPIRED undo = reusable capacity
    • % used is misleading
    • Always analyze at PDB level in multitenant setups
    • Monitoring must align with Oracle internals

    Conclusion

    UNDO tablespace monitoring must evolve from:

    ❌ Static space-based metrics
    ➡️
    ✅ Dynamic lifecycle-based analysis

    A tablespace showing 99% utilization can still be completely healthy if most of its extents are reusable.

    Level Insight

    “UNDO is not a storage problem — it is a lifecycle problem. The difference between false alarms and accurate diagnosis lies in understanding how Oracle transitions undo extents.”

    Final Thought

    👉 Don’t ask:
    “How full is UNDO?”

    👉 Ask:
    “How much of it is actually in use?”

    Author

    Syed Anwar Ahmed
    Oracle Apps DBA | Oracle EBS | Performance & Troubleshooting

  • When ADOP Remembers Too Much: Fixing Patch Failures Caused by Stale Metadata in Oracle EBS

    During an Oracle E-Business Suite ADOP patching cycle in a multi-node environment, the apply phase failed on one node while completing successfully on others. Despite retries — including downtime mode — the issue persisted, pointing to a deeper inconsistency within the patching framework.

    Symptoms Observed

    • ADOP session status: FAILED
    • Patch applied successfully on some nodes, failed on admin node
    • Repeated failures even with restart=no, abandon=yes, and downtime mode
    • No immediate actionable error from standard logs

    Timeline of Events

    T0 -- Patch execution initiated (ADOP apply phase)
T1 -- Failure observed on admin node
T2 -- Retry using downtime mode -- Failure persists
T3 -- ADOP session review shows inconsistent state
T4 -- Internal metadata tables analyzed
T5 -- Cleanup performed (tables + restart directory)
T6 -- Patch re-executed -- Success across all nodes

    Investigation

    Step 1: Check ADOP Session State

    Query the ADOP session status to understand the current state across all nodes:

    -- Check current ADOP session status
SELECT session_id, node_name, phase, status,
       start_date, end_date
FROM applsys.ad_adop_sessions
ORDER BY start_date DESC;

-- Check apply phase status per node
SELECT s.session_id, n.node_name, p.phase_code,
       p.status, p.start_date, p.end_date
FROM applsys.ad_adop_sessions s,
     applsys.ad_adop_session_phases p,
     applsys.fnd_nodes n
WHERE s.session_id = p.session_id
AND p.node_id = n.node_id
ORDER BY p.start_date DESC;

    The existing session showed status FAILED with the apply phase partially completed — a clear indicator of inconsistent execution state across nodes.

    Step 2: Check adalldefaults.txt

    Reviewed the defaults file for any relevant configuration:

    cat $APPL_TOP/admin/$TWO_TASK/adalldefaults.txt | grep -i missing
-- Key parameter found:
-- MISSING_TRANSLATED_VERSION = No

    Modifying and retrying with this parameter had no impact, confirming the issue was not translation-related.

    Step 3: Check Install Processes Table

    -- Check for stale entries in FND_INSTALL_PROCESSES
SELECT COUNT(*) FROM applsys.fnd_install_processes;

-- View stale entries in detail
SELECT process_status, process_name, last_update_date
FROM applsys.fnd_install_processes
ORDER BY last_update_date DESC;

-- Check AD_DEFERRED_JOBS
SELECT COUNT(*) FROM applsys.ad_deferred_jobs;
SELECT * FROM applsys.ad_deferred_jobs;

    Observation: FND_INSTALL_PROCESSES contained stale entries from the failed session. AD_DEFERRED_JOBS was empty.

    Root Cause

    The failure was caused by stale and inconsistent ADOP metadata tables — specifically APPLSYS.FND_INSTALL_PROCESSES and APPLSYS.AD_DEFERRED_JOBS. ADOP internally relies on these tables to track patch progress checkpoints, deferred job execution, and restart state management. When these tables retain entries from failed or incomplete sessions, ADOP assumes an incorrect execution state, leading to patch reconciliation failure, apply phase breakdown, and node-level inconsistencies.

    Resolution Steps

    Step 1: Backup Critical Tables

    -- Always backup before any cleanup
CREATE TABLE applsys.fnd_install_processes_bak AS
SELECT * FROM applsys.fnd_install_processes;

CREATE TABLE applsys.ad_deferred_jobs_bak AS
SELECT * FROM applsys.ad_deferred_jobs;

-- Verify backups
SELECT COUNT(*) FROM applsys.fnd_install_processes_bak;
SELECT COUNT(*) FROM applsys.ad_deferred_jobs_bak;

    Step 2: Drop Stale Metadata Tables

    Dropping these tables forces ADOP to rebuild clean metadata during the next run:

    DROP TABLE applsys.fnd_install_processes;
DROP TABLE applsys.ad_deferred_jobs;

    Step 3: Reset the Restart Directory

    The restart directory can silently preserve failure states. Back it up and create a fresh one:

    cd $APPL_TOP/admin/$TWO_TASK

-- Backup existing restart directory
mv restart restart_bkp_$(date +%Y%m%d)

-- Create fresh restart directory
mkdir restart

-- Verify
ls -la | grep restart

    Step 4: Re-run the Patch

    adop phase=apply \
     patches=<patch_id> \
     restart=no \
     abandon=yes \
     apply_mode=downtime

    The patch completed successfully across all nodes after the metadata cleanup.

    Before vs After

    ComponentBefore FixAfter Fix
    ADOP SessionFailedSuccessful
    Node ConsistencyPartialFull
    Restart BehaviorStuckClean
    Patch ExecutionIncompleteCompleted

    Key Takeaways

    • ADOP is state-driven — even when logs appear clean, internal metadata drives execution decisions
    • Partial success is a clue — if some nodes succeed and one fails, focus on local metadata, not the patch itself
    • The restart directory matters — it can silently preserve failure states and must be validated before retrying
    • Downtime mode is not a fix-all — even in downtime, ADOP still reads metadata tables; corruption persists unless cleaned
    • Always backup before cleanup — never drop tables without creating a backup first

    When NOT to Use This Approach

    Avoid applying this fix if the issue is caused by missing database patches (ETCC warnings), file system or permission issues, incorrect patch sequencing, or environment misconfiguration. Always validate the root cause before performing any metadata cleanup.

    This scenario highlights a subtle but critical behavior in ADOP — sometimes patch failures are not caused by the patch itself, but by what the system remembers about past attempts. By resetting stale metadata, we allow ADOP to re-evaluate the environment cleanly, leading to successful execution.

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.

  • Oracle EBS Login Issue: Real-Time Production Incident and Fix (OACORE + WebLogic Filter)

    📌 Introduction

    In Oracle E-Business Suite (EBS) environments, login failures are often perceived as simple application issues. However, in complex architectures, they can originate from multiple interacting layers across the application and middleware stack.

    In this blog, I’ll walk through a real-world production incident where an OACORE JVM issue combined with WebLogic security filtering resulted in complete login inaccessibility.

    This case highlights the importance of analyzing both performance and security layers together when troubleshooting critical application outages.

    ⚠️ Issue Summary

    • Users were unable to access the EBS login page
    • Pages were hanging or not loading
    • WebLogic console reported:
    Connection rejected, filter blocked Socket

    🔍 Initial Observation

    From the application server:

    • Load Average: ~10+

    👉 This indicated:

    • High CPU utilization
    • System under heavy stress
    • Potential JVM thread contention

    🔬 Detailed Analysis

    • One of the OACORE managed server JVMs became unresponsive
    • Long-running threads caused thread pool exhaustion
    • Incoming user requests began queueing

    At the same time:

    • WebLogic connection filter was actively enforcing access rules
    • Legitimate requests were being rejected under stressed conditions

    🧠 Understanding the Components

    OACORE (Application Layer)

    Handles:

    • Login requests
    • Forms processing
    • Core application logic

    If JVM threads are exhausted:
    👉 Requests queue → login hangs

    Oracle WebLogic Server Connection Filter

    EBS environments may use:

    oracle.apps.ad.tools.configuration.wls.filter.EBSConnectionFilterImpl

    This filter:

    • Enforces IP-based access control
    • Overrides default allow rules

    If misconfigured or stressed:
    👉 Legitimate traffic may be blocked

    🎯 Root Cause Analysis (RCA)

    The login issue was not caused by a single failure point, but by a combination of application tier resource exhaustion and restrictive middleware-level access control.

    • High CPU utilization and long-running threads caused one OACORE JVM to become unresponsive
    • Thread pool exhaustion led to request queuing, preventing new login requests from being processed

    Simultaneously:

    • The WebLogic connection filter (EBSConnectionFilterImpl) enforced strict access control policies
    • Under high load conditions, legitimate client requests were rejected with “filter blocked Socket”

    This interaction between performance degradation and security enforcement amplified the impact, resulting in complete login inaccessibility despite partial system availability.

    🛠️ Resolution Approach (Controlled & Safe)

    The resolution approach focused on stabilizing the JVM layer while validating and correcting middleware-level access controls in a controlled manner.

    🔹 Step 1: Identify Unresponsive JVM

    ps -ef | grep oacore

    ✔ Identify JVM with abnormal CPU or stuck behavior

    🔹 Step 2: Handle Stuck JVM (Controlled Action)

    ⚠️ Important Note:

    Forcefully terminating JVM processes should NOT be performed without validation.

    ✔ Recommended Approach:

    • Confirm the process is unresponsive
    • Ensure no critical transactions are running
    • Prefer controlled shutdown where possible

    ✔ Example (Only if fully unresponsive and approved):

    kill -9 <PID>

    👉 Node Manager can restart the JVM automatically after termination

    🔹 Step 3: Rolling Restart of OACORE

    admanagedsrvctl.sh stop oacore_server1
    admanagedsrvctl.sh start oacore_server1

    ✔ Ensures clean JVM state
    ✔ Restores thread pool balance

    🔹 Step 4: Validate WebLogic Connection Filter

    ⚠️ Important Note:

    The WebLogic connection filter is a security control and should NOT be disabled permanently.

    ✔ In this case:

    • Filter behavior was validated as part of troubleshooting
    • Temporary relaxation was used to confirm impact

    ✔ Recommended Approach:

    • Review allowed IP ranges
    • Validate filter configuration
    • Re-enable filter after correction

    The filter was re-enabled after validation and correction of configuration.

    🔹 Step 5: Restart Admin Server (If Required)

    adadminsrvctl.sh stop
    adadminsrvctl.sh start

    ✔ Ensures configuration changes are applied

    ✅ Final Outcome

    • System load reduced from ~10 → ~1.5
    • OACORE JVMs stabilized
    • Login page restored
    • Users successfully accessed the application

    📊 Key Learnings

    1. OACORE JVM issues can manifest as complete application outages due to thread exhaustion and request queuing.
    2. WebLogic connection filters can unintentionally block legitimate traffic, especially under high load conditions.
    3. Multi-layer failures (application + middleware + security) significantly amplify incident impact.
    4. System load metrics (CPU, load average) provide early indicators of JVM stress and should be monitored proactively.

    🧩 Preventive Measures

    • Monitor JVM thread utilization proactively
    • Review connection filter configurations periodically
    • Avoid peak concurrent load spikes
    • Implement alerting for high CPU / load conditions
    • Validate session and request handling patterns

    🔐 Governance Considerations

    • Avoid forceful JVM termination without proper validation
    • Do not disable security controls without understanding impact
    • Follow change management procedures in production
    • Coordinate with application and security teams before changes

    🏁 Conclusion

    This incident demonstrates that login failures in Oracle EBS are not always isolated to a single component but can result from complex interactions across application performance and middleware security layers.

    The combination of JVM resource exhaustion and connection filtering behavior created a compounded failure scenario, leading to complete login disruption.

    A structured, multi-layer troubleshooting approach—focused on performance, configuration, and governance—enabled effective resolution while minimizing risk.

    This reinforces the importance of analyzing both system behavior and security controls together when addressing critical production incidents.

    💡 Pro Tip

    When troubleshooting Oracle EBS login issues, always validate both:

    • JVM health (thread utilization, CPU load)
    • Middleware controls (connection filters, access rules)

    Ignoring either layer can lead to incomplete or misleading diagnosis.

  • Oracle EBS OACORE Server in FAILED_NOT_RESTARTABLE State: Real-Time Issue, RCA and Fix

    In Oracle E-Business Suite (EBS) environments, application tier stability is critical to ensure seamless user experience. However, there are scenarios where managed servers behave unexpectedly and require manual intervention. This post walks through a real-world production issue where an OACORE managed server entered a FAILED_NOT_RESTARTABLE state, its impact, root cause analysis, and how it was resolved.

    Environment Details

    • Oracle E-Business Suite: R12.2.x
    • Application Tier: WebLogic Managed Servers
    • Component Impacted: OACORE Server (oacore_server1)
    • Environment Type: Production

    Problem Statement

    An alert was received indicating oacore_server1 was in FAILED_NOT_RESTARTABLE state. Upon verification, the server was running but Node Manager could not auto-restart it.

    Key Observations

    Despite the OACORE server being in a failed state, the application remained accessible and functional — traffic was being handled by other OACORE servers. This is due to the multi-OACORE architecture with load balancing via OHS/Web tier. However, this creates a hidden risk: load redistribution increases pressure on remaining servers and can lead to cascading failures if not addressed promptly.

    Detailed Analysis

    • Managed server restart attempts failed during initialization
    • Bulk concurrent requests were actively running
    • CPU utilization spiked on the application tier
    • JVM resources were under pressure

    Understanding FAILED_NOT_RESTARTABLE

    In Oracle WebLogic Server, a managed server is marked as FAILED_NOT_RESTARTABLE after repeated unsuccessful restart attempts. This is a protective mechanism designed to prevent unstable restart loops when the server cannot recover successfully.

    Root Cause Analysis

    The OACORE managed server entered FAILED_NOT_RESTARTABLE state due to repeated startup failures following an unclean or resource-constrained shutdown. High CPU utilization and heavy concurrent workload placed JVM resources under pressure, preventing a clean restart cycle. Residual runtime artifacts (such as incomplete shutdown state or resource locks) prevented successful reinitialization, causing WebLogic to mark the server as FAILED_NOT_RESTARTABLE.

    Resolution

    cd $ADMIN_SCRIPTS_HOME
./admanagedsrvctl.sh stop oacore_server1
./admanagedsrvctl.sh start oacore_server1

    After the controlled restart, the server returned to RUNNING state with all deployments active and the application stable.

    Identify Inactive Forms Sessions

    Inactive sessions holding resources can contribute to JVM pressure. Use this query to identify them safely — do not terminate without proper validation and approvals:

    SELECT s.sid,
       s.serial#,
       s.username,
       s.status,
       s.program,
       s.machine,
       ROUND(s.last_call_et/3600,2) AS hours_inactive
FROM v$session s
WHERE s.status = 'INACTIVE'
AND s.username = 'APPS'
AND s.program LIKE 'frmweb%'
AND s.last_call_et > 28800   -- 8 hours
ORDER BY hours_inactive DESC;

    Reference only — do NOT execute without validation:

    ALTER SYSTEM KILL SESSION 'SID,SERIAL#' IMMEDIATE;

    Automate Session Monitoring

    Use this script to monitor inactive sessions every 8 hours via cron:

    #!/bin/bash
export ORACLE_SID=your_sid
export ORACLE_HOME=/path/to/oracle_home
export PATH=$ORACLE_HOME/bin:$PATH

sqlplus -s / as sysdba <<EOF
SET LINES 200
SET PAGES 200
SELECT COUNT(*) AS inactive_sessions
FROM v\$session
WHERE status='INACTIVE'
AND username='APPS'
AND program LIKE 'frmweb%'
AND last_call_et > 28800;
EXIT;
EOF
    # Crontab entry - every 8 hours
0 */8 * * * /path/to/inactive_sessions.sh >> /tmp/inactive_sessions.log

    DBA Quick Commands

    -- Check system load
top
uptime
ps -ef | grep oacore

-- Check running concurrent requests
SELECT request_id, phase_code, status_code
FROM fnd_concurrent_requests
WHERE phase_code = 'R';

    Key Takeaways

    • Application may appear healthy even when an OACORE server fails due to load balancing
    • FAILED_NOT_RESTARTABLE is a protective mechanism, not the root cause itself
    • Resource pressure and restart failures must be analyzed together
    • Controlled and governed actions are critical in production environments
    • Proactive session monitoring via automation helps prevent recurrence

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.