Tag: 12.2

  • Oracle EBS R12.2 Multi-Node Patching in Downtime Mode: Recovering a Failed Slave Node with restart=yes

    Alhamdulillah, sharing another real patching session — this one from an Oracle E-Business Suite R12.2 multi-node environment. While applying a patch in downtime mode, I encountered an interesting situation: the patch completed successfully on the master node but failed on the slave node, while the final ADOP output still showed:

    adop exiting with status = 0 (Success)

    This post walks through the complete sequence: pre-checks, the slave-node failure, investigation, recovery using restart=yes, and the SQL checks used to verify the final patch state.

    Environment

    • Oracle E-Business Suite R12.2.13
    • AD/TXK Delta 15 (ADOP C.Delta.15)
    • Two application nodes: appsnode1 (master/admin node) and webnode1 (slave/web node)
    • Oracle Database 19c on dbnode1
    • Patch mode: downtime (single file system apply; no online patching cutover involved)

    Pre-Checks Before the Apply

    Before starting the patch, I checked three things: the current ADOP session status, application processes on all application nodes, and the current file-system edition.

    From the master node:

    [applmgr@appsnode1]$ adop -status
    Node Name Node Type Phase Status
    --------------- ---------- --------------- ---------------
    appsnode1 master APPLY ACTIVE
    CLEANUP NOT STARTED
    webnode1 slave APPLY ACTIVE
    CLEANUP NOT STARTED

    I also confirmed I was operating from the run edition:

    [applmgr@appsnode1]$ echo $FILE_EDITION
    run

    Then I checked for application-tier processes:

    [applmgr@appsnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|FNDCRM|FNDOPP|oacore|forms|WebLogic|NodeManager' | grep -v grep

    There was no output, confirming the expected services were down on the master node.

    Important: In a multi-node environment, repeat these process checks on every application node. Checking only the master node is not sufficient — a condition on a slave node can otherwise go unnoticed until the apply reaches it.

    Starting the Downtime Apply

    The downtime apply was initiated from the master node:

    [applmgr@appsnode1]$ adop phase=apply apply_mode=downtime patches=<patch_number>

    ADOP manages execution across the registered application-tier nodes, so there was no need to manually start ADOP on the slave node.

    During initialization, ETCC reported missing database fixes:

    [WARNING] ETCC: The following required database fixes have not been applied to node dbnode1:
    ...
    Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

    ETCC warnings should always be reviewed before proceeding. In this case, the apply was allowed to continue based on the maintenance plan, while the outstanding database fixes were documented and tracked separately for a subsequent maintenance activity.

    Master Succeeded, Slave Failed

    The patch application succeeded on the master node but failed on the slave:

    Applying <patch> patch(es) on admin node: [appsnode1].
    txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
    [ERROR] adop phase=apply failed on Node: "webnode1"
    Summary report for current adop session:
    Node webnode1: Failed
    - Apply status: Failed
    Node appsnode1: Completed successfully
    - Apply status: Completed successfully
    adop exiting with status = 0 (Success)

    This is an important operational lesson. Despite the slave-node failure being clearly reported in the summary, the ADOP invocation ended with adop exiting with status = 0 (Success).

    Therefore, relying only on the shell return code or the final ADOP exit line is not sufficient for determining whether every application node completed successfully. The per-node summary must also be reviewed.

    I confirmed the state with:

    adop -status -detail

    which showed:

    Node Name Node Type Phase Status
    --------------- ---------- --------------- ---------------
    appsnode1 master APPLY ACTIVE
    webnode1 slave APPLY FAILED

    For automated patching this matters even more: scripts should validate the ADOP session and per-node phase status rather than treating $? = 0 alone as proof of a successful multi-node patch.

    Investigating the Slave Node

    I then logged into webnode1. Application services were expected to be completely down, so I checked for NodeManager:

    [applmgr@webnode1]$ ps -ef | grep -i nodemanager | grep -v grep
    applmgr 4148261 1 0 Jun18 ... weblogic.NodeManager -v

    A NodeManager process was still running, and had been since a previous month.

    I checked whether other WebLogic or EBS application processes were present:

    [applmgr@webnode1]$ ps -ef | grep -i weblogic | grep -v grep

    Only the NodeManager process remained. Concurrent Manager processes were also absent:

    [applmgr@webnode1]$ ps -ef | grep FNDLIBR | grep -v grep

    No output was returned.

    Since the environment was in a planned patching outage and no managed servers were running, the stale NodeManager process was stopped:

    [applmgr@webnode1]$ kill 4148261

    I then verified again:

    [applmgr@webnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|oacore|forms|WebLogic|NodeManager' | grep -v grep

    No relevant processes remained.

    A Note About Root Cause

    The presence of the stale NodeManager was an abnormal condition that needed to be corrected before retrying the patch. However, unless the ADOP logs explicitly identify that process as the reason for the failure, it is safer not to state that NodeManager was definitively the root cause.

    What can be established from this incident is:

    1. The slave-node apply failed.
    2. A stale NodeManager process was discovered on that node.
    3. The process was stopped after confirming application services were intended to be down.
    4. The failed ADOP apply subsequently completed successfully with restart=yes.

    For a formal RCA, the relevant ADOP and slave-node logs should be used to establish the exact failure mechanism.

    Also, before terminating any process, verify that:

    • It belongs to the expected EBS environment/file system.
    • No managed servers depend on it.
    • Application services are intended to be down.
    • The action complies with your production change procedure.

    Recovering the Failed Apply with restart=yes

    After cleaning up the slave node, I returned to the master node. The existing failed apply was resumed using:

    [applmgr@appsnode1]$ adop phase=apply patches=<patch_number> apply_mode=downtime restart=yes

    ADOP detected the existing session:

    Checking for existing adop sessions.
    Application tier services are down.
    Continuing with the existing session [Session ID: <n>].

    The apply was then processed successfully:

    Applying <patch> patch(es) on admin node: [appsnode1].
    txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
    txkADOPEvalSrvStatus.pl returned SUCCESS

    The final summary showed both nodes successful:

    Summary report for current adop session:
    Node webnode1: Completed successfully
    - Apply status: Completed successfully
    Node appsnode1: Completed successfully
    - Apply status: Completed successfully

    The important point here is that I resumed the existing ADOP session rather than initiating an unrelated fresh patching attempt. The recovery command was executed from the master/admin node, allowing ADOP to coordinate processing on the registered slave node.

    For the exact semantics of restart=yes in downtime mode and multi-node processing, refer to the Oracle E-Business Suite Patching Guide and the relevant My Oracle Support documentation. What I can report from this incident is the observed behaviour above: ADOP continued the existing session and both nodes subsequently reported success.

    Post-Restart Validation

    After the successful rerun, I checked the detailed ADOP status:

    [applmgr@appsnode1]$ adop -status -detail

    The nodes no longer showed a failed apply state.

    I also scanned the latest ADOP logs:

    [applmgr@appsnode1]$ adopscanlog -latest=yes
    Scanning .../log/adop/<session>/ directory ...
    No Errors.

    adopscanlog is a useful quick check after an ADOP phase because it searches the session logs for reported errors that may otherwise be easy to miss.

    SQL Verification: Confirming Patch Registration

    I don’t rely only on the ADOP console output. After patching, I also verify the database-side patch records.

    1. Check AD_BUGS

    SET LINES 200
    COL BUG_NUMBER FORMAT A15
    COL CREATION_DATE FORMAT A22
    SELECT bug_number,
    creation_date
    FROM ad_bugs
    WHERE bug_number = '<patch_number>';

    This confirms whether the specified bug/patch number is recorded in AD_BUGS. Because patch contents and registration behaviour can vary, this check should be used together with ADOP status, logs, and the relevant applied-patch records rather than as standalone proof of complete patch application. Running the same query before patching is also useful, because it establishes a clear before-and-after state.

    2. Check Applied Patch Records

    SELECT applied_patch_id,
    patch_name,
    patch_type,
    source_code,
    creation_date,
    last_update_date,
    data_model_done_flag
    FROM ad_applied_patches
    WHERE patch_name = '<patch_number>';

    Depending on the patch structure, this helps confirm the corresponding patch-driver records.

    3. Verify AD/TXK Code Levels

    COL ABBREVIATION FORMAT A15
    COL CODELEVEL FORMAT A25
    SELECT abbreviation,
    codelevel
    FROM ad_trackable_entities
    WHERE abbreviation IN ('ad','txk')
    ORDER BY abbreviation;

    For this environment the query returned:

    ABBREVIATION CODELEVEL
    --------------- -------------------------
    ad C.15
    txk C.15

    This check is especially important when applying AD/TXK-related patches or preparing for an EBS Release Update Pack.

    4. Verify the EBS Release

    SELECT release_name
    FROM fnd_product_groups;

    Expected for this environment: 12.2.13.

    5. Verify Node Registration

    COL NODE_NAME FORMAT A25
    SELECT node_name,
    support_cp,
    support_forms,
    support_web,
    support_admin
    FROM fnd_nodes
    ORDER BY node_name;

    This provides a quick sanity check of application-tier node registration and responsibilities.

    6. Verify ADOP Valid Nodes

    SELECT *
    FROM adop_valid_nodes
    ORDER BY node_name;

    Both application nodes should appear as expected.

    In this case, the final checks confirmed:

    • The bug/patch number was recorded in AD_BUGS.
    • The corresponding applied-patch records were present.
    • AD and TXK were at the expected C.15 level.
    • EBS remained at Release 12.2.13.
    • Both application nodes were properly registered in FND_NODES.
    • Both nodes appeared correctly in ADOP_VALID_NODES.
    • adopscanlog reported no errors after the successful rerun.

    Together, these checks provide much stronger verification than relying on the final ADOP console message alone.

    Complete the Session with Cleanup

    After confirming the apply completed successfully, the session still showed:

    CLEANUP NOT STARTED

    The cleanup phase was then run:

    adop phase=cleanup

    After cleanup, perform another status and log review to make sure the session is in the expected final state.

    Key Takeaways

    Don’t rely only on ADOP’s exit status in a multi-node environment. In this incident, ADOP printed status = 0 (Success) even though the per-node summary clearly showed the slave node had failed.

    Check every application node before a downtime apply. A clean process list on the master does not guarantee the slave nodes are equally clean.

    Use the per-node ADOP status as part of your success criteriaadop -status -detail.

    Investigate the failed node before retrying. In this case a stale NodeManager process was discovered and removed before the successful retry. Unless supported by the logs, however, avoid treating correlation as a confirmed root cause.

    Resume the existing failed apply appropriately — after correcting the condition on the slave node, the existing apply was resumed from the master node with restart=yes.

    Use adopscanlog -latest=yes after the rerun.

    Verify patch registration in the databaseAD_BUGS, AD_APPLIED_PATCHES, AD_TRACKABLE_ENTITIES, FND_NODES, ADOP_VALID_NODES, FND_PRODUCT_GROUPS.

    Run ETCC and review its findings. If required database fixes are deferred, document and track them for the appropriate maintenance window rather than allowing them to become forgotten technical debt.

    Complete the ADOP cleanup phaseadop phase=cleanup.

    Final Thought

    Multi-node EBS patching adds a layer of validation that is easy to overlook: success on the master node does not automatically mean success across the application tier.

    The most useful lesson from this incident was not the retry command itself, but the validation process around it:

    Check every node → read the ADOP per-node summary → investigate the failed node → resume the existing session → scan the logs → verify the database records → complete cleanup.

    That sequence gives you a far more defensible and auditable patching result than relying on a single Success message at the end of the command.

    I hope this helps anyone troubleshooting a failed application-tier node during Oracle E-Business Suite R12.2 downtime patching. Feel free to leave a comment if you have questions.


    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS R12.2 AutoConfig Template Version Conflict After Patching: How to Preserve Customizations

    Alhamdulillah, sharing another real troubleshooting session — this one from an Oracle E-Business Suite R12.2 multi-node environment. After applying a patch, AutoConfig aborted on the primary application node with a template version conflict. This post walks through the exact error, how I identified which template entries were genuine customizations versus Oracle-shipped content, and the correct way to rebase a custom template so AutoConfig completes cleanly.

    Environment

    • Oracle E-Business Suite R12.2.13, AD/TXK Delta 15
    • Two application nodes: appsnode1 (primary/admin) and webnode1 (secondary web node)
    • Database: Oracle 19c on dbnode1
    • Third-party customization: More4Apps servlet registered in oacore web.xml

    The Error

    After patching, running AutoConfig on the run file system failed immediately:

    [applmgr@appsnode1 scripts]$ sh adautocfg.sh
    Enter the APPS user password:
    The log file for this session is located at: .../admin/log/<timestamp>/adconfig.log
    wlsDomainName: EBS_domain
    WLS Domain Name is VALID.
    AutoConfig is configuring the Applications environment...
    AutoConfig will consider the custom templates if present.
    Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

    The log pinpointed the conflicting template:

    [ FND_TOP ]
    
    [ VERSION CONFLICTS INFORMATION ]
    Template shipped by oracle is having a version different than the template lying in custom directory.
    Template shipped by Oracle : $FND_TOP/admin/template/oacore_web_xml_FMW.tmp (version: 120.17.12020000.33)
    Custom template            : $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp (version: 120.17.12020000.31)
    Please resolve the differences between the two templates or refer to the Oracle E-Business Suite Setup Guide for further details.

    What happened is straightforward: the patch delivered a newer version of oacore_web_xml_FMW.tmp (120.17.12020000.33), but a custom copy of the older version (120.17.12020000.31) exists under $FND_TOP/admin/template/custom. AutoConfig detects the mismatch and refuses to run, because blindly using the stale custom template would silently discard whatever Oracle changed in the new version.

    Root Cause Analysis: What Is Actually a Customization?

    This is the critical analysis step. Before touching anything, you need to answer one question precisely: which lines in the custom template are genuine customizations, and which are just old Oracle-shipped content?

    First, take a backup and diff the new shipped template against the custom one:

    cd $FND_TOP/admin/template
    
    cp -p custom/oacore_web_xml_FMW.tmp \
          custom/oacore_web_xml_FMW.tmp.pre_patch_25Jul2026
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

    The diff showed three differences beyond the header:

    1. A m4aServlet servlet and servlet-mapping (More4Apps) present only in the custom template
    2. A SyncServlet servlet and servlet-mapping present only in the custom template
    3. A RequestAuditReport (QP) servlet present only in the new Oracle template

    At first glance it looks like there are two customizations to carry forward: m4aServlet and SyncServlet. This is where a wrong assumption would corrupt the template. To verify, I compared the old shipped template against the new shipped template. In an R12.2 dual file system this is easy — the other file system still had the previous patch level, so both shipped versions were available:

    diff -u \
      <old_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp \
      <new_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp

    Key excerpt from the shipped-vs-shipped diff:

    -   <servlet>
    -     <servlet-name>SyncServlet</servlet-name>
    -     <servlet-class>oracle.apps.jtf.cac.sync.transport.SyncServlet</servlet-class>
    -  </servlet>
    ...
    +     <servlet>
    +         <servlet-name>RequestAuditReport</servlet-name>
    +         <servlet-class>oracle.apps.qp.servlet.RequestAuditReport</servlet-class>
    +    </servlet>

    This settled the question:

    • SyncServlet was Oracle-shipped content in version .31 and is no longer present in the .33 shipped template. It was never a customer customization, so it should NOT be carried forward.
    • RequestAuditReport is new Oracle-shipped content in .33 — it must be retained.
    • m4aServlet was not present in either shipped template version compared — it exists only in the custom directory, which identifies it as the genuine site-specific customization.

    If I had merged the old custom template on top of the new one, I would have reintroduced a servlet that is no longer present in Oracle’s newer shipped template. This is why “just copy your custom entries into the new template” advice found on many forums is dangerous — verify against the shipped versions first.

    The Fix: Rebase the Custom Template

    The correct approach: replace the custom template with the new shipped version, then re-apply only the genuine customization.

    cd $FND_TOP/admin/template
    
    # Rebase custom template on the new shipped version
    cp -p oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    # Re-insert the only genuine customization (m4aServlet)
    sed -i '/<!-- FND Servlets -->/a\
    \
      <servlet>\
        <servlet-name>m4aServlet</servlet-name>\
        <servlet-class>com.more4apps.r12.servlet.XmlServlet</servlet-class>\
      </servlet>\
    \
      <servlet-mapping>\
        <servlet-name>m4aServlet</servlet-name>\
        <url-pattern>/m4aServlet/*</url-pattern>\
      </servlet-mapping>\
    ' custom/oacore_web_xml_FMW.tmp

    This matches the procedure in the Oracle E-Business Suite Setup Guide: when a patch delivers a newer version of a template you have customized, copy the new Oracle template into the custom directory and reapply the still-required customizations to that copy — never the other way around.

    Verify before rerunning AutoConfig — the headers must match, and the only diff must be the customization:

    grep '\$Header' oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    oacore_web_xml_FMW.tmp:        version 120.17.12020000.33
    custom/oacore_web_xml_FMW.tmp: version 120.17.12020000.33
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    --- only the m4aServlet servlet + mapping block should appear ---

    As an additional best-practice validation, Oracle recommends running the AutoConfig configuration-check utility to preview the impact of template changes before the actual AutoConfig run:

    $AD_TOP/bin/adchkcfg.sh contextfile=$CONTEXT_FILE

    Rerun AutoConfig:

    cd $ADMIN_SCRIPTS_HOME
    sh adautocfg.sh
    
    ...
    Configuring templates from all of the product tops...
            Configuring AD_TOP........COMPLETED
            Configuring FND_TOP.......COMPLETED
            ...
    AutoConfig completed successfully.

    A Bonus Finding on the Second Node

    While repeating the check on the secondary web node, the header comparison came back clean — custom template already at .33 — but something was off:

    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp
    (no output)
    
    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp_ORIG
    61:    <servlet-name>m4aServlet</servlet-name>

    The current custom template matched the newer shipped version, but the m4aServlet customization was missing. Interestingly, the _ORIG backup still contained the customization. This indicated that during an earlier template update, the customization was not carried forward. AutoConfig had been running fine and web.xml was being generated — just without the third-party servlet.

    I re-inserted the m4aServlet block using the same sed command, verified the diff, and ran AutoConfig on that node as well:

    AutoConfig completed successfully.

    This is the real lesson of the post. A template version conflict at least fails loudly. A customization that is accidentally omitted during a template rebase may not — AutoConfig can complete successfully while the generated configuration no longer contains the required customization, and you find out later when the third-party tool stops working after a bounce.

    An Additional Message Observed

    The AutoConfig log in this run also contained:

    ECC not enabled, setting FND_ECC_ENABLED to FALSE
    [ FND_ECC_ENABLED ]
    INFO : Error updating/creating profile option value.

    In this case Enterprise Command Center is not configured, AutoConfig continued past this message, and the run ultimately exited with status 0. Always evaluate such messages in the context of your own environment rather than assuming every INFO : Error entry can be ignored.

    Post-Fix Checklist

    1. Confirm the customization landed in the generated web.xml (get the target path from the adconfig log and grep for your servlet).
    2. Bounce the application services (at least the oacore managed servers) so the new web.xml is deployed.
    3. Test the customization end to end — for More4Apps, hit the servlet URL from the wizard.
    4. If your patching cycle requires it, regenerate appsutil.zip (perl $AD_TOP/bin/admkappsutil.pl) and refresh the database tier.
    5. Repeat the custom template verification on every application node — as seen above, nodes can drift.

    Key Takeaways

    • Never resolve this error by copying the old custom template over the new shipped one, and never blindly merge either. The Oracle-documented procedure is to copy the new shipped template into the custom directory and re-apply only verified customizations to that copy.
    • Use the dual file system to your advantage: diff old-shipped vs new-shipped to separate Oracle’s changes from your customizations.
    • After any template rebase, diff the shipped and custom templates — the output should contain nothing but your customizations.
    • Audit custom templates on all nodes periodically. A missing customization may not trigger an AutoConfig error, allowing the generated configuration to differ from what you expect.

    I hope this helps someone facing the same conflict. Feel free to leave a comment if you have questions.


    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS R12.2 Clone Redirects to PROD After Login: Diagnosing Stale Configuration References

    When you finish cloning an Oracle E-Business Suite R12.2 environment, the moment of truth is the login page. You hit the clone’s web entry URL, the page loads, you type your credentials… and the browser quietly throws you onto the production URL. On a multi-node app tier this is more than an annoyance — it means your freshly cloned, supposedly isolated environment is reaching back into PROD, and a careless tester could authenticate against the wrong system entirely.

    I ran into exactly this on a recent R12.2.11 clone built on Oracle Cloud Infrastructure: a three-node application tier sharing a single run/patch file system over an FSS (File Storage Service) NFS mount. The clone came up, services started, but every login attempt redirected to the PROD web entry host. The initial investigation uncovered two primary issues — phantom WebLogic managed servers carried over from the source domain, and a missing DNS A-record for the clone’s web entry hostname. Along the way I also identified several other clone-related configuration areas that can produce the same symptom, including stale profile options, OHS configuration remnants, load balancer redirects, and custom code references.

    This post walks through how the symptom presents, how to diagnose it cleanly, and the supported way to fix each cause.

    Throughout, I use placeholder names — clone-apps.example.com for the clone web entry host, prod-apps.example.com for production, and appnode1/2/3 for the three app-tier nodes. Substitute your own values.


    The architecture (why a shared file system matters here)

    The clone app tier looked like this:

    • Three application-tier nodes (appnode1, appnode2, appnode3).
    • A shared dual file system (fs1 run / fs2 patch) hosted on an OCI FSS NFS export, mounted identically on all three nodes.
    • A single WebLogic domain (EBS_domain_<SID>) living on that shared file system.

    The shared file system is the detail that makes the “phantom managed server” problem sticky. Because config.xml and the managed-server definitions physically live on the shared FSS mount, any leftover managed-server entries from the source domain are visible to every node at once. AutoConfig regenerates context-driven artifacts, but it does not, on its own, delete managed servers that no longer belong to the topology.


    The symptom

    After adcfgclone completed and the application services started:

    • The clone login page (https://clone-apps.example.com:<port>/OA_HTML/AppsLogin) rendered correctly.
    • On submitting credentials — or sometimes immediately on the redirect to the home page — the browser landed on https://prod-apps.example.com/....
    • The WebLogic Admin Console showed more managed servers than the three-node clone should have, several of them in an unreachable / shutdown state.

    Two independent problems reinforced each other: stale host references remained inside the cloned WebLogic topology, while the clone web entry hostname could not be resolved correctly. Either issue can cause redirect anomalies, but together they consistently redirected users back to PROD.


    First, rule out the context file

    Before investigating WebLogic, verify the clone context file itself doesn’t still contain production hostnames. AutoConfig can only generate correct configuration if the context values are correct, so everything downstream is built from here:

    grep -i "prod-apps.example.com" $CONTEXT_FILE
    grep -i "prod" $CONTEXT_FILE

    If PROD survives in the context, fix it there and re-run AutoConfig before chasing anything downstream — otherwise you’ll be debugging generated artifacts while the source of the bad values sits upstream.


    Root cause 1 — Phantom WebLogic managed servers

    A correctly provisioned three-node oacore service should have oacore_server1, oacore_server2, oacore_server3 (and the matching oafm, forms, forms-c4ws servers per node). The cloned domain carried extra managed servers that mapped to the source environment’s nodes — servers that pointed at listen addresses and host references belonging to PROD.

    These phantom servers do three harmful things:

    1. They keep PROD host references alive inside config.xml.
    2. They confuse the EBS service control and the Admin Server’s view of the cluster.
    3. They can answer (or fail to answer) requests in ways that surface PROD URLs.

    Diagnosing it

    Inventory what the domain actually contains versus what the topology should be:

    # What managed servers does the domain config believe in?
    grep -E "<name>|<listen-address>" \
    $EBS_DOMAIN_HOME/config/config.xml | grep -iE "oacore|oafm|forms|server"
    # What does EBS think the nodes are?
    sqlplus apps/<pwd> <<'EOF'
    COL node_name FORMAT a20
    COL server_address FORMAT a18
    COL support_cp FORMAT a10
    COL support_web FORMAT a10
    COL support_admin FORMAT a13
    SELECT node_name,
    NVL(server_address,'-') AS server_address,
    NVL(support_cp,'-') AS support_cp,
    NVL(support_web,'-') AS support_web,
    NVL(support_admin,'-') AS support_admin
    FROM fnd_nodes
    ORDER BY node_name;
    EOF

    Any managed server or fnd_nodes row that references a host which is not one of appnode1/2/3 is a phantom artifact from the source.

    It’s also worth checking the role assignments — sometimes the hostname is correct but node registrations are duplicated or carry the wrong roles:

    COL node_name FORMAT a30
    COL support_db FORMAT a10
    SELECT node_name,
    support_db,
    support_cp,
    support_web,
    support_admin
    FROM fnd_nodes
    ORDER BY node_name;

    Fixing it (the supported way)

    Do not hand-edit config.xml. EBS R12.2 ships a provisioning utility to add and delete managed servers cleanly, keeping the domain, AutoConfig, and the database registration in sync. Stop the affected services first, then delete each phantom server:

    # Stop the managed servers / services before topology changes
    $ADMIN_SCRIPTS_HOME/adstpall.sh apps/<apps_pwd>
    # Delete a phantom managed server (repeat per orphaned server / service type)
    perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
    ebs-delete-managedserver \
    -contextfile=$CONTEXT_FILE \
    -managedsrvname=oacore_server4 \
    -servicetype=oacore \
    -promptmsg=hide

    After removing every phantom server, run AutoConfig on each app node so the context, the domain, and fnd_nodes agree:

    $ADMIN_SCRIPTS_HOME/adautocfg.sh

    Then confirm fnd_nodes only lists the three real clone nodes, and the Admin Console only shows the expected per-node managed servers. If you find stale node rows after the clean-up, the standard FND_CONC_CLONE.SETUP_CLEAN → AutoConfig sequence on each tier is the canonical way to rebuild the node registration. Run it as APPS, then re-run AutoConfig on every tier:

    EXEC FND_CONC_CLONE.SETUP_CLEAN;
    COMMIT;

    (Always take that step with the DBA team’s sign-off on a shared environment.)


    Root cause 2 — The missing DNS A-record

    With the phantom servers gone, the redirect still misbehaved intermittently. The reason was simpler and entirely outside EBS: the clone’s web entry hostname had no DNS A-record.

    The login flow builds its target URL from the AutoConfig web entry variables. Check them:

    grep -E "s_webentryhost|s_webentrydomain|s_webentryurlprotocol|s_active_webport|s_url_protocol|s_login_page" \
    $CONTEXT_FILE

    It’s worth proving the generated login URL, not just the host variable — if AutoConfig hasn’t fully taken, these still show a PROD host and you’ll be chasing DNS for a problem that lives in the context:

    grep -i "webentry" $CONTEXT_FILE
    grep -i "login_page" $CONTEXT_FILE

    The context correctly named clone-apps.example.com as s_webentryhost. But on the app nodes — and for clients — that name did not resolve:

    nslookup clone-apps.example.com
    # ** server can't find clone-apps.example.com: NXDOMAIN
    dig +short clone-apps.example.com
    # (empty)

    Because the clone web entry host was not resolvable, requests that relied on generated URLs could not consistently resolve back to the clone environment. At the same time, stale PROD references still lived inside the cloned WebLogic topology. Together these two conditions caused redirects to be generated using PROD host information, sending users away from the clone — which is why fixing the phantom servers alone wasn’t enough. To be clear, EBS does not contain any built-in mechanism that redirects a clone to production. Such redirects are almost always caused by stale configuration, profile options, WebLogic topology artifacts, load balancer settings, or DNS resolution — not by EBS doing anything magical. The browser is simply following a redirect built from stale host values. Name resolution has to work and the topology has to be clean.

    Fixing it

    Add an A-record for the clone web entry host in the appropriate DNS zone, pointing at the clone’s web-tier listen address (or its load balancer / OCI public-or-private IP, depending on your access path):

    clone-apps.example.com. IN A 10.x.x.x

    If a DNS change isn’t immediately possible and you only need the app nodes to resolve it for validation, a temporary /etc/hosts entry on each of appnode1/2/3 will confirm the theory — but a proper A-record is the real fix, because clients need to resolve it too:

    10.x.x.x clone-apps.example.com clone-apps

    Verify resolution from each node and re-test:

    for n in appnode1 appnode2 appnode3; do
    echo "== $n =="; ssh $n "nslookup clone-apps.example.com | tail -3"
    done

    Before concluding the issue persists, test using an incognito/private browser session or clear the browser cache. Browsers frequently cache redirects, cookies, and DNS information that can make a corrected environment appear unchanged.


    Root cause 3 — Stale profile option URLs (the other usual suspect)

    Even when DNS resolves and the WebLogic topology is clean, a clone can still throw users at PROD because profile option values were copied straight from the source. These are among the most common causes of a redirect-to-PROD, and they deserve a deliberate check rather than a passing glance. The usual culprits are APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, and ICX_FORMS_LAUNCHER.

    Target them directly:

    COL profile_option_name FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
    fpov.profile_option_value
    FROM fnd_profile_option_values fpov,
    fnd_profile_options_vl fpo
    WHERE fpov.profile_option_id = fpo.profile_option_id
    AND fpo.profile_option_name IN
    ('APPS_WEB_AGENT',
    'APPS_FRAMEWORK_AGENT',
    'APPS_SERVLET_AGENT',
    'ICX_FORMS_LAUNCHER');

    Or sweep more broadly for any value still carrying a PROD host:

    COL profile_option_name FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
    fpov.profile_option_value
    FROM fnd_profile_option_values fpov,
    fnd_profile_options_vl fpo
    WHERE fpov.profile_option_id = fpo.profile_option_id
    AND UPPER(fpov.profile_option_value) LIKE '%PROD%';

    Most of these are AutoConfig-managed, so the right fix is almost always to correct the context and re-run AutoConfig rather than to update the profile value by hand. Hand-updating a profile that AutoConfig owns just means it reverts on the next run. If AutoConfig keeps recreating the wrong value, fix the context file or web entry settings first; otherwise the next AutoConfig run simply reintroduces the problem.


    Other places a PROD reference hides: OHS and the OCI load balancer

    A surprising share of clone redirects originate outside the database and the WebLogic domain entirely — in the web tier configuration that AutoConfig generates, or in the load balancer sitting in front of it. Worth checking these early rather than last.

    Oracle HTTP Server (OHS). Generated OHS config — mod_wl_ohs.conf chief among them — can carry PROD host references. Rather than hard-coding version-specific paths, grep the web-tier and FMW homes broadly:

    grep -R "prod-apps.example.com" \
    $INST_TOP \
    $EBS_DOMAIN_HOME/config \
    $FMW_HOME \
    $ORACLE_HOME 2>/dev/null

    OCI load balancer / reverse proxy. If an OCI Load Balancer (or any reverse proxy) fronts the environment, the redirect can be introduced at that layer even after EBS is fully corrected. Verify:

    • Host header preservation — the backend should receive the clone host, not a rewritten PROD one.
    • Backend set configuration — backends point at the clone app nodes, not PROD.
    • SSL/TLS termination — the protocol and host the LB forwards match what AutoConfig expects (s_webentryurlprotocol, s_active_webport).
    • Redirect rules / rule sets — no listener rule is rewriting the host to PROD.

    I’ve seen an OCI LB listener keep sending users to PROD long after the EBS tier itself was spotless, purely because of a stale redirect rule on the listener.

    Custom code and JARs. Customizations are a notorious hiding place — hardcoded URLs in custom packages, JSPs, or Java survive every clone untouched by AutoConfig. Grep the custom homes too:

    grep -R "prod-apps.example.com" \
    $XX_TOP \
    $JAVA_TOP \
    $CUSTOM_TOP 2>/dev/null

    Validation checklist after remediation

    1. nslookup / dig resolves clone-apps.example.com from all three app nodes and from a client workstation.

    2. WebLogic Admin Console lists only the expected per-node managed servers, all in RUNNING state.

    3. fnd_nodes contains only appnode1/2/3 (check support_cp, support_web, support_admin); no PROD host references remain.

    4. No stale source context files are still registered. Old registrations sometimes survive a clone and cause confusion later during AutoConfig or service management:

    COL node_name FORMAT a20
    COL ctx_file FORMAT a60
    SELECT node_name,
    ctx_type,
    NVL(path,'-') AS ctx_file,
    status
    FROM fnd_oam_context_files
    WHERE status = 'S'
    ORDER BY node_name;

    In a clean clone, every active context file should belong to the clone environment. Any remaining source-environment context registration should be reviewed and removed before further AutoConfig runs.

    5. The profile options from Root cause 3 (APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, ICX_FORMS_LAUNCHER) all carry clone values, not PROD — re-run the targeted query from that section to confirm.

    6. The login page loads from the clone URL and — critically — the post-login redirect stays on clone-apps.example.com, never bouncing to prod-apps.example.com.

    7. Sweep the entire configuration for the PROD hostname, not just the context file. PROD remnants love to hide in OHS configs, mod_wl_ohs.conf, generated XML, and custom integrations:

    grep -R "prod-apps.example.com" \
    $INST_TOP \
    $EBS_DOMAIN_HOME/config \
    $FMW_HOME 2>/dev/null

    Anything this turns up needs to be corrected (and usually re-generated via AutoConfig) before users find it for you.

    8. No context file registered in the database still points at a PROD path. Stale PROD context registrations can cause odd behaviour long after the clone:

    COL node_name FORMAT a20
    COL path FORMAT a70
    SELECT node_name,
    path
    FROM fnd_oam_context_files
    WHERE UPPER(path) LIKE '%PROD%';

    9. If an OCI Load Balancer or reverse proxy fronts the environment, verify its listener rules, backend sets, host-header forwarding, and SSL termination settings contain no PROD references.


    Lessons learned / a small post-clone checklist

    Cloning R12.2 onto a shared file system multi-node tier adds two checks that single-node clones let you skip:

    • Audit the managed-server topology immediately after adcfgclone. On a shared file system the source’s managed servers ride along inside config.xml. Delete phantoms with adProvisionEBS.pl ebs-delete-managedserver, never by editing XML, and re-run AutoConfig.
    • Resolve the web entry hostname before you trust the login page. A clone that “logs you into PROD” is very often a DNS problem wearing an EBS costume. Create the A-record as part of the clone runbook, not as a reaction to the redirect.
    • Treat a redirect-to-PROD as a stop-the-line event. Until both topology and name resolution are clean, assume the clone can still touch production and keep testers out.
    • Run FND_CONC_CLONE.SETUP_CLEAN before AutoConfig in cloned environments. This rebuilds node registrations and context metadata cleanly and helps prevent stale topology information from the source environment persisting in the clone. (On a shared environment, run it with the DBA team’s sign-off.)

    In this case, three unglamorous root causes — leftover managed servers, a missing DNS record, and stale profile option URLs — combined to produce the redirect. In practice, however, most redirect-to-PROD incidents come down to one broader problem: stale production references surviving the clone process. The fastest path to resolution is a systematic sweep of WebLogic, DNS, profile options, context files, OHS configuration, load balancers, and custom code until every production reference is gone.


    Have you hit a different flavour of the clone redirect? The web entry variables, SSL/load-balancer termination, and s_login_page overrides each have their own way of sending you to the wrong host — happy to compare notes in the comments.

  • When ADOP Remembers Too Much: Fixing Patch Failures Caused by Stale Metadata in Oracle EBS

    During an Oracle E-Business Suite ADOP patching cycle in a multi-node environment, the apply phase failed on one node while completing successfully on others. Despite retries — including downtime mode — the issue persisted, pointing to a deeper inconsistency within the patching framework.


    Symptoms Observed

    • ADOP session status: FAILED
    • Patch applied successfully on some nodes, failed on admin node
    • Repeated failures even with restart=no, abandon=yes, and downtime mode
    • No immediate actionable error from standard logs

    Timeline of Events

    T0 -- Patch execution initiated (ADOP apply phase)
    T1 -- Failure observed on admin node
    T2 -- Retry using downtime mode -- Failure persists
    T3 -- ADOP session review shows inconsistent state
    T4 -- Internal metadata tables analyzed
    T5 -- Cleanup performed (tables + restart directory)
    T6 -- Patch re-executed -- Success across all nodes

    Investigation

    Step 1: Check ADOP Session State

    Query the ADOP session status to understand the current state across all nodes:

    -- Check current ADOP session status
    SELECT session_id, node_name, phase, status,
           start_date, end_date
    FROM applsys.ad_adop_sessions
    ORDER BY start_date DESC;
    
    -- Check apply phase status per node
    SELECT s.session_id, n.node_name, p.phase_code,
           p.status, p.start_date, p.end_date
    FROM applsys.ad_adop_sessions s,
         applsys.ad_adop_session_phases p,
         applsys.fnd_nodes n
    WHERE s.session_id = p.session_id
    AND p.node_id = n.node_id
    ORDER BY p.start_date DESC;

    The existing session showed status FAILED with the apply phase partially completed — a clear indicator of inconsistent execution state across nodes.

    Step 2: Check adalldefaults.txt

    Reviewed the defaults file for any relevant configuration:

    cat $APPL_TOP/admin/$TWO_TASK/adalldefaults.txt | grep -i missing
    -- Key parameter found:
    -- MISSING_TRANSLATED_VERSION = No

    Modifying and retrying with this parameter had no impact, confirming the issue was not translation-related.

    Step 3: Check Install Processes Table

    -- Check for stale entries in FND_INSTALL_PROCESSES
    SELECT COUNT(*) FROM applsys.fnd_install_processes;
    
    -- View stale entries in detail
    SELECT process_status, process_name, last_update_date
    FROM applsys.fnd_install_processes
    ORDER BY last_update_date DESC;
    
    -- Check AD_DEFERRED_JOBS
    SELECT COUNT(*) FROM applsys.ad_deferred_jobs;
    SELECT * FROM applsys.ad_deferred_jobs;

    Observation: FND_INSTALL_PROCESSES contained stale entries from the failed session. AD_DEFERRED_JOBS was empty.


    Root Cause

    The failure was caused by stale and inconsistent ADOP metadata tables — specifically APPLSYS.FND_INSTALL_PROCESSES and APPLSYS.AD_DEFERRED_JOBS. ADOP internally relies on these tables to track patch progress checkpoints, deferred job execution, and restart state management. When these tables retain entries from failed or incomplete sessions, ADOP assumes an incorrect execution state, leading to patch reconciliation failure, apply phase breakdown, and node-level inconsistencies.


    Resolution Steps

    Step 1: Backup Critical Tables

    -- Always backup before any cleanup
    CREATE TABLE applsys.fnd_install_processes_bak AS
    SELECT * FROM applsys.fnd_install_processes;
    
    CREATE TABLE applsys.ad_deferred_jobs_bak AS
    SELECT * FROM applsys.ad_deferred_jobs;
    
    -- Verify backups
    SELECT COUNT(*) FROM applsys.fnd_install_processes_bak;
    SELECT COUNT(*) FROM applsys.ad_deferred_jobs_bak;

    Step 2: Drop Stale Metadata Tables

    Dropping these tables forces ADOP to rebuild clean metadata during the next run:

    DROP TABLE applsys.fnd_install_processes;
    DROP TABLE applsys.ad_deferred_jobs;

    Step 3: Reset the Restart Directory

    The restart directory can silently preserve failure states. Back it up and create a fresh one:

    cd $APPL_TOP/admin/$TWO_TASK
    
    -- Backup existing restart directory
    mv restart restart_bkp_$(date +%Y%m%d)
    
    -- Create fresh restart directory
    mkdir restart
    
    -- Verify
    ls -la | grep restart

    Step 4: Re-run the Patch

    adop phase=apply \
         patches=<patch_id> \
         restart=no \
         abandon=yes \
         apply_mode=downtime

    The patch completed successfully across all nodes after the metadata cleanup.


    Before vs After

    ComponentBefore FixAfter Fix
    ADOP SessionFailedSuccessful
    Node ConsistencyPartialFull
    Restart BehaviorStuckClean
    Patch ExecutionIncompleteCompleted

    Key Takeaways

    • ADOP is state-driven — even when logs appear clean, internal metadata drives execution decisions
    • Partial success is a clue — if some nodes succeed and one fails, focus on local metadata, not the patch itself
    • The restart directory matters — it can silently preserve failure states and must be validated before retrying
    • Downtime mode is not a fix-all — even in downtime, ADOP still reads metadata tables; corruption persists unless cleaned
    • Always backup before cleanup — never drop tables without creating a backup first

    When NOT to Use This Approach

    Avoid applying this fix if the issue is caused by missing database patches (ETCC warnings), file system or permission issues, incorrect patch sequencing, or environment misconfiguration. Always validate the root cause before performing any metadata cleanup.


    This scenario highlights a subtle but critical behavior in ADOP — sometimes patch failures are not caused by the patch itself, but by what the system remembers about past attempts. By resetting stale metadata, we allow ADOP to re-evaluate the environment cleanly, leading to successful execution.

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.

  • Oracle EBS Login Issue: Real-Time Production Incident and Fix (OACORE + WebLogic Filter)


    📌 Introduction

    In Oracle E-Business Suite (EBS) environments, login failures are often perceived as simple application issues. However, in complex architectures, they can originate from multiple interacting layers across the application and middleware stack.

    In this blog, I’ll walk through a real-world production incident where an OACORE JVM issue combined with WebLogic security filtering resulted in complete login inaccessibility.

    This case highlights the importance of analyzing both performance and security layers together when troubleshooting critical application outages.


    ⚠️ Issue Summary

    • Users were unable to access the EBS login page
    • Pages were hanging or not loading
    • WebLogic console reported:
    Connection rejected, filter blocked Socket

    🔍 Initial Observation

    From the application server:

    • Load Average: ~10+

    👉 This indicated:

    • High CPU utilization
    • System under heavy stress
    • Potential JVM thread contention

    🔬 Detailed Analysis

    • One of the OACORE managed server JVMs became unresponsive
    • Long-running threads caused thread pool exhaustion
    • Incoming user requests began queueing

    At the same time:

    • WebLogic connection filter was actively enforcing access rules
    • Legitimate requests were being rejected under stressed conditions

    🧠 Understanding the Components

    OACORE (Application Layer)

    Handles:

    • Login requests
    • Forms processing
    • Core application logic

    If JVM threads are exhausted:
    👉 Requests queue → login hangs


    Oracle WebLogic Server Connection Filter

    EBS environments may use:

    oracle.apps.ad.tools.configuration.wls.filter.EBSConnectionFilterImpl

    This filter:

    • Enforces IP-based access control
    • Overrides default allow rules

    If misconfigured or stressed:
    👉 Legitimate traffic may be blocked


    🎯 Root Cause Analysis (RCA)

    The login issue was not caused by a single failure point, but by a combination of application tier resource exhaustion and restrictive middleware-level access control.

    • High CPU utilization and long-running threads caused one OACORE JVM to become unresponsive
    • Thread pool exhaustion led to request queuing, preventing new login requests from being processed

    Simultaneously:

    • The WebLogic connection filter (EBSConnectionFilterImpl) enforced strict access control policies
    • Under high load conditions, legitimate client requests were rejected with “filter blocked Socket”

    This interaction between performance degradation and security enforcement amplified the impact, resulting in complete login inaccessibility despite partial system availability.


    🛠️ Resolution Approach (Controlled & Safe)

    The resolution approach focused on stabilizing the JVM layer while validating and correcting middleware-level access controls in a controlled manner.


    🔹 Step 1: Identify Unresponsive JVM

    ps -ef | grep oacore

    ✔ Identify JVM with abnormal CPU or stuck behavior


    🔹 Step 2: Handle Stuck JVM (Controlled Action)

    ⚠️ Important Note:

    Forcefully terminating JVM processes should NOT be performed without validation.

    ✔ Recommended Approach:

    • Confirm the process is unresponsive
    • Ensure no critical transactions are running
    • Prefer controlled shutdown where possible

    ✔ Example (Only if fully unresponsive and approved):

    kill -9 <PID>

    👉 Node Manager can restart the JVM automatically after termination


    🔹 Step 3: Rolling Restart of OACORE

    admanagedsrvctl.sh stop oacore_server1
    admanagedsrvctl.sh start oacore_server1

    ✔ Ensures clean JVM state
    ✔ Restores thread pool balance


    🔹 Step 4: Validate WebLogic Connection Filter

    ⚠️ Important Note:

    The WebLogic connection filter is a security control and should NOT be disabled permanently.

    ✔ In this case:

    • Filter behavior was validated as part of troubleshooting
    • Temporary relaxation was used to confirm impact

    ✔ Recommended Approach:

    • Review allowed IP ranges
    • Validate filter configuration
    • Re-enable filter after correction

    The filter was re-enabled after validation and correction of configuration.


    🔹 Step 5: Restart Admin Server (If Required)

    adadminsrvctl.sh stop
    adadminsrvctl.sh start

    ✔ Ensures configuration changes are applied


    ✅ Final Outcome

    • System load reduced from ~10 → ~1.5
    • OACORE JVMs stabilized
    • Login page restored
    • Users successfully accessed the application

    📊 Key Learnings

    1. OACORE JVM issues can manifest as complete application outages due to thread exhaustion and request queuing.
    2. WebLogic connection filters can unintentionally block legitimate traffic, especially under high load conditions.
    3. Multi-layer failures (application + middleware + security) significantly amplify incident impact.
    4. System load metrics (CPU, load average) provide early indicators of JVM stress and should be monitored proactively.

    🧩 Preventive Measures

    • Monitor JVM thread utilization proactively
    • Review connection filter configurations periodically
    • Avoid peak concurrent load spikes
    • Implement alerting for high CPU / load conditions
    • Validate session and request handling patterns

    🔐 Governance Considerations

    • Avoid forceful JVM termination without proper validation
    • Do not disable security controls without understanding impact
    • Follow change management procedures in production
    • Coordinate with application and security teams before changes

    🏁 Conclusion

    This incident demonstrates that login failures in Oracle EBS are not always isolated to a single component but can result from complex interactions across application performance and middleware security layers.

    The combination of JVM resource exhaustion and connection filtering behavior created a compounded failure scenario, leading to complete login disruption.

    A structured, multi-layer troubleshooting approach—focused on performance, configuration, and governance—enabled effective resolution while minimizing risk.

    This reinforces the importance of analyzing both system behavior and security controls together when addressing critical production incidents.


    💡 Pro Tip

    When troubleshooting Oracle EBS login issues, always validate both:

    • JVM health (thread utilization, CPU load)
    • Middleware controls (connection filters, access rules)

    Ignoring either layer can lead to incomplete or misleading diagnosis.

  • Oracle EBS OACORE Server in FAILED_NOT_RESTARTABLE State: Real-Time Issue, RCA and Fix

    In Oracle E-Business Suite (EBS) environments, application tier stability is critical to ensure seamless user experience. However, there are scenarios where managed servers behave unexpectedly and require manual intervention. This post walks through a real-world production issue where an OACORE managed server entered a FAILED_NOT_RESTARTABLE state, its impact, root cause analysis, and how it was resolved.


    Environment Details

    • Oracle E-Business Suite: R12.2.x
    • Application Tier: WebLogic Managed Servers
    • Component Impacted: OACORE Server (oacore_server1)
    • Environment Type: Production

    Problem Statement

    An alert was received indicating oacore_server1 was in FAILED_NOT_RESTARTABLE state. Upon verification, the server was running but Node Manager could not auto-restart it.


    Key Observations

    Despite the OACORE server being in a failed state, the application remained accessible and functional — traffic was being handled by other OACORE servers. This is due to the multi-OACORE architecture with load balancing via OHS/Web tier. However, this creates a hidden risk: load redistribution increases pressure on remaining servers and can lead to cascading failures if not addressed promptly.


    Detailed Analysis

    • Managed server restart attempts failed during initialization
    • Bulk concurrent requests were actively running
    • CPU utilization spiked on the application tier
    • JVM resources were under pressure

    Understanding FAILED_NOT_RESTARTABLE

    In Oracle WebLogic Server, a managed server is marked as FAILED_NOT_RESTARTABLE after repeated unsuccessful restart attempts. This is a protective mechanism designed to prevent unstable restart loops when the server cannot recover successfully.


    Root Cause Analysis

    The OACORE managed server entered FAILED_NOT_RESTARTABLE state due to repeated startup failures following an unclean or resource-constrained shutdown. High CPU utilization and heavy concurrent workload placed JVM resources under pressure, preventing a clean restart cycle. Residual runtime artifacts (such as incomplete shutdown state or resource locks) prevented successful reinitialization, causing WebLogic to mark the server as FAILED_NOT_RESTARTABLE.


    Resolution

    cd $ADMIN_SCRIPTS_HOME
    ./admanagedsrvctl.sh stop oacore_server1
    ./admanagedsrvctl.sh start oacore_server1

    After the controlled restart, the server returned to RUNNING state with all deployments active and the application stable.


    Identify Inactive Forms Sessions

    Inactive sessions holding resources can contribute to JVM pressure. Use this query to identify them safely — do not terminate without proper validation and approvals:

    SELECT s.sid,
           s.serial#,
           s.username,
           s.status,
           s.program,
           s.machine,
           ROUND(s.last_call_et/3600,2) AS hours_inactive
    FROM v$session s
    WHERE s.status = 'INACTIVE'
    AND s.username = 'APPS'
    AND s.program LIKE 'frmweb%'
    AND s.last_call_et > 28800   -- 8 hours
    ORDER BY hours_inactive DESC;

    Reference only — do NOT execute without validation:

    ALTER SYSTEM KILL SESSION 'SID,SERIAL#' IMMEDIATE;

    Automate Session Monitoring

    Use this script to monitor inactive sessions every 8 hours via cron:

    #!/bin/bash
    export ORACLE_SID=your_sid
    export ORACLE_HOME=/path/to/oracle_home
    export PATH=$ORACLE_HOME/bin:$PATH
    
    sqlplus -s / as sysdba <<EOF
    SET LINES 200
    SET PAGES 200
    SELECT COUNT(*) AS inactive_sessions
    FROM v\$session
    WHERE status='INACTIVE'
    AND username='APPS'
    AND program LIKE 'frmweb%'
    AND last_call_et > 28800;
    EXIT;
    EOF
    # Crontab entry - every 8 hours
    0 */8 * * * /path/to/inactive_sessions.sh >> /tmp/inactive_sessions.log

    DBA Quick Commands

    -- Check system load
    top
    uptime
    ps -ef | grep oacore
    
    -- Check running concurrent requests
    SELECT request_id, phase_code, status_code
    FROM fnd_concurrent_requests
    WHERE phase_code = 'R';

    Key Takeaways

    • Application may appear healthy even when an OACORE server fails due to load balancing
    • FAILED_NOT_RESTARTABLE is a protective mechanism, not the root cause itself
    • Resource pressure and restart failures must be analyzed together
    • Controlled and governed actions are critical in production environments
    • Proactive session monitoring via automation helps prevent recurrence

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.