Syed Anwar Ahmed – Oracle DBA Blog

Tag: 12.2

  • Oracle EBS R12.2 Clone Redirects to PROD After Login: Diagnosing Stale Configuration References

    When you finish cloning an Oracle E-Business Suite R12.2 environment, the moment of truth is the login page. You hit the clone’s web entry URL, the page loads, you type your credentials… and the browser quietly throws you onto the production URL. On a multi-node app tier this is more than an annoyance — it means your freshly cloned, supposedly isolated environment is reaching back into PROD, and a careless tester could authenticate against the wrong system entirely.

    I ran into exactly this on a recent R12.2.11 clone built on Oracle Cloud Infrastructure: a three-node application tier sharing a single run/patch file system over an FSS (File Storage Service) NFS mount. The clone came up, services started, but every login attempt redirected to the PROD web entry host. The initial investigation uncovered two primary issues — phantom WebLogic managed servers carried over from the source domain, and a missing DNS A-record for the clone’s web entry hostname. Along the way I also identified several other clone-related configuration areas that can produce the same symptom, including stale profile options, OHS configuration remnants, load balancer redirects, and custom code references.

    This post walks through how the symptom presents, how to diagnose it cleanly, and the supported way to fix each cause.

    Throughout, I use placeholder names — clone-apps.example.com for the clone web entry host, prod-apps.example.com for production, and appnode1/2/3 for the three app-tier nodes. Substitute your own values.

    The architecture (why a shared file system matters here)

    The clone app tier looked like this:

    • Three application-tier nodes (appnode1, appnode2, appnode3).
    • A shared dual file system (fs1 run / fs2 patch) hosted on an OCI FSS NFS export, mounted identically on all three nodes.
    • A single WebLogic domain (EBS_domain_<SID>) living on that shared file system.

    The shared file system is the detail that makes the “phantom managed server” problem sticky. Because config.xml and the managed-server definitions physically live on the shared FSS mount, any leftover managed-server entries from the source domain are visible to every node at once. AutoConfig regenerates context-driven artifacts, but it does not, on its own, delete managed servers that no longer belong to the topology.

    The symptom

    After adcfgclone completed and the application services started:

    • The clone login page (https://clone-apps.example.com:<port>/OA_HTML/AppsLogin) rendered correctly.
    • On submitting credentials — or sometimes immediately on the redirect to the home page — the browser landed on https://prod-apps.example.com/....
    • The WebLogic Admin Console showed more managed servers than the three-node clone should have, several of them in an unreachable / shutdown state.

    Two independent problems reinforced each other: stale host references remained inside the cloned WebLogic topology, while the clone web entry hostname could not be resolved correctly. Either issue can cause redirect anomalies, but together they consistently redirected users back to PROD.

    First, rule out the context file

    Before investigating WebLogic, verify the clone context file itself doesn’t still contain production hostnames. AutoConfig can only generate correct configuration if the context values are correct, so everything downstream is built from here:

    grep -i "prod-apps.example.com" $CONTEXT_FILE
    grep -i "prod"                  $CONTEXT_FILE

    If PROD survives in the context, fix it there and re-run AutoConfig before chasing anything downstream — otherwise you’ll be debugging generated artifacts while the source of the bad values sits upstream.

    Root cause 1 — Phantom WebLogic managed servers

    A correctly provisioned three-node oacore service should have oacore_server1, oacore_server2, oacore_server3 (and the matching oafm, forms, forms-c4ws servers per node). The cloned domain carried extra managed servers that mapped to the source environment’s nodes — servers that pointed at listen addresses and host references belonging to PROD.

    These phantom servers do three harmful things:

    1. They keep PROD host references alive inside config.xml.
    2. They confuse the EBS service control and the Admin Server’s view of the cluster.
    3. They can answer (or fail to answer) requests in ways that surface PROD URLs.

    Diagnosing it

    Inventory what the domain actually contains versus what the topology should be:

    # What managed servers does the domain config believe in?
    grep -E "<name>|<listen-address>" \
      $EBS_DOMAIN_HOME/config/config.xml | grep -iE "oacore|oafm|forms|server"
    # What does EBS think the nodes are?
    sqlplus apps/<pwd> <<'EOF'
    COL node_name      FORMAT a20
    COL server_address FORMAT a18
    COL support_cp     FORMAT a10
    COL support_web    FORMAT a10
    COL support_admin  FORMAT a13
    SELECT node_name,
           NVL(server_address,'-')           AS server_address,
           NVL(support_cp,'-')               AS support_cp,
           NVL(support_web,'-')              AS support_web,
           NVL(support_admin,'-')            AS support_admin
    FROM   fnd_nodes
    ORDER  BY node_name;
    EOF

    Any managed server or fnd_nodes row that references a host which is not one of appnode1/2/3 is a phantom artifact from the source.

    It’s also worth checking the role assignments — sometimes the hostname is correct but node registrations are duplicated or carry the wrong roles:

    COL node_name    FORMAT a30
    COL support_db   FORMAT a10
    SELECT node_name,
           support_db,
           support_cp,
           support_web,
           support_admin
    FROM   fnd_nodes
    ORDER  BY node_name;

    Fixing it (the supported way)

    Do not hand-edit config.xml. EBS R12.2 ships a provisioning utility to add and delete managed servers cleanly, keeping the domain, AutoConfig, and the database registration in sync. Stop the affected services first, then delete each phantom server:

    # Stop the managed servers / services before topology changes
    $ADMIN_SCRIPTS_HOME/adstpall.sh apps/<apps_pwd>
    # Delete a phantom managed server (repeat per orphaned server / service type)
    perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
      ebs-delete-managedserver \
      -contextfile=$CONTEXT_FILE \
      -managedsrvname=oacore_server4 \
      -servicetype=oacore \
      -promptmsg=hide

    After removing every phantom server, run AutoConfig on each app node so the context, the domain, and fnd_nodes agree:

    $ADMIN_SCRIPTS_HOME/adautocfg.sh

    Then confirm fnd_nodes only lists the three real clone nodes, and the Admin Console only shows the expected per-node managed servers. If you find stale node rows after the clean-up, the standard FND_CONC_CLONE.SETUP_CLEAN → AutoConfig sequence on each tier is the canonical way to rebuild the node registration. Run it as APPS, then re-run AutoConfig on every tier:

    EXEC FND_CONC_CLONE.SETUP_CLEAN;
    COMMIT;

    (Always take that step with the DBA team’s sign-off on a shared environment.)

    Root cause 2 — The missing DNS A-record

    With the phantom servers gone, the redirect still misbehaved intermittently. The reason was simpler and entirely outside EBS: the clone’s web entry hostname had no DNS A-record.

    The login flow builds its target URL from the AutoConfig web entry variables. Check them:

    grep -E "s_webentryhost|s_webentrydomain|s_webentryurlprotocol|s_active_webport|s_url_protocol|s_login_page" \
      $CONTEXT_FILE

    It’s worth proving the generated login URL, not just the host variable — if AutoConfig hasn’t fully taken, these still show a PROD host and you’ll be chasing DNS for a problem that lives in the context:

    grep -i "webentry"   $CONTEXT_FILE
    grep -i "login_page" $CONTEXT_FILE

    The context correctly named clone-apps.example.com as s_webentryhost. But on the app nodes — and for clients — that name did not resolve:

    nslookup clone-apps.example.com
    # ** server can't find clone-apps.example.com: NXDOMAIN
    dig +short clone-apps.example.com
    # (empty)

    Because the clone web entry host was not resolvable, requests that relied on generated URLs could not consistently resolve back to the clone environment. At the same time, stale PROD references still lived inside the cloned WebLogic topology. Together these two conditions caused redirects to be generated using PROD host information, sending users away from the clone — which is why fixing the phantom servers alone wasn’t enough. To be clear, EBS does not contain any built-in mechanism that redirects a clone to production. Such redirects are almost always caused by stale configuration, profile options, WebLogic topology artifacts, load balancer settings, or DNS resolution — not by EBS doing anything magical. The browser is simply following a redirect built from stale host values. Name resolution has to work and the topology has to be clean.

    Fixing it

    Add an A-record for the clone web entry host in the appropriate DNS zone, pointing at the clone’s web-tier listen address (or its load balancer / OCI public-or-private IP, depending on your access path):

    clone-apps.example.com.    IN    A    10.x.x.x

    If a DNS change isn’t immediately possible and you only need the app nodes to resolve it for validation, a temporary /etc/hosts entry on each of appnode1/2/3 will confirm the theory — but a proper A-record is the real fix, because clients need to resolve it too:

    10.x.x.x    clone-apps.example.com    clone-apps

    Verify resolution from each node and re-test:

    for n in appnode1 appnode2 appnode3; do
      echo "== $n =="; ssh $n "nslookup clone-apps.example.com | tail -3"
    done

    Before concluding the issue persists, test using an incognito/private browser session or clear the browser cache. Browsers frequently cache redirects, cookies, and DNS information that can make a corrected environment appear unchanged.

    Root cause 3 — Stale profile option URLs (the other usual suspect)

    Even when DNS resolves and the WebLogic topology is clean, a clone can still throw users at PROD because profile option values were copied straight from the source. These are among the most common causes of a redirect-to-PROD, and they deserve a deliberate check rather than a passing glance. The usual culprits are APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, and ICX_FORMS_LAUNCHER.

    Target them directly:

    COL profile_option_name  FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
           fpov.profile_option_value
    FROM   fnd_profile_option_values fpov,
           fnd_profile_options_vl    fpo
    WHERE  fpov.profile_option_id = fpo.profile_option_id
    AND    fpo.profile_option_name IN
           ('APPS_WEB_AGENT',
            'APPS_FRAMEWORK_AGENT',
            'APPS_SERVLET_AGENT',
            'ICX_FORMS_LAUNCHER');

    Or sweep more broadly for any value still carrying a PROD host:

    COL profile_option_name  FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
           fpov.profile_option_value
    FROM   fnd_profile_option_values fpov,
           fnd_profile_options_vl    fpo
    WHERE  fpov.profile_option_id = fpo.profile_option_id
    AND    UPPER(fpov.profile_option_value) LIKE '%PROD%';

    Most of these are AutoConfig-managed, so the right fix is almost always to correct the context and re-run AutoConfig rather than to update the profile value by hand. Hand-updating a profile that AutoConfig owns just means it reverts on the next run. If AutoConfig keeps recreating the wrong value, fix the context file or web entry settings first; otherwise the next AutoConfig run simply reintroduces the problem.

    Other places a PROD reference hides: OHS and the OCI load balancer

    A surprising share of clone redirects originate outside the database and the WebLogic domain entirely — in the web tier configuration that AutoConfig generates, or in the load balancer sitting in front of it. Worth checking these early rather than last.

    Oracle HTTP Server (OHS). Generated OHS config — mod_wl_ohs.conf chief among them — can carry PROD host references. Rather than hard-coding version-specific paths, grep the web-tier and FMW homes broadly:

    grep -R "prod-apps.example.com" \
      $INST_TOP \
      $EBS_DOMAIN_HOME/config \
      $FMW_HOME \
      $ORACLE_HOME 2>/dev/null

    OCI load balancer / reverse proxy. If an OCI Load Balancer (or any reverse proxy) fronts the environment, the redirect can be introduced at that layer even after EBS is fully corrected. Verify:

    • Host header preservation — the backend should receive the clone host, not a rewritten PROD one.
    • Backend set configuration — backends point at the clone app nodes, not PROD.
    • SSL/TLS termination — the protocol and host the LB forwards match what AutoConfig expects (s_webentryurlprotocol, s_active_webport).
    • Redirect rules / rule sets — no listener rule is rewriting the host to PROD.

    I’ve seen an OCI LB listener keep sending users to PROD long after the EBS tier itself was spotless, purely because of a stale redirect rule on the listener.

    Custom code and JARs. Customizations are a notorious hiding place — hardcoded URLs in custom packages, JSPs, or Java survive every clone untouched by AutoConfig. Grep the custom homes too:

    grep -R "prod-apps.example.com" \
      $XX_TOP \
      $JAVA_TOP \
      $CUSTOM_TOP 2>/dev/null

    Validation checklist after remediation

    1. nslookup / dig resolves clone-apps.example.com from all three app nodes and from a client workstation.

    2. WebLogic Admin Console lists only the expected per-node managed servers, all in RUNNING state.

    3. fnd_nodes contains only appnode1/2/3 (check support_cp, support_web, support_admin); no PROD host references remain.

    4. No stale source context files are still registered. Old registrations sometimes survive a clone and cause confusion later during AutoConfig or service management:

    COL node_name    FORMAT a20
    COL ctx_file     FORMAT a60
    SELECT node_name,
           ctx_type,
           NVL(path,'-')  AS ctx_file,
           status
    FROM   fnd_oam_context_files
    WHERE  status = 'S'
    ORDER  BY node_name;

    In a clean clone, every active context file should belong to the clone environment. Any remaining source-environment context registration should be reviewed and removed before further AutoConfig runs.

    5. The profile options from Root cause 3 (APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, ICX_FORMS_LAUNCHER) all carry clone values, not PROD — re-run the targeted query from that section to confirm.

    6. The login page loads from the clone URL and — critically — the post-login redirect stays on clone-apps.example.com, never bouncing to prod-apps.example.com.

    7. Sweep the entire configuration for the PROD hostname, not just the context file. PROD remnants love to hide in OHS configs, mod_wl_ohs.conf, generated XML, and custom integrations:

    grep -R "prod-apps.example.com" \
      $INST_TOP \
      $EBS_DOMAIN_HOME/config \
      $FMW_HOME 2>/dev/null

    Anything this turns up needs to be corrected (and usually re-generated via AutoConfig) before users find it for you.

    8. No context file registered in the database still points at a PROD path. Stale PROD context registrations can cause odd behaviour long after the clone:

    COL node_name FORMAT a20
    COL path      FORMAT a70
    SELECT node_name,
           path
    FROM   fnd_oam_context_files
    WHERE  UPPER(path) LIKE '%PROD%';

    9. If an OCI Load Balancer or reverse proxy fronts the environment, verify its listener rules, backend sets, host-header forwarding, and SSL termination settings contain no PROD references.

    Lessons learned / a small post-clone checklist

    Cloning R12.2 onto a shared file system multi-node tier adds two checks that single-node clones let you skip:

    • Audit the managed-server topology immediately after adcfgclone. On a shared file system the source’s managed servers ride along inside config.xml. Delete phantoms with adProvisionEBS.pl ebs-delete-managedserver, never by editing XML, and re-run AutoConfig.
    • Resolve the web entry hostname before you trust the login page. A clone that “logs you into PROD” is very often a DNS problem wearing an EBS costume. Create the A-record as part of the clone runbook, not as a reaction to the redirect.
    • Treat a redirect-to-PROD as a stop-the-line event. Until both topology and name resolution are clean, assume the clone can still touch production and keep testers out.
    • Run FND_CONC_CLONE.SETUP_CLEAN before AutoConfig in cloned environments. This rebuilds node registrations and context metadata cleanly and helps prevent stale topology information from the source environment persisting in the clone. (On a shared environment, run it with the DBA team’s sign-off.)

    In this case, three unglamorous root causes — leftover managed servers, a missing DNS record, and stale profile option URLs — combined to produce the redirect. In practice, however, most redirect-to-PROD incidents come down to one broader problem: stale production references surviving the clone process. The fastest path to resolution is a systematic sweep of WebLogic, DNS, profile options, context files, OHS configuration, load balancers, and custom code until every production reference is gone.

    Have you hit a different flavour of the clone redirect? The web entry variables, SSL/load-balancer termination, and s_login_page overrides each have their own way of sending you to the wrong host — happy to compare notes in the comments.

  • When ADOP Remembers Too Much: Fixing Patch Failures Caused by Stale Metadata in Oracle EBS

    During an Oracle E-Business Suite ADOP patching cycle in a multi-node environment, the apply phase failed on one node while completing successfully on others. Despite retries — including downtime mode — the issue persisted, pointing to a deeper inconsistency within the patching framework.

    Symptoms Observed

    • ADOP session status: FAILED
    • Patch applied successfully on some nodes, failed on admin node
    • Repeated failures even with restart=no, abandon=yes, and downtime mode
    • No immediate actionable error from standard logs

    Timeline of Events

    T0 -- Patch execution initiated (ADOP apply phase)
T1 -- Failure observed on admin node
T2 -- Retry using downtime mode -- Failure persists
T3 -- ADOP session review shows inconsistent state
T4 -- Internal metadata tables analyzed
T5 -- Cleanup performed (tables + restart directory)
T6 -- Patch re-executed -- Success across all nodes

    Investigation

    Step 1: Check ADOP Session State

    Query the ADOP session status to understand the current state across all nodes:

    -- Check current ADOP session status
SELECT session_id, node_name, phase, status,
       start_date, end_date
FROM applsys.ad_adop_sessions
ORDER BY start_date DESC;

-- Check apply phase status per node
SELECT s.session_id, n.node_name, p.phase_code,
       p.status, p.start_date, p.end_date
FROM applsys.ad_adop_sessions s,
     applsys.ad_adop_session_phases p,
     applsys.fnd_nodes n
WHERE s.session_id = p.session_id
AND p.node_id = n.node_id
ORDER BY p.start_date DESC;

    The existing session showed status FAILED with the apply phase partially completed — a clear indicator of inconsistent execution state across nodes.

    Step 2: Check adalldefaults.txt

    Reviewed the defaults file for any relevant configuration:

    cat $APPL_TOP/admin/$TWO_TASK/adalldefaults.txt | grep -i missing
-- Key parameter found:
-- MISSING_TRANSLATED_VERSION = No

    Modifying and retrying with this parameter had no impact, confirming the issue was not translation-related.

    Step 3: Check Install Processes Table

    -- Check for stale entries in FND_INSTALL_PROCESSES
SELECT COUNT(*) FROM applsys.fnd_install_processes;

-- View stale entries in detail
SELECT process_status, process_name, last_update_date
FROM applsys.fnd_install_processes
ORDER BY last_update_date DESC;

-- Check AD_DEFERRED_JOBS
SELECT COUNT(*) FROM applsys.ad_deferred_jobs;
SELECT * FROM applsys.ad_deferred_jobs;

    Observation: FND_INSTALL_PROCESSES contained stale entries from the failed session. AD_DEFERRED_JOBS was empty.

    Root Cause

    The failure was caused by stale and inconsistent ADOP metadata tables — specifically APPLSYS.FND_INSTALL_PROCESSES and APPLSYS.AD_DEFERRED_JOBS. ADOP internally relies on these tables to track patch progress checkpoints, deferred job execution, and restart state management. When these tables retain entries from failed or incomplete sessions, ADOP assumes an incorrect execution state, leading to patch reconciliation failure, apply phase breakdown, and node-level inconsistencies.

    Resolution Steps

    Step 1: Backup Critical Tables

    -- Always backup before any cleanup
CREATE TABLE applsys.fnd_install_processes_bak AS
SELECT * FROM applsys.fnd_install_processes;

CREATE TABLE applsys.ad_deferred_jobs_bak AS
SELECT * FROM applsys.ad_deferred_jobs;

-- Verify backups
SELECT COUNT(*) FROM applsys.fnd_install_processes_bak;
SELECT COUNT(*) FROM applsys.ad_deferred_jobs_bak;

    Step 2: Drop Stale Metadata Tables

    Dropping these tables forces ADOP to rebuild clean metadata during the next run:

    DROP TABLE applsys.fnd_install_processes;
DROP TABLE applsys.ad_deferred_jobs;

    Step 3: Reset the Restart Directory

    The restart directory can silently preserve failure states. Back it up and create a fresh one:

    cd $APPL_TOP/admin/$TWO_TASK

-- Backup existing restart directory
mv restart restart_bkp_$(date +%Y%m%d)

-- Create fresh restart directory
mkdir restart

-- Verify
ls -la | grep restart

    Step 4: Re-run the Patch

    adop phase=apply \
     patches=<patch_id> \
     restart=no \
     abandon=yes \
     apply_mode=downtime

    The patch completed successfully across all nodes after the metadata cleanup.

    Before vs After

    ComponentBefore FixAfter Fix
    ADOP SessionFailedSuccessful
    Node ConsistencyPartialFull
    Restart BehaviorStuckClean
    Patch ExecutionIncompleteCompleted

    Key Takeaways

    • ADOP is state-driven — even when logs appear clean, internal metadata drives execution decisions
    • Partial success is a clue — if some nodes succeed and one fails, focus on local metadata, not the patch itself
    • The restart directory matters — it can silently preserve failure states and must be validated before retrying
    • Downtime mode is not a fix-all — even in downtime, ADOP still reads metadata tables; corruption persists unless cleaned
    • Always backup before cleanup — never drop tables without creating a backup first

    When NOT to Use This Approach

    Avoid applying this fix if the issue is caused by missing database patches (ETCC warnings), file system or permission issues, incorrect patch sequencing, or environment misconfiguration. Always validate the root cause before performing any metadata cleanup.

    This scenario highlights a subtle but critical behavior in ADOP — sometimes patch failures are not caused by the patch itself, but by what the system remembers about past attempts. By resetting stale metadata, we allow ADOP to re-evaluate the environment cleanly, leading to successful execution.

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.

  • Oracle EBS Login Issue: Real-Time Production Incident and Fix (OACORE + WebLogic Filter)

    📌 Introduction

    In Oracle E-Business Suite (EBS) environments, login failures are often perceived as simple application issues. However, in complex architectures, they can originate from multiple interacting layers across the application and middleware stack.

    In this blog, I’ll walk through a real-world production incident where an OACORE JVM issue combined with WebLogic security filtering resulted in complete login inaccessibility.

    This case highlights the importance of analyzing both performance and security layers together when troubleshooting critical application outages.

    ⚠️ Issue Summary

    • Users were unable to access the EBS login page
    • Pages were hanging or not loading
    • WebLogic console reported:
    Connection rejected, filter blocked Socket

    🔍 Initial Observation

    From the application server:

    • Load Average: ~10+

    👉 This indicated:

    • High CPU utilization
    • System under heavy stress
    • Potential JVM thread contention

    🔬 Detailed Analysis

    • One of the OACORE managed server JVMs became unresponsive
    • Long-running threads caused thread pool exhaustion
    • Incoming user requests began queueing

    At the same time:

    • WebLogic connection filter was actively enforcing access rules
    • Legitimate requests were being rejected under stressed conditions

    🧠 Understanding the Components

    OACORE (Application Layer)

    Handles:

    • Login requests
    • Forms processing
    • Core application logic

    If JVM threads are exhausted:
    👉 Requests queue → login hangs

    Oracle WebLogic Server Connection Filter

    EBS environments may use:

    oracle.apps.ad.tools.configuration.wls.filter.EBSConnectionFilterImpl

    This filter:

    • Enforces IP-based access control
    • Overrides default allow rules

    If misconfigured or stressed:
    👉 Legitimate traffic may be blocked

    🎯 Root Cause Analysis (RCA)

    The login issue was not caused by a single failure point, but by a combination of application tier resource exhaustion and restrictive middleware-level access control.

    • High CPU utilization and long-running threads caused one OACORE JVM to become unresponsive
    • Thread pool exhaustion led to request queuing, preventing new login requests from being processed

    Simultaneously:

    • The WebLogic connection filter (EBSConnectionFilterImpl) enforced strict access control policies
    • Under high load conditions, legitimate client requests were rejected with “filter blocked Socket”

    This interaction between performance degradation and security enforcement amplified the impact, resulting in complete login inaccessibility despite partial system availability.

    🛠️ Resolution Approach (Controlled & Safe)

    The resolution approach focused on stabilizing the JVM layer while validating and correcting middleware-level access controls in a controlled manner.

    🔹 Step 1: Identify Unresponsive JVM

    ps -ef | grep oacore

    ✔ Identify JVM with abnormal CPU or stuck behavior

    🔹 Step 2: Handle Stuck JVM (Controlled Action)

    ⚠️ Important Note:

    Forcefully terminating JVM processes should NOT be performed without validation.

    ✔ Recommended Approach:

    • Confirm the process is unresponsive
    • Ensure no critical transactions are running
    • Prefer controlled shutdown where possible

    ✔ Example (Only if fully unresponsive and approved):

    kill -9 <PID>

    👉 Node Manager can restart the JVM automatically after termination

    🔹 Step 3: Rolling Restart of OACORE

    admanagedsrvctl.sh stop oacore_server1
    admanagedsrvctl.sh start oacore_server1

    ✔ Ensures clean JVM state
    ✔ Restores thread pool balance

    🔹 Step 4: Validate WebLogic Connection Filter

    ⚠️ Important Note:

    The WebLogic connection filter is a security control and should NOT be disabled permanently.

    ✔ In this case:

    • Filter behavior was validated as part of troubleshooting
    • Temporary relaxation was used to confirm impact

    ✔ Recommended Approach:

    • Review allowed IP ranges
    • Validate filter configuration
    • Re-enable filter after correction

    The filter was re-enabled after validation and correction of configuration.

    🔹 Step 5: Restart Admin Server (If Required)

    adadminsrvctl.sh stop
    adadminsrvctl.sh start

    ✔ Ensures configuration changes are applied

    ✅ Final Outcome

    • System load reduced from ~10 → ~1.5
    • OACORE JVMs stabilized
    • Login page restored
    • Users successfully accessed the application

    📊 Key Learnings

    1. OACORE JVM issues can manifest as complete application outages due to thread exhaustion and request queuing.
    2. WebLogic connection filters can unintentionally block legitimate traffic, especially under high load conditions.
    3. Multi-layer failures (application + middleware + security) significantly amplify incident impact.
    4. System load metrics (CPU, load average) provide early indicators of JVM stress and should be monitored proactively.

    🧩 Preventive Measures

    • Monitor JVM thread utilization proactively
    • Review connection filter configurations periodically
    • Avoid peak concurrent load spikes
    • Implement alerting for high CPU / load conditions
    • Validate session and request handling patterns

    🔐 Governance Considerations

    • Avoid forceful JVM termination without proper validation
    • Do not disable security controls without understanding impact
    • Follow change management procedures in production
    • Coordinate with application and security teams before changes

    🏁 Conclusion

    This incident demonstrates that login failures in Oracle EBS are not always isolated to a single component but can result from complex interactions across application performance and middleware security layers.

    The combination of JVM resource exhaustion and connection filtering behavior created a compounded failure scenario, leading to complete login disruption.

    A structured, multi-layer troubleshooting approach—focused on performance, configuration, and governance—enabled effective resolution while minimizing risk.

    This reinforces the importance of analyzing both system behavior and security controls together when addressing critical production incidents.

    💡 Pro Tip

    When troubleshooting Oracle EBS login issues, always validate both:

    • JVM health (thread utilization, CPU load)
    • Middleware controls (connection filters, access rules)

    Ignoring either layer can lead to incomplete or misleading diagnosis.

  • Oracle EBS OACORE Server in FAILED_NOT_RESTARTABLE State: Real-Time Issue, RCA and Fix

    In Oracle E-Business Suite (EBS) environments, application tier stability is critical to ensure seamless user experience. However, there are scenarios where managed servers behave unexpectedly and require manual intervention. This post walks through a real-world production issue where an OACORE managed server entered a FAILED_NOT_RESTARTABLE state, its impact, root cause analysis, and how it was resolved.

    Environment Details

    • Oracle E-Business Suite: R12.2.x
    • Application Tier: WebLogic Managed Servers
    • Component Impacted: OACORE Server (oacore_server1)
    • Environment Type: Production

    Problem Statement

    An alert was received indicating oacore_server1 was in FAILED_NOT_RESTARTABLE state. Upon verification, the server was running but Node Manager could not auto-restart it.

    Key Observations

    Despite the OACORE server being in a failed state, the application remained accessible and functional — traffic was being handled by other OACORE servers. This is due to the multi-OACORE architecture with load balancing via OHS/Web tier. However, this creates a hidden risk: load redistribution increases pressure on remaining servers and can lead to cascading failures if not addressed promptly.

    Detailed Analysis

    • Managed server restart attempts failed during initialization
    • Bulk concurrent requests were actively running
    • CPU utilization spiked on the application tier
    • JVM resources were under pressure

    Understanding FAILED_NOT_RESTARTABLE

    In Oracle WebLogic Server, a managed server is marked as FAILED_NOT_RESTARTABLE after repeated unsuccessful restart attempts. This is a protective mechanism designed to prevent unstable restart loops when the server cannot recover successfully.

    Root Cause Analysis

    The OACORE managed server entered FAILED_NOT_RESTARTABLE state due to repeated startup failures following an unclean or resource-constrained shutdown. High CPU utilization and heavy concurrent workload placed JVM resources under pressure, preventing a clean restart cycle. Residual runtime artifacts (such as incomplete shutdown state or resource locks) prevented successful reinitialization, causing WebLogic to mark the server as FAILED_NOT_RESTARTABLE.

    Resolution

    cd $ADMIN_SCRIPTS_HOME
./admanagedsrvctl.sh stop oacore_server1
./admanagedsrvctl.sh start oacore_server1

    After the controlled restart, the server returned to RUNNING state with all deployments active and the application stable.

    Identify Inactive Forms Sessions

    Inactive sessions holding resources can contribute to JVM pressure. Use this query to identify them safely — do not terminate without proper validation and approvals:

    SELECT s.sid,
       s.serial#,
       s.username,
       s.status,
       s.program,
       s.machine,
       ROUND(s.last_call_et/3600,2) AS hours_inactive
FROM v$session s
WHERE s.status = 'INACTIVE'
AND s.username = 'APPS'
AND s.program LIKE 'frmweb%'
AND s.last_call_et > 28800   -- 8 hours
ORDER BY hours_inactive DESC;

    Reference only — do NOT execute without validation:

    ALTER SYSTEM KILL SESSION 'SID,SERIAL#' IMMEDIATE;

    Automate Session Monitoring

    Use this script to monitor inactive sessions every 8 hours via cron:

    #!/bin/bash
export ORACLE_SID=your_sid
export ORACLE_HOME=/path/to/oracle_home
export PATH=$ORACLE_HOME/bin:$PATH

sqlplus -s / as sysdba <<EOF
SET LINES 200
SET PAGES 200
SELECT COUNT(*) AS inactive_sessions
FROM v\$session
WHERE status='INACTIVE'
AND username='APPS'
AND program LIKE 'frmweb%'
AND last_call_et > 28800;
EXIT;
EOF
    # Crontab entry - every 8 hours
0 */8 * * * /path/to/inactive_sessions.sh >> /tmp/inactive_sessions.log

    DBA Quick Commands

    -- Check system load
top
uptime
ps -ef | grep oacore

-- Check running concurrent requests
SELECT request_id, phase_code, status_code
FROM fnd_concurrent_requests
WHERE phase_code = 'R';

    Key Takeaways

    • Application may appear healthy even when an OACORE server fails due to load balancing
    • FAILED_NOT_RESTARTABLE is a protective mechanism, not the root cause itself
    • Resource pressure and restart failures must be analyzed together
    • Controlled and governed actions are critical in production environments
    • Proactive session monitoring via automation helps prevent recurrence

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.