Tag: AutoConfig

  • Oracle EBS R12.2 ADOP: MWA Template Conflict (AC-00058), a Hung ICM at Cutover, and a Node That Wasn’t There

    Alhamdulillah. This post covers one online patching cycle on an Oracle EBS 12.2 DEV environment (AD/TXK C.Delta.17, two application nodes on a shared file system). Applying a single patch turned into three separate problems:

    1. The apply phase failed because AutoConfig aborted on MWA custom templates.
    2. Someone asked whether a third application node had been missed by the patch.
    3. During cutover, the Internal Concurrent Manager (ICM) did not respond to shutdown.

    Each is short on its own. Together they show why it pays to check evidence before you act on a quick instruction. Hostnames, paths and custom package names below are anonymized.

    Problem 1: apply fails on AutoConfig

    The apply phase completed on the admin node but failed on the second node (the concurrent manager node) at this step:

    EVENT        Running AutoConfig on patch edition file system.
    UNEXPECTED   Error occurred running "sh .../ad/12.0.0/bin/adconfig.sh contextfile=.../fs2/inst/apps/<CONTEXT>/appl/admin/<CONTEXT>.xml -parallel ..."
    UNEXPECTED   Error occurred while executing AutoConfig on current node.
    UNEXPECTED   Apply phase has failed.

    The admin node then reported its own error from txkADOPEvalSrvStatus.pl. That error is only a consequence: adop could not continue because one node had failed. The real error was in the AutoConfig log of the failed node:

    [ VERSION CONFLICTS INFORMATION ]
    Template shipped by Oracle : .../fs2/EBSapps/appl/mwa/12.0.0/admin/template/mwa.cfg(version: 120.2.12020000.12)
    Custom template            : .../fs2/EBSapps/appl/mwa/12.0.0/admin/template/custom/mwa.cfg(version: 120.2.12020000.8)
    [ ERROR_REPORT ]
    AC-00058: Error: Unable to determine the version information for file - .../mwa/12.0.0/admin/template/custom/mwactl.sh; hence nothing can be said about its version conflicts.
    Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

    Two different problems in the same directory:

    • mwa.cfg: a normal version conflict. The patch delivered version .12 of the template, and the custom copy was still at .8.
    • mwactl.sh: AC-00058, which is not a version conflict. AutoConfig could not read a version from the custom file at all.

    Why AC-00058 happened

    The $Header line in the custom mwactl.sh named a different file:

    # $Header: mwaexpctl.sh 120.10.12020000.2 2017/03/14 06:40:25 sowu ship $

    At some point a copy of a related control script had been placed in custom/ under the name mwactl.sh. The header still carried the old file name, so AutoConfig could not match it to the shipped mwactl.sh. Correcting the file name in the header resolved the error. The shipped mwactl.sh was still at 120.10.12020000.2, so after the correction the versions matched.

    Check what the custom file actually does before replacing it

    The fastest fix would have been to replace both custom files with the shipped templates. A diff against the shipped mwactl.sh showed why that would have been a mistake for this file:

    cd $MWA_TOP/admin/template
    diff mwactl.sh custom/mwactl.sh
    < JAVA="%s_afjvaprg%  -Doracle.apps.mwa=${MWA_TOP} ..."
    ---
    > JAVA="${ADMIN_SCRIPTS_HOME}/java.sh  -Doracle.apps.mwa=${MWA_TOP} ... -javaagent:${AF_JLIB}/aspectjweaver.jar"
    
    < CLASSPATH="${AF_CLASSPATH}"
    ---
    > CLASSPATH="${AF_JLIB}/mwaexp.jar:${AF_CLASSPATH}"
    
    <           ${JAVA} oracle.apps.mwa.presentation.telnet.Listener "$@" &
    ---
    >           ${JAVA} XX.oracle.apps.mwa.presentation.telnet.Listener2 "$@" &

    The custom file starts a custom MWA telnet listener, with its own jar and a Java agent. If it had been replaced, AutoConfig would have generated the standard mwactl.sh and the custom listener would no longer have started. Nothing in the patch output would have flagged this. So mwactl.sh was kept, and only its header was corrected.

    mwa.cfg was different. Its only differences from the shipped file were two settings:

    mwa.OutOfOrderScanEnabled=Yes
    mwa.GlobalBarcodeDelimiter=64

    The usual approach is to rebase the custom copy on the new shipped version and re-apply the real customizations. In this case the team lead decided to replace the custom file with the shipped one and not carry the two settings forward. That is a functional decision, so it was recorded with the backup file name. If scanner users report problems with out-of-order scans or barcode delimiters, those two lines are the first place to look.

    The fix

    The file system was shared between both application nodes, so the change was made once. Before trusting that, confirm it: identical df figures on two nodes suggest a shared disk, but a test file created on one node and listed on the other proves it.

    cd $MWA_TOP/admin/template          # patch edition (fs2) path
    TS=$(date +%Y%m%d_%H%M%S)
    cp -p custom/mwa.cfg custom/mwa.cfg.pre_replace_$TS
    cksum custom/mwa.cfg custom/mwa.cfg.pre_replace_$TS
    
    cp -p mwa.cfg custom/mwa.cfg
    
    grep -n '\$Header' mwa.cfg custom/mwa.cfg custom/mwactl.sh
    diff mwa.cfg custom/mwa.cfg && echo "IDENTICAL"
    mwa.cfg:23:# $Header: mwa.cfg 120.2.12020000.12 2026/09/09 12:32:24 ukoneti ship $
    custom/mwa.cfg:23:# $Header: mwa.cfg 120.2.12020000.12 2026/09/09 12:32:24 ukoneti ship $
    custom/mwactl.sh:14:# $Header: mwactl.sh 120.10.12020000.2 2017/03/14 06:40:25 sowu ship $
    IDENTICAL

    Before restarting, I checked the whole patch file system for other custom files without a header:

    find $PATCH_BASE/EBSapps/appl -path "*admin/template/custom/*" -type f \
      ! -name "*.bkp_*" ! -name "*.pre_replace_*" \
      -exec sh -c 'grep -q "\$Header" "$1" || echo "NO HEADER: $1"' _ {} \;

    It found four files in the MWA custom/ directory: a text file, a log, an .ini file and a file named devnull. None of them has a matching template in the parent directory, and the original AutoConfig error had not listed any of them. AutoConfig only compares custom files that have a shipped counterpart, so these were left in place. The .ini file looked like part of the same custom extension.

    Restart the apply

    adop phase=apply patches=<patch_number> restart=yes

    The restart picked up the existing session, skipped the node that had already finished, and ran AutoConfig on the patch file system of the failed node. It completed on both nodes in about six and a half minutes. The status lines Session status: FAILED and ATLEAST ONE PATCH IS ALREADY APPLIED at the start of a restart come from the previous run and are expected.

    After the apply, I checked that the generated script in the patch edition still started the custom listener:

    grep -E "XX|mwaexp.jar" $PATCH_BASE/inst/apps/<CONTEXT>/admin/scripts/mwactl.sh
    CLASSPATH="${AF_JLIB}/mwaexp.jar:${AF_CLASSPATH}"
                    if ${JAVA} XX.oracle.apps.mwa.presentation.telnet.GenerateLauncher start
                ${JAVA} XX.oracle.apps.mwa.presentation.telnet.Listener2 "$@" &

    One note on adop -status: after a successful apply, the APPLY phase shows ACTIVE, not COMPLETED. The apply phase stays open so more patches can be added to the cycle. It changes to COMPLETED at cutover.

    Problem 2: the node that wasn’t there

    Before finalize, a question came in: had the patch been applied on a third application node, an external-facing iSupplier server? The suggested fix was to copy the patch there and run:

    adop phase=apply patches=<patch_list> allnodes=no action=nodb

    That command is the right recovery for a registered node that failed or was unavailable during a cycle. But the adop output had only ever mentioned two nodes, so I checked what the database knew first:

    SET LINES 200 PAGES 100
    COL node_name FORMAT A25
    SELECT NVL(node_name,'-') node_name FROM adop_valid_nodes ORDER BY 1;
    
    COL support_web   FORMAT A4
    COL support_cp    FORMAT A4
    COL support_forms FORMAT A5
    COL status        FORMAT A6
    SELECT NVL(node_name,'-') node_name, NVL(support_web,'-') support_web,
           NVL(support_cp,'-') support_cp, NVL(support_forms,'-') support_forms,
           NVL(status,'-') status
    FROM   fnd_nodes ORDER BY 1;
    
    COL path FORMAT A90
    SELECT NVL(node_name,'-') node_name, NVL(status,'-') status, NVL(path,'-') path
    FROM   fnd_oam_context_files
    WHERE  LOWER(node_name) LIKE '%<node_pattern>%';

    adop_valid_nodes listed only the two nodes in the session. fnd_nodes listed those two, the database node and the standard non-server entries. fnd_oam_context_files returned no rows for the third server. Its patch directory also had nothing newer than a patch staged months earlier, and no adop processes were running on it.

    So the server was not registered in this instance at all. adop had not skipped it. Running an apply there would have failed node validation, and the patch did not need to go there as part of this cycle. The cycle went ahead with the two registered nodes.

    One small thing from this step: in this AD/TXK level, adop_valid_nodes has no node_type column. The columns are NODE_NAME, CONTEXT_NAME, ZD_EDITION_NAME and ZD_SYNC. Run DESC before writing queries against adop tables.

    Problem 3: the ICM would not go down at cutover

    Finalize completed in about eight minutes. The only running concurrent request before cutover was the Online Patching In Progress request (ADZDPATCH), which adop submits for every cycle and cancels itself during cutover. Do not cancel it manually.

    Cutover was run with services left down afterwards:

    adop phase=cutover mtrestart=no

    It then stopped here:

    EVENT        Submitting request for Internal Concurrent Manager shutdown.
    EVENT        Cancelling ADZDPATCH concurrent request
    EVENT        Waiting for Internal Concurrent Manager to go down.

    An ICM shutdown normally waits for running requests to finish, so the first check was for running requests. There were none. The managers were still holding processes, though:

    CONCURRENT_QUEUE_NAME          RUNNING_PROCESSES MAX_PROCESSES
    ------------------------------ ----------------- -------------
    FNDCPOPP                                       5             5
    FNDICM                                         1             0
    FNDSM_<NODE1>                                  1             0
    FNDSM_<NODE2>                                  1             0
    IEXCONMGR                                      1             1
    PASMGR                                         1             1
    STANDARD                                       1            60

    The ICM log on the concurrent manager node showed the process monitor running every 30 seconds. Each pass listed a request in state Q that never progressed:

    Request  : <request_id>
    Priority : 0
    Program  : 0/5
    State    : Q
    
                       Process monitor session started : 04-NOV-2026 05:01:40
                         Process monitor session ended : 04-NOV-2026 05:01:40

    A manual adcmctl.sh stop only submitted a second shutdown request, which queued behind the first. adcmctl.sh status still reported the ICM as active more than an hour after cutover started. The ICM database session was not blocked:

    COL program  FORMAT A20
    COL event    FORMAT A35
    COL blocker  FORMAT A10
    SELECT s.sid, RPAD(NVL(s.program,'-'),20) program, RPAD(NVL(s.event,'-'),35) event,
           NVL(TO_CHAR(s.blocking_session),'-') blocker, s.seconds_in_wait
    FROM   v$session s, v$process p
    WHERE  s.paddr = p.addr
    AND    s.process = '<icm_os_pid>';
      SID PROGRAM              EVENT                               BLOCKER    SECONDS_IN_WAIT
    ----- -------------------- ----------------------------------- ---------- ---------------
     3511 ICM@<NODE2>          SQL*Net message from client         -                       16

    So the ICM was alive and idle, not waiting on a lock, and not acting on its shutdown requests. I could not establish why from the available logs. With no running requests and the cutover blocked, the team lead approved killing the remaining concurrent manager processes on the concurrent manager node: the ICM, its startmgr parent shell, the Service Manager, the Output Post Processor JVMs and the remaining manager processes. Once they were gone, adop saw the ICM as down and the cutover carried on.

    Two things to handle if you end up here:

    • Leave adop alone. Kill the manager processes, never the adop or adopmon session.
    • Clear the queued shutdown requests before the managers start again. If they are still pending, the new ICM can pick them up and shut itself down straight after startup.
    SELECT request_id, phase_code, status_code
    FROM   fnd_concurrent_requests
    WHERE  request_id IN (<shutdown_request_1>, <shutdown_request_2>);

    Option worth knowing: cm_wait. Cutover accepts a cm_wait parameter: the maximum number of minutes adop waits for the concurrent managers to shut down before it stops them forcefully. Without it, cutover waits as long as the managers take. For a cycle like this one, the command would be:

    adop phase=cutover cm_wait=3 mtrestart=no

    A forced shutdown also ends any request that is still running when the timer expires, so agree the value with the application team before the cutover window. I did not use cm_wait in this cycle, so I can’t say whether it would have cleared this particular hang.

    What normal cutover output looks like after that

    A few lines in the rest of the cutover look alarming but are expected:

    • DB Cutover did not complete on Admin node ... Will wait for another minute and check again. on the second node. It polls every minute until the admin node finishes the database cutover.
    • Entered into KILL SESSIONS LOOP inside RENAME_AW. adop clears sessions on the OLAP analytic workspace before renaming it, and the next lines show it finishing.
    • ORA-00028: your session has been killed right after Switching to Patch Edition. adop kills old-edition sessions, including its own, and reconnects. The log itself says this is normal.

    Cutover completed on both nodes in about 1 hour 22 minutes. Most of that was the wait on the ICM.

    After cutover

    With mtrestart=no, services stay down until you start them. The order I followed:

    1. Re-source the run environment and confirm $RUN_BASE now points to the other file system.
    2. Confirm no manager processes from the old run edition are left on either node.
    3. Clear any pending shutdown requests.
    4. Start services on the admin node, then the second node.
    5. Check the ICM, the custom MWA listener process, front-end login and a test concurrent request.
    6. Run adop phase=cleanup, then adop phase=fs_clone so the corrected custom templates reach the other file system. Until then, an AutoConfig run on the old file system would hit the same version conflict.

    In this cycle the post-cutover steps were completed and the cleanup phase finished successfully.

    Lessons

    • Read the AutoConfig log, not the adop summary. The admin node error pointed at a status script. The real cause was two lines in the failed node’s adconfig.log.
    • A version conflict and AC-00058 are different problems. One means the custom template is older. The other means AutoConfig cannot read a version at all, often because the $Header line is missing or names a different file.
    • Diff before you replace. “Back up and replace with the shipped file” was correct for mwa.cfg and would have quietly broken the custom MWA listener if applied to mwactl.sh.
    • Check node registration before patching a node. adop_valid_nodes, fnd_nodes and fnd_oam_context_files answer “was this node missed?” in under a minute.
    • A hung ICM at cutover needs evidence first, then approval. Running requests, queue counts, the ICM log and the ICM session’s wait event together showed there was nothing to wait for. I could not find the cause, so I have not claimed one.

    ما شاء الله, a single patch, three different lessons. If you have seen an ICM ignore its shutdown requests during cutover and found the cause, I would like to hear about it in the comments.

  • Oracle EBS R12.2 AutoConfig Template Version Conflict After Patching: How to Preserve Customizations

    Alhamdulillah, sharing another real troubleshooting session — this one from an Oracle E-Business Suite R12.2 multi-node environment. After applying a patch, AutoConfig aborted on the primary application node with a template version conflict. This post walks through the exact error, how I identified which template entries were genuine customizations versus Oracle-shipped content, and the correct way to rebase a custom template so AutoConfig completes cleanly.

    Environment

    • Oracle E-Business Suite R12.2.13, AD/TXK Delta 15
    • Two application nodes: appsnode1 (primary/admin) and webnode1 (secondary web node)
    • Database: Oracle 19c on dbnode1
    • Third-party customization: More4Apps servlet registered in oacore web.xml

    The Error

    After patching, running AutoConfig on the run file system failed immediately:

    [applmgr@appsnode1 scripts]$ sh adautocfg.sh
    Enter the APPS user password:
    The log file for this session is located at: .../admin/log/<timestamp>/adconfig.log
    wlsDomainName: EBS_domain
    WLS Domain Name is VALID.
    AutoConfig is configuring the Applications environment...
    AutoConfig will consider the custom templates if present.
    Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

    The log pinpointed the conflicting template:

    [ FND_TOP ]
    
    [ VERSION CONFLICTS INFORMATION ]
    Template shipped by oracle is having a version different than the template lying in custom directory.
    Template shipped by Oracle : $FND_TOP/admin/template/oacore_web_xml_FMW.tmp (version: 120.17.12020000.33)
    Custom template            : $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp (version: 120.17.12020000.31)
    Please resolve the differences between the two templates or refer to the Oracle E-Business Suite Setup Guide for further details.

    What happened is straightforward: the patch delivered a newer version of oacore_web_xml_FMW.tmp (120.17.12020000.33), but a custom copy of the older version (120.17.12020000.31) exists under $FND_TOP/admin/template/custom. AutoConfig detects the mismatch and refuses to run, because blindly using the stale custom template would silently discard whatever Oracle changed in the new version.

    Root Cause Analysis: What Is Actually a Customization?

    This is the critical analysis step. Before touching anything, you need to answer one question precisely: which lines in the custom template are genuine customizations, and which are just old Oracle-shipped content?

    First, take a backup and diff the new shipped template against the custom one:

    cd $FND_TOP/admin/template
    
    cp -p custom/oacore_web_xml_FMW.tmp \
          custom/oacore_web_xml_FMW.tmp.pre_patch_25Jul2026
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

    The diff showed three differences beyond the header:

    1. A m4aServlet servlet and servlet-mapping (More4Apps) present only in the custom template
    2. A SyncServlet servlet and servlet-mapping present only in the custom template
    3. A RequestAuditReport (QP) servlet present only in the new Oracle template

    At first glance it looks like there are two customizations to carry forward: m4aServlet and SyncServlet. This is where a wrong assumption would corrupt the template. To verify, I compared the old shipped template against the new shipped template. In an R12.2 dual file system this is easy — the other file system still had the previous patch level, so both shipped versions were available:

    diff -u \
      <old_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp \
      <new_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp

    Key excerpt from the shipped-vs-shipped diff:

    -   <servlet>
    -     <servlet-name>SyncServlet</servlet-name>
    -     <servlet-class>oracle.apps.jtf.cac.sync.transport.SyncServlet</servlet-class>
    -  </servlet>
    ...
    +     <servlet>
    +         <servlet-name>RequestAuditReport</servlet-name>
    +         <servlet-class>oracle.apps.qp.servlet.RequestAuditReport</servlet-class>
    +    </servlet>

    This settled the question:

    • SyncServlet was Oracle-shipped content in version .31 and is no longer present in the .33 shipped template. It was never a customer customization, so it should NOT be carried forward.
    • RequestAuditReport is new Oracle-shipped content in .33 — it must be retained.
    • m4aServlet was not present in either shipped template version compared — it exists only in the custom directory, which identifies it as the genuine site-specific customization.

    If I had merged the old custom template on top of the new one, I would have reintroduced a servlet that is no longer present in Oracle’s newer shipped template. This is why “just copy your custom entries into the new template” advice found on many forums is dangerous — verify against the shipped versions first.

    The Fix: Rebase the Custom Template

    The correct approach: replace the custom template with the new shipped version, then re-apply only the genuine customization.

    cd $FND_TOP/admin/template
    
    # Rebase custom template on the new shipped version
    cp -p oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    # Re-insert the only genuine customization (m4aServlet)
    sed -i '/<!-- FND Servlets -->/a\
    \
      <servlet>\
        <servlet-name>m4aServlet</servlet-name>\
        <servlet-class>com.more4apps.r12.servlet.XmlServlet</servlet-class>\
      </servlet>\
    \
      <servlet-mapping>\
        <servlet-name>m4aServlet</servlet-name>\
        <url-pattern>/m4aServlet/*</url-pattern>\
      </servlet-mapping>\
    ' custom/oacore_web_xml_FMW.tmp

    This matches the procedure in the Oracle E-Business Suite Setup Guide: when a patch delivers a newer version of a template you have customized, copy the new Oracle template into the custom directory and reapply the still-required customizations to that copy — never the other way around.

    Verify before rerunning AutoConfig — the headers must match, and the only diff must be the customization:

    grep '\$Header' oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    oacore_web_xml_FMW.tmp:        version 120.17.12020000.33
    custom/oacore_web_xml_FMW.tmp: version 120.17.12020000.33
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    --- only the m4aServlet servlet + mapping block should appear ---

    As an additional best-practice validation, Oracle recommends running the AutoConfig configuration-check utility to preview the impact of template changes before the actual AutoConfig run:

    $AD_TOP/bin/adchkcfg.sh contextfile=$CONTEXT_FILE

    Rerun AutoConfig:

    cd $ADMIN_SCRIPTS_HOME
    sh adautocfg.sh
    
    ...
    Configuring templates from all of the product tops...
            Configuring AD_TOP........COMPLETED
            Configuring FND_TOP.......COMPLETED
            ...
    AutoConfig completed successfully.

    A Bonus Finding on the Second Node

    While repeating the check on the secondary web node, the header comparison came back clean — custom template already at .33 — but something was off:

    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp
    (no output)
    
    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp_ORIG
    61:    <servlet-name>m4aServlet</servlet-name>

    The current custom template matched the newer shipped version, but the m4aServlet customization was missing. Interestingly, the _ORIG backup still contained the customization. This indicated that during an earlier template update, the customization was not carried forward. AutoConfig had been running fine and web.xml was being generated — just without the third-party servlet.

    I re-inserted the m4aServlet block using the same sed command, verified the diff, and ran AutoConfig on that node as well:

    AutoConfig completed successfully.

    This is the real lesson of the post. A template version conflict at least fails loudly. A customization that is accidentally omitted during a template rebase may not — AutoConfig can complete successfully while the generated configuration no longer contains the required customization, and you find out later when the third-party tool stops working after a bounce.

    An Additional Message Observed

    The AutoConfig log in this run also contained:

    ECC not enabled, setting FND_ECC_ENABLED to FALSE
    [ FND_ECC_ENABLED ]
    INFO : Error updating/creating profile option value.

    In this case Enterprise Command Center is not configured, AutoConfig continued past this message, and the run ultimately exited with status 0. Always evaluate such messages in the context of your own environment rather than assuming every INFO : Error entry can be ignored.

    Post-Fix Checklist

    1. Confirm the customization landed in the generated web.xml (get the target path from the adconfig log and grep for your servlet).
    2. Bounce the application services (at least the oacore managed servers) so the new web.xml is deployed.
    3. Test the customization end to end — for More4Apps, hit the servlet URL from the wizard.
    4. If your patching cycle requires it, regenerate appsutil.zip (perl $AD_TOP/bin/admkappsutil.pl) and refresh the database tier.
    5. Repeat the custom template verification on every application node — as seen above, nodes can drift.

    Key Takeaways

    • Never resolve this error by copying the old custom template over the new shipped one, and never blindly merge either. The Oracle-documented procedure is to copy the new shipped template into the custom directory and re-apply only verified customizations to that copy.
    • Use the dual file system to your advantage: diff old-shipped vs new-shipped to separate Oracle’s changes from your customizations.
    • After any template rebase, diff the shipped and custom templates — the output should contain nothing but your customizations.
    • Audit custom templates on all nodes periodically. A missing customization may not trigger an AutoConfig error, allowing the generated configuration to differ from what you expect.

    I hope this helps someone facing the same conflict. Feel free to leave a comment if you have questions.


    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS R12.2 Clone Redirects to PROD After Login: Diagnosing Stale Configuration References

    When you finish cloning an Oracle E-Business Suite R12.2 environment, the moment of truth is the login page. You hit the clone’s web entry URL, the page loads, you type your credentials… and the browser quietly throws you onto the production URL. On a multi-node app tier this is more than an annoyance — it means your freshly cloned, supposedly isolated environment is reaching back into PROD, and a careless tester could authenticate against the wrong system entirely.

    I ran into exactly this on a recent R12.2.11 clone built on Oracle Cloud Infrastructure: a three-node application tier sharing a single run/patch file system over an FSS (File Storage Service) NFS mount. The clone came up, services started, but every login attempt redirected to the PROD web entry host. The initial investigation uncovered two primary issues — phantom WebLogic managed servers carried over from the source domain, and a missing DNS A-record for the clone’s web entry hostname. Along the way I also identified several other clone-related configuration areas that can produce the same symptom, including stale profile options, OHS configuration remnants, load balancer redirects, and custom code references.

    This post walks through how the symptom presents, how to diagnose it cleanly, and the supported way to fix each cause.

    Throughout, I use placeholder names — clone-apps.example.com for the clone web entry host, prod-apps.example.com for production, and appnode1/2/3 for the three app-tier nodes. Substitute your own values.


    The architecture (why a shared file system matters here)

    The clone app tier looked like this:

    • Three application-tier nodes (appnode1, appnode2, appnode3).
    • A shared dual file system (fs1 run / fs2 patch) hosted on an OCI FSS NFS export, mounted identically on all three nodes.
    • A single WebLogic domain (EBS_domain_<SID>) living on that shared file system.

    The shared file system is the detail that makes the “phantom managed server” problem sticky. Because config.xml and the managed-server definitions physically live on the shared FSS mount, any leftover managed-server entries from the source domain are visible to every node at once. AutoConfig regenerates context-driven artifacts, but it does not, on its own, delete managed servers that no longer belong to the topology.


    The symptom

    After adcfgclone completed and the application services started:

    • The clone login page (https://clone-apps.example.com:<port>/OA_HTML/AppsLogin) rendered correctly.
    • On submitting credentials — or sometimes immediately on the redirect to the home page — the browser landed on https://prod-apps.example.com/....
    • The WebLogic Admin Console showed more managed servers than the three-node clone should have, several of them in an unreachable / shutdown state.

    Two independent problems reinforced each other: stale host references remained inside the cloned WebLogic topology, while the clone web entry hostname could not be resolved correctly. Either issue can cause redirect anomalies, but together they consistently redirected users back to PROD.


    First, rule out the context file

    Before investigating WebLogic, verify the clone context file itself doesn’t still contain production hostnames. AutoConfig can only generate correct configuration if the context values are correct, so everything downstream is built from here:

    grep -i "prod-apps.example.com" $CONTEXT_FILE
    grep -i "prod" $CONTEXT_FILE

    If PROD survives in the context, fix it there and re-run AutoConfig before chasing anything downstream — otherwise you’ll be debugging generated artifacts while the source of the bad values sits upstream.


    Root cause 1 — Phantom WebLogic managed servers

    A correctly provisioned three-node oacore service should have oacore_server1, oacore_server2, oacore_server3 (and the matching oafm, forms, forms-c4ws servers per node). The cloned domain carried extra managed servers that mapped to the source environment’s nodes — servers that pointed at listen addresses and host references belonging to PROD.

    These phantom servers do three harmful things:

    1. They keep PROD host references alive inside config.xml.
    2. They confuse the EBS service control and the Admin Server’s view of the cluster.
    3. They can answer (or fail to answer) requests in ways that surface PROD URLs.

    Diagnosing it

    Inventory what the domain actually contains versus what the topology should be:

    # What managed servers does the domain config believe in?
    grep -E "<name>|<listen-address>" \
    $EBS_DOMAIN_HOME/config/config.xml | grep -iE "oacore|oafm|forms|server"
    # What does EBS think the nodes are?
    sqlplus apps/<pwd> <<'EOF'
    COL node_name FORMAT a20
    COL server_address FORMAT a18
    COL support_cp FORMAT a10
    COL support_web FORMAT a10
    COL support_admin FORMAT a13
    SELECT node_name,
    NVL(server_address,'-') AS server_address,
    NVL(support_cp,'-') AS support_cp,
    NVL(support_web,'-') AS support_web,
    NVL(support_admin,'-') AS support_admin
    FROM fnd_nodes
    ORDER BY node_name;
    EOF

    Any managed server or fnd_nodes row that references a host which is not one of appnode1/2/3 is a phantom artifact from the source.

    It’s also worth checking the role assignments — sometimes the hostname is correct but node registrations are duplicated or carry the wrong roles:

    COL node_name FORMAT a30
    COL support_db FORMAT a10
    SELECT node_name,
    support_db,
    support_cp,
    support_web,
    support_admin
    FROM fnd_nodes
    ORDER BY node_name;

    Fixing it (the supported way)

    Do not hand-edit config.xml. EBS R12.2 ships a provisioning utility to add and delete managed servers cleanly, keeping the domain, AutoConfig, and the database registration in sync. Stop the affected services first, then delete each phantom server:

    # Stop the managed servers / services before topology changes
    $ADMIN_SCRIPTS_HOME/adstpall.sh apps/<apps_pwd>
    # Delete a phantom managed server (repeat per orphaned server / service type)
    perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
    ebs-delete-managedserver \
    -contextfile=$CONTEXT_FILE \
    -managedsrvname=oacore_server4 \
    -servicetype=oacore \
    -promptmsg=hide

    After removing every phantom server, run AutoConfig on each app node so the context, the domain, and fnd_nodes agree:

    $ADMIN_SCRIPTS_HOME/adautocfg.sh

    Then confirm fnd_nodes only lists the three real clone nodes, and the Admin Console only shows the expected per-node managed servers. If you find stale node rows after the clean-up, the standard FND_CONC_CLONE.SETUP_CLEAN → AutoConfig sequence on each tier is the canonical way to rebuild the node registration. Run it as APPS, then re-run AutoConfig on every tier:

    EXEC FND_CONC_CLONE.SETUP_CLEAN;
    COMMIT;

    (Always take that step with the DBA team’s sign-off on a shared environment.)


    Root cause 2 — The missing DNS A-record

    With the phantom servers gone, the redirect still misbehaved intermittently. The reason was simpler and entirely outside EBS: the clone’s web entry hostname had no DNS A-record.

    The login flow builds its target URL from the AutoConfig web entry variables. Check them:

    grep -E "s_webentryhost|s_webentrydomain|s_webentryurlprotocol|s_active_webport|s_url_protocol|s_login_page" \
    $CONTEXT_FILE

    It’s worth proving the generated login URL, not just the host variable — if AutoConfig hasn’t fully taken, these still show a PROD host and you’ll be chasing DNS for a problem that lives in the context:

    grep -i "webentry" $CONTEXT_FILE
    grep -i "login_page" $CONTEXT_FILE

    The context correctly named clone-apps.example.com as s_webentryhost. But on the app nodes — and for clients — that name did not resolve:

    nslookup clone-apps.example.com
    # ** server can't find clone-apps.example.com: NXDOMAIN
    dig +short clone-apps.example.com
    # (empty)

    Because the clone web entry host was not resolvable, requests that relied on generated URLs could not consistently resolve back to the clone environment. At the same time, stale PROD references still lived inside the cloned WebLogic topology. Together these two conditions caused redirects to be generated using PROD host information, sending users away from the clone — which is why fixing the phantom servers alone wasn’t enough. To be clear, EBS does not contain any built-in mechanism that redirects a clone to production. Such redirects are almost always caused by stale configuration, profile options, WebLogic topology artifacts, load balancer settings, or DNS resolution — not by EBS doing anything magical. The browser is simply following a redirect built from stale host values. Name resolution has to work and the topology has to be clean.

    Fixing it

    Add an A-record for the clone web entry host in the appropriate DNS zone, pointing at the clone’s web-tier listen address (or its load balancer / OCI public-or-private IP, depending on your access path):

    clone-apps.example.com. IN A 10.x.x.x

    If a DNS change isn’t immediately possible and you only need the app nodes to resolve it for validation, a temporary /etc/hosts entry on each of appnode1/2/3 will confirm the theory — but a proper A-record is the real fix, because clients need to resolve it too:

    10.x.x.x clone-apps.example.com clone-apps

    Verify resolution from each node and re-test:

    for n in appnode1 appnode2 appnode3; do
    echo "== $n =="; ssh $n "nslookup clone-apps.example.com | tail -3"
    done

    Before concluding the issue persists, test using an incognito/private browser session or clear the browser cache. Browsers frequently cache redirects, cookies, and DNS information that can make a corrected environment appear unchanged.


    Root cause 3 — Stale profile option URLs (the other usual suspect)

    Even when DNS resolves and the WebLogic topology is clean, a clone can still throw users at PROD because profile option values were copied straight from the source. These are among the most common causes of a redirect-to-PROD, and they deserve a deliberate check rather than a passing glance. The usual culprits are APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, and ICX_FORMS_LAUNCHER.

    Target them directly:

    COL profile_option_name FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
    fpov.profile_option_value
    FROM fnd_profile_option_values fpov,
    fnd_profile_options_vl fpo
    WHERE fpov.profile_option_id = fpo.profile_option_id
    AND fpo.profile_option_name IN
    ('APPS_WEB_AGENT',
    'APPS_FRAMEWORK_AGENT',
    'APPS_SERVLET_AGENT',
    'ICX_FORMS_LAUNCHER');

    Or sweep more broadly for any value still carrying a PROD host:

    COL profile_option_name FORMAT a30
    COL profile_option_value FORMAT a80
    SELECT fpo.profile_option_name,
    fpov.profile_option_value
    FROM fnd_profile_option_values fpov,
    fnd_profile_options_vl fpo
    WHERE fpov.profile_option_id = fpo.profile_option_id
    AND UPPER(fpov.profile_option_value) LIKE '%PROD%';

    Most of these are AutoConfig-managed, so the right fix is almost always to correct the context and re-run AutoConfig rather than to update the profile value by hand. Hand-updating a profile that AutoConfig owns just means it reverts on the next run. If AutoConfig keeps recreating the wrong value, fix the context file or web entry settings first; otherwise the next AutoConfig run simply reintroduces the problem.


    Other places a PROD reference hides: OHS and the OCI load balancer

    A surprising share of clone redirects originate outside the database and the WebLogic domain entirely — in the web tier configuration that AutoConfig generates, or in the load balancer sitting in front of it. Worth checking these early rather than last.

    Oracle HTTP Server (OHS). Generated OHS config — mod_wl_ohs.conf chief among them — can carry PROD host references. Rather than hard-coding version-specific paths, grep the web-tier and FMW homes broadly:

    grep -R "prod-apps.example.com" \
    $INST_TOP \
    $EBS_DOMAIN_HOME/config \
    $FMW_HOME \
    $ORACLE_HOME 2>/dev/null

    OCI load balancer / reverse proxy. If an OCI Load Balancer (or any reverse proxy) fronts the environment, the redirect can be introduced at that layer even after EBS is fully corrected. Verify:

    • Host header preservation — the backend should receive the clone host, not a rewritten PROD one.
    • Backend set configuration — backends point at the clone app nodes, not PROD.
    • SSL/TLS termination — the protocol and host the LB forwards match what AutoConfig expects (s_webentryurlprotocol, s_active_webport).
    • Redirect rules / rule sets — no listener rule is rewriting the host to PROD.

    I’ve seen an OCI LB listener keep sending users to PROD long after the EBS tier itself was spotless, purely because of a stale redirect rule on the listener.

    Custom code and JARs. Customizations are a notorious hiding place — hardcoded URLs in custom packages, JSPs, or Java survive every clone untouched by AutoConfig. Grep the custom homes too:

    grep -R "prod-apps.example.com" \
    $XX_TOP \
    $JAVA_TOP \
    $CUSTOM_TOP 2>/dev/null

    Validation checklist after remediation

    1. nslookup / dig resolves clone-apps.example.com from all three app nodes and from a client workstation.

    2. WebLogic Admin Console lists only the expected per-node managed servers, all in RUNNING state.

    3. fnd_nodes contains only appnode1/2/3 (check support_cp, support_web, support_admin); no PROD host references remain.

    4. No stale source context files are still registered. Old registrations sometimes survive a clone and cause confusion later during AutoConfig or service management:

    COL node_name FORMAT a20
    COL ctx_file FORMAT a60
    SELECT node_name,
    ctx_type,
    NVL(path,'-') AS ctx_file,
    status
    FROM fnd_oam_context_files
    WHERE status = 'S'
    ORDER BY node_name;

    In a clean clone, every active context file should belong to the clone environment. Any remaining source-environment context registration should be reviewed and removed before further AutoConfig runs.

    5. The profile options from Root cause 3 (APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, ICX_FORMS_LAUNCHER) all carry clone values, not PROD — re-run the targeted query from that section to confirm.

    6. The login page loads from the clone URL and — critically — the post-login redirect stays on clone-apps.example.com, never bouncing to prod-apps.example.com.

    7. Sweep the entire configuration for the PROD hostname, not just the context file. PROD remnants love to hide in OHS configs, mod_wl_ohs.conf, generated XML, and custom integrations:

    grep -R "prod-apps.example.com" \
    $INST_TOP \
    $EBS_DOMAIN_HOME/config \
    $FMW_HOME 2>/dev/null

    Anything this turns up needs to be corrected (and usually re-generated via AutoConfig) before users find it for you.

    8. No context file registered in the database still points at a PROD path. Stale PROD context registrations can cause odd behaviour long after the clone:

    COL node_name FORMAT a20
    COL path FORMAT a70
    SELECT node_name,
    path
    FROM fnd_oam_context_files
    WHERE UPPER(path) LIKE '%PROD%';

    9. If an OCI Load Balancer or reverse proxy fronts the environment, verify its listener rules, backend sets, host-header forwarding, and SSL termination settings contain no PROD references.


    Lessons learned / a small post-clone checklist

    Cloning R12.2 onto a shared file system multi-node tier adds two checks that single-node clones let you skip:

    • Audit the managed-server topology immediately after adcfgclone. On a shared file system the source’s managed servers ride along inside config.xml. Delete phantoms with adProvisionEBS.pl ebs-delete-managedserver, never by editing XML, and re-run AutoConfig.
    • Resolve the web entry hostname before you trust the login page. A clone that “logs you into PROD” is very often a DNS problem wearing an EBS costume. Create the A-record as part of the clone runbook, not as a reaction to the redirect.
    • Treat a redirect-to-PROD as a stop-the-line event. Until both topology and name resolution are clean, assume the clone can still touch production and keep testers out.
    • Run FND_CONC_CLONE.SETUP_CLEAN before AutoConfig in cloned environments. This rebuilds node registrations and context metadata cleanly and helps prevent stale topology information from the source environment persisting in the clone. (On a shared environment, run it with the DBA team’s sign-off.)

    In this case, three unglamorous root causes — leftover managed servers, a missing DNS record, and stale profile option URLs — combined to produce the redirect. In practice, however, most redirect-to-PROD incidents come down to one broader problem: stale production references surviving the clone process. The fastest path to resolution is a systematic sweep of WebLogic, DNS, profile options, context files, OHS configuration, load balancers, and custom code until every production reference is gone.


    Have you hit a different flavour of the clone redirect? The web entry variables, SSL/load-balancer termination, and s_login_page overrides each have their own way of sending you to the wrong host — happy to compare notes in the comments.