Tag: Troubleshooting

When you finish cloning an Oracle E-Business Suite R12.2 environment, the moment of truth is the login page. You hit the clone’s web entry URL, the page loads, you type your credentials… and the browser quietly throws you onto the production URL. On a multi-node app tier this is more than an annoyance — it means your freshly cloned, supposedly isolated environment is reaching back into PROD, and a careless tester could authenticate against the wrong system entirely.

I ran into exactly this on a recent R12.2.11 clone built on Oracle Cloud Infrastructure: a three-node application tier sharing a single run/patch file system over an FSS (File Storage Service) NFS mount. The clone came up, services started, but every login attempt redirected to the PROD web entry host. The initial investigation uncovered two primary issues — phantom WebLogic managed servers carried over from the source domain, and a missing DNS A-record for the clone’s web entry hostname. Along the way I also identified several other clone-related configuration areas that can produce the same symptom, including stale profile options, OHS configuration remnants, load balancer redirects, and custom code references.

This post walks through how the symptom presents, how to diagnose it cleanly, and the supported way to fix each cause.

Throughout, I use placeholder names — clone-apps.example.com for the clone web entry host, prod-apps.example.com for production, and appnode1/2/3 for the three app-tier nodes. Substitute your own values.

The architecture (why a shared file system matters here)

The clone app tier looked like this:

• Three application-tier nodes (appnode1, appnode2, appnode3).
• A shared dual file system (fs1 run / fs2 patch) hosted on an OCI FSS NFS export, mounted identically on all three nodes.
• A single WebLogic domain (EBS_domain_<SID>) living on that shared file system.

The shared file system is the detail that makes the “phantom managed server” problem sticky. Because config.xml and the managed-server definitions physically live on the shared FSS mount, any leftover managed-server entries from the source domain are visible to every node at once. AutoConfig regenerates context-driven artifacts, but it does not, on its own, delete managed servers that no longer belong to the topology.

The symptom

After adcfgclone completed and the application services started:

• The clone login page (https://clone-apps.example.com:<port>/OA_HTML/AppsLogin) rendered correctly.
• On submitting credentials — or sometimes immediately on the redirect to the home page — the browser landed on https://prod-apps.example.com/....
• The WebLogic Admin Console showed more managed servers than the three-node clone should have, several of them in an unreachable / shutdown state.

Two independent problems reinforced each other: stale host references remained inside the cloned WebLogic topology, while the clone web entry hostname could not be resolved correctly. Either issue can cause redirect anomalies, but together they consistently redirected users back to PROD.

First, rule out the context file

Before investigating WebLogic, verify the clone context file itself doesn’t still contain production hostnames. AutoConfig can only generate correct configuration if the context values are correct, so everything downstream is built from here:

grep -i "prod-apps.example.com" $CONTEXT_FILE
grep -i "prod"                  $CONTEXT_FILE

If PROD survives in the context, fix it there and re-run AutoConfig before chasing anything downstream — otherwise you’ll be debugging generated artifacts while the source of the bad values sits upstream.

Root cause 1 — Phantom WebLogic managed servers

A correctly provisioned three-node oacore service should have oacore_server1, oacore_server2, oacore_server3 (and the matching oafm, forms, forms-c4ws servers per node). The cloned domain carried extra managed servers that mapped to the source environment’s nodes — servers that pointed at listen addresses and host references belonging to PROD.

These phantom servers do three harmful things:

1. They keep PROD host references alive inside config.xml.
2. They confuse the EBS service control and the Admin Server’s view of the cluster.
3. They can answer (or fail to answer) requests in ways that surface PROD URLs.

Diagnosing it

Inventory what the domain actually contains versus what the topology should be:

# What managed servers does the domain config believe in?
grep -E "<name>|<listen-address>" \
  $EBS_DOMAIN_HOME/config/config.xml | grep -iE "oacore|oafm|forms|server"
# What does EBS think the nodes are?
sqlplus apps/<pwd> <<'EOF'
COL node_name      FORMAT a20
COL server_address FORMAT a18
COL support_cp     FORMAT a10
COL support_web    FORMAT a10
COL support_admin  FORMAT a13
SELECT node_name,
       NVL(server_address,'-')           AS server_address,
       NVL(support_cp,'-')               AS support_cp,
       NVL(support_web,'-')              AS support_web,
       NVL(support_admin,'-')            AS support_admin
FROM   fnd_nodes
ORDER  BY node_name;
EOF

Any managed server or fnd_nodes row that references a host which is not one of appnode1/2/3 is a phantom artifact from the source.

It’s also worth checking the role assignments — sometimes the hostname is correct but node registrations are duplicated or carry the wrong roles:

COL node_name    FORMAT a30
COL support_db   FORMAT a10
SELECT node_name,
       support_db,
       support_cp,
       support_web,
       support_admin
FROM   fnd_nodes
ORDER  BY node_name;

Fixing it (the supported way)

Do not hand-edit config.xml. EBS R12.2 ships a provisioning utility to add and delete managed servers cleanly, keeping the domain, AutoConfig, and the database registration in sync. Stop the affected services first, then delete each phantom server:

# Stop the managed servers / services before topology changes
$ADMIN_SCRIPTS_HOME/adstpall.sh apps/<apps_pwd>
# Delete a phantom managed server (repeat per orphaned server / service type)
perl $AD_TOP/patch/115/bin/adProvisionEBS.pl \
  ebs-delete-managedserver \
  -contextfile=$CONTEXT_FILE \
  -managedsrvname=oacore_server4 \
  -servicetype=oacore \
  -promptmsg=hide

After removing every phantom server, run AutoConfig on each app node so the context, the domain, and fnd_nodes agree:

$ADMIN_SCRIPTS_HOME/adautocfg.sh

Then confirm fnd_nodes only lists the three real clone nodes, and the Admin Console only shows the expected per-node managed servers. If you find stale node rows after the clean-up, the standard FND_CONC_CLONE.SETUP_CLEAN → AutoConfig sequence on each tier is the canonical way to rebuild the node registration. Run it as APPS, then re-run AutoConfig on every tier:

EXEC FND_CONC_CLONE.SETUP_CLEAN;
COMMIT;

(Always take that step with the DBA team’s sign-off on a shared environment.)

Root cause 2 — The missing DNS A-record

With the phantom servers gone, the redirect still misbehaved intermittently. The reason was simpler and entirely outside EBS: the clone’s web entry hostname had no DNS A-record.

The login flow builds its target URL from the AutoConfig web entry variables. Check them:

grep -E "s_webentryhost|s_webentrydomain|s_webentryurlprotocol|s_active_webport|s_url_protocol|s_login_page" \
  $CONTEXT_FILE

It’s worth proving the generated login URL, not just the host variable — if AutoConfig hasn’t fully taken, these still show a PROD host and you’ll be chasing DNS for a problem that lives in the context:

grep -i "webentry"   $CONTEXT_FILE
grep -i "login_page" $CONTEXT_FILE

The context correctly named clone-apps.example.com as s_webentryhost. But on the app nodes — and for clients — that name did not resolve:

nslookup clone-apps.example.com
# ** server can't find clone-apps.example.com: NXDOMAIN
dig +short clone-apps.example.com
# (empty)

Because the clone web entry host was not resolvable, requests that relied on generated URLs could not consistently resolve back to the clone environment. At the same time, stale PROD references still lived inside the cloned WebLogic topology. Together these two conditions caused redirects to be generated using PROD host information, sending users away from the clone — which is why fixing the phantom servers alone wasn’t enough. To be clear, EBS does not contain any built-in mechanism that redirects a clone to production. Such redirects are almost always caused by stale configuration, profile options, WebLogic topology artifacts, load balancer settings, or DNS resolution — not by EBS doing anything magical. The browser is simply following a redirect built from stale host values. Name resolution has to work and the topology has to be clean.

Fixing it

Add an A-record for the clone web entry host in the appropriate DNS zone, pointing at the clone’s web-tier listen address (or its load balancer / OCI public-or-private IP, depending on your access path):

clone-apps.example.com.    IN    A    10.x.x.x

If a DNS change isn’t immediately possible and you only need the app nodes to resolve it for validation, a temporary /etc/hosts entry on each of appnode1/2/3 will confirm the theory — but a proper A-record is the real fix, because clients need to resolve it too:

10.x.x.x    clone-apps.example.com    clone-apps

Verify resolution from each node and re-test:

for n in appnode1 appnode2 appnode3; do
  echo "== $n =="; ssh $n "nslookup clone-apps.example.com | tail -3"
done

Before concluding the issue persists, test using an incognito/private browser session or clear the browser cache. Browsers frequently cache redirects, cookies, and DNS information that can make a corrected environment appear unchanged.

Root cause 3 — Stale profile option URLs (the other usual suspect)

Even when DNS resolves and the WebLogic topology is clean, a clone can still throw users at PROD because profile option values were copied straight from the source. These are among the most common causes of a redirect-to-PROD, and they deserve a deliberate check rather than a passing glance. The usual culprits are APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, and ICX_FORMS_LAUNCHER.

Target them directly:

COL profile_option_name  FORMAT a30
COL profile_option_value FORMAT a80
SELECT fpo.profile_option_name,
       fpov.profile_option_value
FROM   fnd_profile_option_values fpov,
       fnd_profile_options_vl    fpo
WHERE  fpov.profile_option_id = fpo.profile_option_id
AND    fpo.profile_option_name IN
       ('APPS_WEB_AGENT',
        'APPS_FRAMEWORK_AGENT',
        'APPS_SERVLET_AGENT',
        'ICX_FORMS_LAUNCHER');

Or sweep more broadly for any value still carrying a PROD host:

COL profile_option_name  FORMAT a30
COL profile_option_value FORMAT a80
SELECT fpo.profile_option_name,
       fpov.profile_option_value
FROM   fnd_profile_option_values fpov,
       fnd_profile_options_vl    fpo
WHERE  fpov.profile_option_id = fpo.profile_option_id
AND    UPPER(fpov.profile_option_value) LIKE '%PROD%';

Most of these are AutoConfig-managed, so the right fix is almost always to correct the context and re-run AutoConfig rather than to update the profile value by hand. Hand-updating a profile that AutoConfig owns just means it reverts on the next run. If AutoConfig keeps recreating the wrong value, fix the context file or web entry settings first; otherwise the next AutoConfig run simply reintroduces the problem.

Other places a PROD reference hides: OHS and the OCI load balancer

A surprising share of clone redirects originate outside the database and the WebLogic domain entirely — in the web tier configuration that AutoConfig generates, or in the load balancer sitting in front of it. Worth checking these early rather than last.

Oracle HTTP Server (OHS). Generated OHS config — mod_wl_ohs.conf chief among them — can carry PROD host references. Rather than hard-coding version-specific paths, grep the web-tier and FMW homes broadly:

grep -R "prod-apps.example.com" \
  $INST_TOP \
  $EBS_DOMAIN_HOME/config \
  $FMW_HOME \
  $ORACLE_HOME 2>/dev/null

OCI load balancer / reverse proxy. If an OCI Load Balancer (or any reverse proxy) fronts the environment, the redirect can be introduced at that layer even after EBS is fully corrected. Verify:

• Host header preservation — the backend should receive the clone host, not a rewritten PROD one.
• Backend set configuration — backends point at the clone app nodes, not PROD.
• SSL/TLS termination — the protocol and host the LB forwards match what AutoConfig expects (s_webentryurlprotocol, s_active_webport).
• Redirect rules / rule sets — no listener rule is rewriting the host to PROD.

I’ve seen an OCI LB listener keep sending users to PROD long after the EBS tier itself was spotless, purely because of a stale redirect rule on the listener.

Custom code and JARs. Customizations are a notorious hiding place — hardcoded URLs in custom packages, JSPs, or Java survive every clone untouched by AutoConfig. Grep the custom homes too:

grep -R "prod-apps.example.com" \
  $XX_TOP \
  $JAVA_TOP \
  $CUSTOM_TOP 2>/dev/null

Validation checklist after remediation

1. nslookup / dig resolves clone-apps.example.com from all three app nodes and from a client workstation.

2. WebLogic Admin Console lists only the expected per-node managed servers, all in RUNNING state.

3. fnd_nodes contains only appnode1/2/3 (check support_cp, support_web, support_admin); no PROD host references remain.

4. No stale source context files are still registered. Old registrations sometimes survive a clone and cause confusion later during AutoConfig or service management:

COL node_name    FORMAT a20
COL ctx_file     FORMAT a60
SELECT node_name,
       ctx_type,
       NVL(path,'-')  AS ctx_file,
       status
FROM   fnd_oam_context_files
WHERE  status = 'S'
ORDER  BY node_name;

In a clean clone, every active context file should belong to the clone environment. Any remaining source-environment context registration should be reviewed and removed before further AutoConfig runs.

5. The profile options from Root cause 3 (APPS_WEB_AGENT, APPS_FRAMEWORK_AGENT, APPS_SERVLET_AGENT, ICX_FORMS_LAUNCHER) all carry clone values, not PROD — re-run the targeted query from that section to confirm.

6. The login page loads from the clone URL and — critically — the post-login redirect stays on clone-apps.example.com, never bouncing to prod-apps.example.com.

7. Sweep the entire configuration for the PROD hostname, not just the context file. PROD remnants love to hide in OHS configs, mod_wl_ohs.conf, generated XML, and custom integrations:

grep -R "prod-apps.example.com" \
  $INST_TOP \
  $EBS_DOMAIN_HOME/config \
  $FMW_HOME 2>/dev/null

Anything this turns up needs to be corrected (and usually re-generated via AutoConfig) before users find it for you.

8. No context file registered in the database still points at a PROD path. Stale PROD context registrations can cause odd behaviour long after the clone:

COL node_name FORMAT a20
COL path      FORMAT a70
SELECT node_name,
       path
FROM   fnd_oam_context_files
WHERE  UPPER(path) LIKE '%PROD%';

9. If an OCI Load Balancer or reverse proxy fronts the environment, verify its listener rules, backend sets, host-header forwarding, and SSL termination settings contain no PROD references.

Lessons learned / a small post-clone checklist

Cloning R12.2 onto a shared file system multi-node tier adds two checks that single-node clones let you skip:

• Audit the managed-server topology immediately after adcfgclone. On a shared file system the source’s managed servers ride along inside config.xml. Delete phantoms with adProvisionEBS.pl ebs-delete-managedserver, never by editing XML, and re-run AutoConfig.
• Resolve the web entry hostname before you trust the login page. A clone that “logs you into PROD” is very often a DNS problem wearing an EBS costume. Create the A-record as part of the clone runbook, not as a reaction to the redirect.
• Treat a redirect-to-PROD as a stop-the-line event. Until both topology and name resolution are clean, assume the clone can still touch production and keep testers out.
• Run FND_CONC_CLONE.SETUP_CLEAN before AutoConfig in cloned environments. This rebuilds node registrations and context metadata cleanly and helps prevent stale topology information from the source environment persisting in the clone. (On a shared environment, run it with the DBA team’s sign-off.)

In this case, three unglamorous root causes — leftover managed servers, a missing DNS record, and stale profile option URLs — combined to produce the redirect. In practice, however, most redirect-to-PROD incidents come down to one broader problem: stale production references surviving the clone process. The fastest path to resolution is a systematic sweep of WebLogic, DNS, profile options, context files, OHS configuration, load balancers, and custom code until every production reference is gone.

Have you hit a different flavour of the clone redirect? The web entry variables, SSL/load-balancer termination, and s_login_page overrides each have their own way of sending you to the wrong host — happy to compare notes in the comments.