Syed Anwar Ahmed – Oracle DBA Blog

Category: Troubleshooting

Oracle Database and EBS real-world troubleshooting guides and RCA

  • AHF 26.9 Upgrade on RAC: Fixing a Remote Node SSH Failure

    Alhamdulillah. I recently upgraded Oracle AHF from 26.5.1.0.0 to 26.9.0.0.0 on a two-node RAC. Node 1 upgraded cleanly. Node 2 failed because the installer couldn’t authenticate over SSH as root. Here’s what happened and how I finished the job with a local upgrade.

    Hostnames and client-specific details are omitted.

    Environment

    • Platform: Linux x86-64
    • Setup: two-node Oracle RAC
    • Initial AHF version: 26.5.1.0.0
    • Target AHF version: 26.9.0.0.0
    • AHF home: /opt/oracle.ahf
    • AHF data directory: <oracle_base>/oracle.ahf/data

    The paths in the examples below are placeholders. Replace them with your actual installer and staging directories. Run the signature check and ahf_setup from the extracted installer directory, and run each command with the privileges the target paths require.

    The Problem

    I launched the cluster-wide installer from Node 1:

    cd /path/to/ahf_installer
./ahf_setup

    Node 1 upgraded fine. When it moved on to Node 2, the installer kept asking for the root password, then failed:

    [ERROR] : [node2] Failed to Upgrade AHF. Exit Status : 255

    The summary showed Node 1 on 26.9.0.0.0 and Node 2 still on 26.5.1.0.0. The installer still printed a general success message at the end, even though the cluster was only half upgraded.

    Lesson: read the per-node summary. Don’t trust the final success line.

    The Fix: Local Upgrade on Node 2

    Step 1: Copy and verify the package

    I copied the installer ZIP from Node 1 to Node 2 using an approved transfer method, then checked its integrity:

    unzip -t AHF-LINUX_v26.9.0.zip
sha256sum AHF-LINUX_v26.9.0.zip

    The SHA-256 matched the checksum from the source node.

    Step 2: Extract and verify the signature

    mkdir -p /path/to/ahf_stage/26.9.0

unzip -o AHF-LINUX_v26.9.0.zip \
  -d /path/to/ahf_stage/26.9.0

cd /path/to/ahf_stage/26.9.0

openssl dgst -sha256 \
  -verify ./oracle-tfa.pub \
  -signature ./ahf_setup.dat \
  ./ahf_setup

    Expected output:

    Verified OK

    Step 3: Run the local upgrade

    Oracle’s upgrade guide says to run ahf_setup as root. My account had authorized sudo privileges, so I ran the installer locally with sudo:

    sudo ./ahf_setup -local

    The installer detected the existing AHF installation and prompted for confirmation. The local upgrade completed successfully. Its final summary reported both Node 2 and Node 1 at AHF version 26.9.0.0.0 with status UPGRADED. I then verified the version and TFA status from both nodes.

    Step 4: Verify on both nodes

    sudo /opt/oracle.ahf/bin/tfactl version
sudo /opt/oracle.ahf/bin/tfactl status

    Results on both nodes:

    • AHF version: 26.9.0.0.0
    • Matching build ID: 260900020260911121902
    • TFA status: RUNNING
    • Inventory status: COMPLETE
    • No version mismatch between nodes

    Observed Failure

    The installer couldn’t authenticate as root over SSH to the remote node. Oracle documents secure daemon socket communication for upgrading an existing AHF cluster; in this case, the installer nevertheless prompted for root SSH authentication.

    My logs show the immediate failure: the root SSH authentication attempt failed. They don’t show why the installer took that path or why the authentication failed, and I didn’t dig into it further.

    Best Practices

    • Confirm the target version and compatibility before you start.
    • Verify the installer checksum and signature before running it.
    • Run the installer as root, or with sudo if your account is authorized.
    • Confirm the required privileges and cluster communication prerequisites before upgrading. If unexpected SSH authentication prompts occur, investigate the issue rather than changing SSH settings without approval.
    • If remote authentication fails, use the documented local-upgrade procedure. Don’t make unapproved SSH or security changes.
    • Check the per-node summary and TFA status on every node.
    • Keep the installer logs and record the final version and build ID for change management.

    Conclusion

    ما شاء الله, both RAC nodes are now on AHF 26.9.0.0.0. The takeaway: a success message from the installer doesn’t guarantee every node was upgraded. Check each node’s status and confirm TFA is running before you close the change.

    Always follow the Oracle documentation and your organization’s change-management process for production work.

    References

  • Oracle EBS Workflow Mailer After an SMTP Migration: Why Mail Still Used the Old Relay

    Hostnames, IP addresses, usernames, email addresses, and notification IDs in this post are masked with # characters.

    Alhamdulillah, this investigation started with a simple question: after changing the SMTP server in Oracle E-Business Suite, why was mail still going through the old relay?

    It turned up three separate findings. The EBS Workflow Mailer configuration pointed to the new SMTP server. OS-level Sendmail still used the old relay. Some failed Workflow notifications were associated with a disabled recipient preference, and the reason for that preference is not yet established.

    This post documents the checks, the configuration change, and the evidence collected during validation. It is an interim investigation: the checks still open are listed in section 9.

    Environment

    • Oracle E-Business Suite R12.2
    • Oracle Database 11.2.0.4
    • Linux 7 application tier
    • Sendmail 8.14.7
    • systemd service management

    1. The initial problem

    The Workflow Mailer’s outbound SMTP server had been changed to the new server (####################). Network connectivity to the new server was successful, but the logs showed messages being relayed through the old relay (####################) before the Sendmail configuration change.

    At the same time, two notifications for one user showed MAIL_STATUS = FAILED.

    The first step was to verify the EBS configuration, then trace the actual mail path through the application and operating-system logs.

    2. Finding 1: EBS configuration and OS-level Sendmail did not match

    Verify the Workflow Mailer configuration

    SET LINESIZE 200
    COL parameter_name   FORMAT A20
    COL parameter_value  FORMAT A30
    COL last_update_date FORMAT A22
    SELECT RPAD(p.parameter_name,20) parameter_name,
           RPAD(NVL(v.parameter_value,'-'),30) parameter_value,
           TO_CHAR(v.last_update_date,'DD-MON-YY HH24:MI:SS')
               last_update_date
    FROM   fnd_svc_comp_param_vals v,
           fnd_svc_comp_params_b p,
           fnd_svc_components c
    WHERE  v.component_id = c.component_id
    AND    v.parameter_id = p.parameter_id
    AND    c.component_name = 'Workflow Notification Mailer'
    AND    p.parameter_name = 'OUTBOUND_SERVER';

    OUTBOUND_SERVER showed the new SMTP server. The Sendmail log showed a different route:

    sendmail[####]: ##############:
    from=<wfmailer@#######################>,
    msgid=<########.####.#############.JavaMail.wfmailer@#######################>,
    proto=ESMTP, daemon=MTA,
    relay=####################### [###.###.###.###]
    sendmail[####]: ##############:
    to=<##################>, mailer=relay,
    relay=#################### [###.###.###.###],
    dsn=2.0.0, stat=Sent

    The logs confirmed that messages submitted to local Sendmail were forwarded through the old smart host. The sender and the JavaMail.wfmailer message ID point to messages generated by the Workflow Mailer. Oracle’s Workflow documentation describes the Mailer sending notification messages by SMTP to the SMTP server configured for it (see Outbound Notification Mailer Processing, from an older Workflow release), and notes that a UNIX host running Sendmail can serve as that SMTP server (see the Administrator’s Guide). The Mailer’s previous OUTBOUND_SERVER value was not captured, so the logs do not show why it handed these messages to local Sendmail. Further tracing was needed to establish the route used by the affected Workflow notifications.

    Check the Workflow Mailer process start time

    SET LINESIZE 200
    COL queue_name FORMAT A20
    COL status     FORMAT A10
    COL os_pid     FORMAT A10
    COL started    FORMAT A22
    SELECT RPAD(q.concurrent_queue_name,20) queue_name,
           RPAD(NVL(p.process_status_code,'-'),10) status,
           RPAD(NVL(TO_CHAR(p.os_process_id),'-'),10) os_pid,
           TO_CHAR(p.process_start_date,'DD-MON-YY HH24:MI:SS')
               started
    FROM   fnd_concurrent_processes p,
           fnd_concurrent_queues q
    WHERE  p.concurrent_queue_id = q.concurrent_queue_id
    AND    p.queue_application_id = q.application_id
    AND    q.concurrent_queue_name = 'WFMLRSVC'
    AND    p.process_status_code = 'A';

    The process had started 11 days before the parameter change. An OS-level check with ps -o pid,lstart,cmd -p <os_pid> showed the same start date.

    That raised the possibility that the Workflow Mailer had not been restarted after the change. The previous parameter value was not captured, though, so the start time alone could not show which SMTP server the running process had been using.

    Lesson: After changing Workflow Mailer parameters, check the active service process and follow the real message route in the logs. The database parameter alone doesn’t prove which server handled a message.

    Query note: last_update_date on the fnd_svc_components row is not the time a parameter changed. Use last_update_date on fnd_svc_comp_param_vals, as in the query above.

    3. Finding 2: Some failed notifications were associated with a disabled recipient preference

    Two notifications remained OPEN with MAIL_STATUS = FAILED. The recipient’s notification preference was DISABLED.

    SET LINESIZE 220
    COL recipient_role FORMAT A20
    COL mail_status    FORMAT A12
    COL status         FORMAT A10
    COL message_name   FORMAT A30
    COL email_address  FORMAT A30
    COL notif_pref     FORMAT A12
    SELECT n.notification_id,
           RPAD(n.recipient_role,20) recipient_role,
           RPAD(NVL(n.mail_status,'-'),12) mail_status,
           RPAD(n.status,10) status,
           RPAD(NVL(n.message_name,'-'),30) message_name,
           RPAD(NVL(r.email_address,'-'),30) email_address,
           RPAD(NVL(r.notification_preference,'-'),12) notif_pref
    FROM   wf_notifications n,
           wf_roles r
    WHERE  n.notification_id IN (#######,#######)
    AND    r.name = n.recipient_role;

    Example output:

    NOTIFICATION_ID RECIPIENT_ROLE  MAIL_STATUS  STATUS  MESSAGE_NAME            EMAIL_ADDRESS        NOTIF_PREF
    --------------- --------------- ------------ ------- ----------------------- -------------------- ----------
            ####### ######          FAILED       OPEN    PO_PO_HAS_BEEN_APPROVE  ##################   DISABLED
            ####### ######          FAILED       OPEN    PO_REQ_APPROVED         ##################   DISABLED

    The preference was also found in WF_LOCAL_ROLES and traced to the Workflow MAILTYPE preference in FND_USER_PREFERENCES. Both notifications remained in the user’s Worklist.

    A DISABLED preference does not by itself explain a FAILED status. Oracle documents that when every address for a recipient is invalid and delivery fails, the Mailer sets the notification’s mail status to FAILED, sets the recipient’s preference to DISABLED, and notifies SYSADMIN (see the Oracle EBS technology blog). So DISABLED can be the result of an earlier delivery failure, not only a choice the user made.

    I did not check when or why this preference was set, or whether the failures began before it was set. I also have not yet confirmed whether these two messages ever reached Sendmail. Both checks are listed in section 9.

    4. Finding 3: Historical failures need to be separated from the migration issue

    Before attributing every failure to the migration, I collected a baseline of notification statuses.

    SET LINESIZE 200
    COL mail_status FORMAT A12
    COL cnt         FORMAT 99999999
    SELECT RPAD(NVL(mail_status,'-'),12) mail_status,
           COUNT(*) cnt
    FROM   wf_notifications
    WHERE  mail_status IN ('MAIL','FAILED','ERROR')
    GROUP BY mail_status;
    Mail statusCountObservation
    MAIL0No records in this status
    FAILED156115 associated with disabled preferences, 39 with missing recipient roles, 2 canceled
    ERROR2,4252,255 created on one day for a single administrator role; the rest were older

    The dates showed that these records were not all from the migration period. The counts show association, not cause: a disabled preference can be a cause or a consequence of a failure. Historical records, missing roles, disabled preferences and SMTP delivery issues need separate investigation.

    5. Changing the Sendmail smart host

    The change was limited to the Sendmail configuration. These commands illustrate the procedure used here. Run them only after confirming the approved SMTP target, any maintenance requirements and the rollback plan.

    Pre-change checks

    date
    systemctl status sendmail --no-pager
    mailq
    grep -ni "SMART_HOST" /etc/mail/sendmail.mc
    grep -ni "^DS" /etc/mail/sendmail.cf
    nc -vz -w 5 #################### 25

    Back up the configuration

    cp -p /etc/mail/sendmail.mc \
      /etc/mail/sendmail.mc.bak_$(date +%Y%m%d_%H%M%S)
    cp -p /etc/mail/sendmail.cf \
      /etc/mail/sendmail.cf.bak_$(date +%Y%m%d_%H%M%S)

    Update the smart host

    Edit /etc/mail/sendmail.mc and change only the active SMART_HOST definition:

    define(`SMART_HOST', `####################')dnl

    Regenerate the configuration and verify:

    cd /etc/mail
    make
    grep -ni "^DS" /etc/mail/sendmail.cf

    Expected output:

    101:DS####################

    Restart Sendmail:

    systemctl restart sendmail
    systemctl status sendmail --no-pager

    The service returned to active (running) and was accepting connections.

    The change did not touch /etc/hosts, /etc/mail/access or /etc/hosts.allow. Those files serve other purposes and shouldn’t be edited just to change the smart host, unless a separate requirement justifies it.

    Rollback: restore sendmail.mc from its backup, run make, confirm the DS line, and restart Sendmail.

    6. Verify delivery through the new SMTP server

    A controlled test mail was sent from the application server to an approved test mailbox.

    echo "Test email from application server after SMTP migration" \
      | mail -s "SMTP Migration Test" ####################

    The relevant log entries:

    STARTTLS=client, relay=####################.,
    verify=FAIL, cipher=ECDHE-RSA-AES256-GCM-SHA384
    to=<####################>, mailer=relay,
    relay=####################. [###.###.###.###],
    dsn=2.0.0, stat=Sent

    ما شاء الله, the test message went through the new relay on the first attempt.

    What does stat=Sent mean? The next SMTP server accepted the message for delivery. It does not confirm arrival in the recipient’s inbox. Check the mailbox, and use the SMTP server’s message trace if delivery fails later.

    What does verify=FAIL mean? Sendmail did not verify the server certificate under its current settings, even though the SMTP transaction succeeded. Successful delivery is not proof that certificate validation is healthy, so review it separately with the mail or security team.

    7. What the investigation proved

    Confirmed by configuration and logs:

    • The EBS Workflow Mailer was configured with the new SMTP server.
    • OS-level Sendmail still had the old smart host before the change, and messages submitted to it were forwarded through that host.
    • The Sendmail configuration was updated, regenerated, and restarted successfully.
    • A controlled test message was relayed through the new SMTP server and accepted for delivery.
    • The two example notifications had MAIL_STATUS = FAILED, remained OPEN, and had a recipient preference of DISABLED.

    Not established by these checks:

    • Which SMTP server the running Workflow Mailer process was using internally before it was restarted, and why it handed messages to local Sendmail.
    • Whether the two failed notifications ever reached Sendmail.
    • When and why the recipient’s preference became DISABLED, and whether the failures began before it.
    • The cause of every historical FAILED or ERROR notification.
    • Whether the original affected notifications were delivered after the Sendmail change.
    • Whether the recipient received the test email.
    • The cause of the STARTTLS certificate-verification warning.

    A successful test email validates one segment of the mail path. The original Workflow notification still needs its own test.

    8. Practical lessons

    1. Check both EBS and OS-level SMTP settings. The Workflow Mailer parameter and the Sendmail smart host can differ.
    2. Compare the parameter change time with the process start time. Restart the Workflow Mailer component when the change procedure requires it, and confirm the new start time.
    3. Trace a real message end to end. Use the notification record, the Mailer logs, the Sendmail logs and recipient confirmation.
    4. Don’t assume a failed notification is either an SMTP problem or a preference problem. A disabled preference can be a cause or a consequence of a delivery failure, so check when it was set, along with the recipient role, email address and notification status.
    5. Capture a baseline before changing anything. Old FAILED and ERROR records can have unrelated causes.
    6. Verify the handoff and the final delivery separately. stat=Sent is evidence of SMTP acceptance, not of inbox delivery.
    7. Review certificate verification on its own. Successful delivery doesn’t remove the need to investigate TLS warnings.

    9. Status and remaining checks

    The Sendmail change is complete and the test message was accepted by the new relay. These checks are still open:

    1. Failed notifications: search the Sendmail log for the affected recipient to establish whether the two failed messages ever reached Sendmail.
    2. Preference history: establish when and why the recipient’s MAILTYPE preference became DISABLED (who last updated it, and whether the first FAILED notification came before or after), and whether SYSADMIN received an invalid-address notice.
    3. Workflow Mailer process: verify the active WFMLRSVC process against the parameter change time, and restart the component under the approved change procedure if needed.
    4. End-to-end Workflow test: submit a test notification, then check the Mailer log and the Sendmail log to confirm it was relayed through the new server.
    5. Certificate verification: investigate verify=FAIL separately with the mail or security team.

    Until those checks are done, this post is an interim investigation. It does not claim that the original Workflow notifications have been fixed.

    Conclusion

    Changing the SMTP server in Oracle EBS may not be the whole job. OS-level Sendmail has its own smart-host setting, and it needs checking when the mail logs show messages leaving through an unexpected relay.

    The Sendmail test confirmed that the new relay accepts mail from the application server. The checks in section 9 will show whether the Workflow Mailer path itself works end to end, and why the affected notifications failed.

    Disclaimer: This article is based on a troubleshooting scenario. All hostnames, IP addresses, usernames, email addresses and notification IDs are masked. Validate all commands and configuration changes against your own environment and change-management procedures before using them in production.

  • Oracle EBS R12.2 DualFS RapidClone: FMW Prerequisite Failures, RPMDB Corruption and Stale Inventory

    Alhamdulillah, this post documents the issues hit during an EBS R12.2 QA application-tier refresh and how each was isolated. Server names, paths and identifiers are generic examples.

    Environment (generic)

    ComponentExample
    App serverqa-app-server01
    DB serverqa-db-server01
    EBS environmentQA_EBS
    Application base/u01/apps
    Pairs file/u01/cloning/pairs/qa_apps_pairs.txt
    Central inventory/opt/oraInventory

    Background

    A DualFS clone recreates both fs1 and fs2. Per the cloning procedure used here, the old fs2, inst and FMW_Home were moved aside and removed first:

    cd /u01/apps/
mv fs2 fs2_old
rm -rf fs2_old &

cd fs1
mv inst inst_old
mv FMW_Home FMW_Home_old
rm -rf inst_old FMW_Home_old &

    Cleanup steps should follow your own environment’s cloning document.

    Symptom

    adcfgclone.pl failed at the FMW prerequisite stage:

    ERROR: Failed to execute FMW pre-requisite check

    Re-running would not have helped. The prerequisite check reads OS state, so that is where the investigation started.

    cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'

    Issue 1: RPM database corruption

    On one app server, rpm --verifydb returned Berkeley DB errors:

    BDB0682 Page ... bad prev_pgno
BDB0683 Page ... overflow item incomplete
Packages: BDB0090 DB_VERIFY_BAD

    Package queries also showed rpmdbNextIterator: skipping h# ... and region trailer: BAD.

    RPMDB corruption was identified and the package state was subsequently validated. If the RPM database is unhealthy, package-based prerequisite results can be incomplete or misleading. Running rpm --verifydb early is a useful way to identify this condition before spending time troubleshooting the Oracle prerequisite scripts.

    Issue 2: Missing compatibility packages

    Two packages were missing: compat-libcap1 and compat-libstdc++-33. After confirming repository availability, both were installed with dnf install, then the RPM database was verified again.

    A similar QA server passed the check without this problem, so package state can differ between servers that look identical.

    Issue 3: refhost.xml OL7 block on an OL8 target

    The target was Oracle Linux 8, but the first OS block in refhost.xml was defined for Oracle Linux 7:

    /u01/apps/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/prereq/linux64/refhost.xml
    <VERSION VALUE="7"/>

    Only that block was changed to VALUE="8". A global sed replacement is risky because the file contains multiple OS definitions. Back up the file, identify the applicable block, change only that, and validate the XML.

    Issue 4: Stale Oracle Central Inventory

    Removing FMW_Home from disk does not remove it from the central inventory. Entries such as these were still registered after the directories were gone:

    /u01/apps/fs1/FMW_Home/oracle_common
/u01/apps/fs1/FMW_Home/webtier
/u01/apps/fs1/FMW_Home/Oracle_EBS-app1

    Documented cloning step. The cloning document’s inventory cleanup applies to the target database server (check /etc/oraInst.loc, remove the DB home entry from ContentsXML/inventory.xml). It does not cover resetting the application-server inventory, so the app-server reset described here is not part of the standard cloning steps.

    Before any inventory change: confirm the active inventory via /etc/oraInst.loc, preserve a copy, then apply the approved procedure. The existing inventory was kept as a backup. Note that the first mv below is conditional: run it only if /opt/oraInventory_old already exists.

    # Only if /opt/oraInventory_old already exists:
mv /opt/oraInventory_old /opt/oraInventory_old_backup

mv /opt/oraInventory /opt/oraInventory_old
mkdir /opt/oraInventory
chown oracle:oinstall /opt/oraInventory

    Issue 5: EXECUTE_FMW_SYS_PREREQS

    The cloning procedure sets this variable before the DualFS run:

    export EXECUTE_FMW_SYS_PREREQS=false
echo $EXECUTE_FMW_SYS_PREREQS

    This skips the FMW OS prerequisite execution. It was set here only after the failures above had been investigated, and only because the procedure specifies it. It is not a general substitute for fixing prerequisite failures.

    Issue 6: SSH disconnect during a multi-hour clone

    RapidClone can run for hours, and the procedure calls for a screen session (or VDI). During this exercise the SSH session dropped mid-run.

    screen -S rapidclone
cd /u01/apps/fs1/EBSapps/comn/clone/bin
export EXECUTE_FMW_SYS_PREREQS=false
perl adcfgclone.pl component=appsTier \
pairsfile=/u01/cloning/pairs/qa_apps_pairs.txt \
dualfs=yes

    After the disconnect, check before restarting anything:

    screen -ls
screen -r <session>

    If the original process is still running, a second adcfgclone.pl would only add complications.

    Result

    After the identified OS, package, prerequisite and inventory issues were addressed, the DualFS clone completed and created both fs1 and fs2.

    ما شاء الله

    The same approach worked on the second QA environment. This establishes that the DualFS RapidClone completed, not that the entire QA refresh is finished: post-clone configuration is separate, and the procedure says not to start services at the end of the clone until the required post-clone steps are completed.

    Summary

    FindingLayerAction
    FMW prerequisite failureOS / packagesInvestigated underlying checks
    RPMDB corruptionOSIdentified the corruption and validated the package state after remediation
    Missing compat RPMsOSInstalled required packages
    OL7 block on OL8 targetPrerequisite configCorrected the applicable block only
    Stale FMW homes in inventoryOracle InventoryBacked up the existing inventory, then applied the approved remediation
    Long-running cloneSessionRan inside screen

    Suggested pre-clone checklist

    cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'
cat /etc/oraInst.loc
ls -ld /u01/apps/fs1 /u01/apps/fs2
ls -l /u01/apps/fs1/EBSapps/comn/clone/bin/adcfgclone.pl

    Post-clone

    RapidClone success is not the end of the refresh. The cloning procedure continues with custom environment configuration, AutoConfig, WebLogic configuration, Concurrent Manager/MWA node configuration, shared filesystem setup and further AutoConfig runs, followed by application validation.

    Takeaway

    RapidClone is often where infrastructure problems become visible, not where they start. When it fails, check the layers underneath before rerunning:

    OS → RPM database → packages → prerequisite config → Oracle Inventory → filesystem → RapidClone → post-clone configuration

    Examples use generic names. No real server names, IP addresses, domains, company names or screenshots are included.

  • Oracle 19c Data Guard: Recovering UNNAMED Datafiles on a Multitenant Physical Standby

    Alhamdulillah.

    Oracle Data Guard physical standbys keep redo synchronized automatically — until the primary adds a datafile that the standby can’t place on disk. When STANDBY_FILE_MANAGEMENT=MANUAL, the standby refuses to guess where the new file should live. Managed Standby Recovery stops instead, leaving the control file ahead of the actual datafiles and the standby parked at a consistent SCN.

    This post walks through a real 19c multitenant incident: MRP0 hit ORA-01274 after a new datafile couldn’t be added, recovery stalled, and the fix required creating the missing file inside the correct pluggable database container, then resuming and verifying redo apply. In this incident, the missing datafile belonged to PROD. Creating the file from within the owning PDB resolved the placeholder, after which Redo Apply could be restarted from CDB$ROOT.

    Failure Diagnosis

    The alert log during redo apply:
    Recovery of Online Redo Log: Thread 1 Group 4 Seq 105902 Reading mem 0
    Mem# 0: /uora/PROD/db/product/19.0.0/dbs/broken3
    2025-05-12T08:49:56.027319-04:00
    PROD(3):File #433 added to control file as ‘UNNAMED00433’ because
    PROD(3):the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
    PROD(3):The file should be manually created to continue.
    PR00 (PID:2796228): MRP0: Background Media Recovery terminated with error 1274
    2025-05-12T08:49:56.092976-04:00
    Errors in file /uora/PROD/db/diag/rdbms/proddr/cdbPROD/trace/cdbPROD_pr00_2796228.trc:
    ORA-01274: cannot add data file that was originally created as
    ‘/uora/PROD/data/proddata/users02.dbf’
    PR00 (PID:2796228): Managed Standby Recovery not using Real Time Apply
    Recovery interrupted!
    2025-05-12T08:49:57.933554-04:00
    Recovery stopped due to failure in applying recovery marker (opcode 17.30).
    Datafiles are recovered to a consistent state at change 36278361292
    but controlfile could be ahead of datafiles.
    Stopping change tracking


    Three things happening in sequence:

    1. **Control file registered a file that doesn’t exist.** The redo stream carried a datafile creation from the primary. The standby’s control file logged it — but because STANDBY_FILE_MANAGEMENT=MANUAL, the standby recorded it as UNNAMED00433 instead of creating it.

    2. **Recovery stopped on the recovery marker.** Opcode 17.30 is the file operation that couldn’t be satisfied. The standby parked at SCN 36278361292 — a point where datafiles are internally consistent, but the control file believes file 433 exists when it doesn’t.

    3. **MRP0 exited.** The managed standby process shut down cleanly. The alert log shows “not using Real Time Apply,” indicating recovery was interrupted and the standby could not continue applying the current redo stream.

    **Root cause:** Configuration, not corruption. STANDBY_FILE_MANAGEMENT=MANUAL means every new primary datafile must be created manually on the standby. With AUTO and proper DB_FILE_NAME_CONVERT / PDB_FILE_NAME_CONVERT setup, new files appear automatically. This environment chose MANUAL, so a routine tablespace addition on the primary became a recovery outage on the standby.

    ## Recovery Workflow

    MRP0 stops → Query UNNAMED files → Switch to PDB → Create datafile → Resume MRP → Verify

    ## Step 1 — Locate the UNNAMED File

    Confirm the standby’s role and find the placeholder. Run from CDB$ROOT:


    SET ECHO OFF PAGESIZE 60 LINESIZE 120 TRIMSPOOL ON
    COL NAME FORMAT A50
    SQL> SELECT name, open_mode, database_role FROM v$database;
    NAME OPEN_MODE DATABASE_ROLE
    ——— ——————– —————-
    CDBSTBY MOUNTED PHYSICAL STANDBY
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    433 /u01/app/oracle/product/19.0.0/dbhome_1/dbs/UNNAMED00433


    File number is 433. Which PDB owns it? Use CON_ID:


    COL CON_ID FORMAT 9999
    COL FILE# FORMAT 9999
    COL NAME FORMAT A50
    SQL> SELECT con_id, file#, name FROM V$DATAFILE WHERE file# = 433;
    CON_ID FILE# NAME
    ———- —— ————————————————–
    3 433 /u01/app/oracle/product/19.0.0/dbhome_1/dbs/UNNAMED00433


    CON_ID = 3 is PROD. That’s where the fix must run.

    ## Step 2 — PDB Recovery: Create the File in the Right Container

    From CDB$ROOT, the command appeared to succeed:


    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00433’
    AS ‘/uora/PROD/data/proddrdata/users02.dbf’;
    Database altered.
    “`

    But checking from the PDB showed the placeholder still registered:


    SQL> ALTER SESSION SET CONTAINER = PROD;
    Session altered.
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    433 /uora/PROD/db/product/19.0.0/dbs/UNNAMED00433


    After switching to the PROD container and reissuing the command:


    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00433’
    AS ‘/uora/PROD/data/proddrdata/users02.dbf’;
    Database altered.
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    no rows selected


    **Important:** Validate this behavior against Oracle 19c documentation and test in non-production before applying to your standby.

    ## Step 3 — Redo Apply Resumption


    SQL> ALTER SESSION SET CONTAINER = CDB$ROOT;
    Session altered.
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE DISCONNECT FROM SESSION;
    Database altered.
    SQL> SELECT PROCESS, STATUS, THREAD#, SEQUENCE# FROM V$MANAGED_STANDBY;
    PROCESS STATUS THREAD# SEQUENCE#
    ———- ————— ——- ———
    MRP0 APPLYING_LOG 1 105902
    ARCH CLOSING 1 105905
    RFS IDLE 1 105906


    MRP0 is back in APPLYING_LOG.

    ## Step 4 — Expect More: The Second UNNAMED File

    Recovery resumed and encountered another placeholder:

    “`sql
    SQL> ALTER SESSION SET CONTAINER = PROD;
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    435 /uora/PROD/db/product/19.0.0/dbs/UNNAMED00435
    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00435’
    AS ‘/uora/PROD/data/proddrdata/transaction_table_16.dbf’;
    Database altered.


    Expect a loop. Every datafile added on the primary after the gap will appear as another UNNAMED entry. Grep the alert log for the full backlog first:


    grep -i ‘unnamed’ alert_cdbstby.log | head -20
    grep ‘ORA-01274’ alert_cdbstby.log | wc -l


    ## Recovery Verification

    Once placeholders are gone, resume recovery:


    ALTER SESSION SET CONTAINER = CDB$ROOT;
    ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE DISCONNECT FROM SESSION;
    “`

    Verify:


    SELECT con_id, file#, name FROM V$DATAFILE WHERE name LIKE ‘%UNNAMED%’;
    SELECT process, status, thread#, sequence# FROM V$MANAGED_STANDBY WHERE process LIKE ‘MRP%’;
    SELECT * FROM V$DATAGUARD_STATS;


    Clean results: zero UNNAMED rows, MRP0 in APPLYING_LOG, apply lag returning to normal.

    ## Alternative: Restore from Backup

    If redo is unavailable or purged:

    1. Identify primary backup with missing file
    2. Restore to standby location
    3. Register with control file via RMAN
    4. Resume recovery

    Follow Oracle RMAN and Data Guard documentation for your configuration.

    ## Lessons Learned

    **Decide STANDBY_FILE_MANAGEMENT deliberately.** MANUAL works for locked-down environments. AUTO prevents this class of manual intervention when file naming and storage support it. ما شاء الله — the infrastructure, when well-designed, removes recovery work.

    **Session container context can affect recovery steps.** In this incident, CREATE DATAFILE from CDB$ROOT appeared to succeed but didn’t resolve the placeholder until run from PROD PDB. Validate against Oracle 19c documentation and test in non-production.

    **Expect a loop.** Every datafile added after the gap arrives as another UNNAMED entry. Grep the alert log for full backlog first.

    **Verify before closing.** Confirm zero UNNAMED files, MRP0 running, settled lag before closing incident.

    ## Conclusion

    ORA-01274 on a physical standby is a configuration boundary, not data loss. This incident resolved via CREATE DATAFILE in the PROD PDB, followed by managed recovery restart from CDB$ROOT. Before applying: Verify steps against Oracle 19c documentation, test in non-production, verify standby status and lag.

  • Oracle EBS R12.2 `adop phase=fs_clone` Fails with JSAFE_PaddingException: Invalid Padding

    Alhamdulillah.

    Introduction

    During a routine Oracle E-Business Suite R12.2 maintenance cycle, adop phase=fs_clone repeatedly failed while rebuilding the PATCH file system (fs2). The initial error message pointed to an edition issue, but the actual failure was buried deeper in the WebLogic configuration logs: the WebLogic Node Manager password could not be decrypted during fs2 configuration.

    After correcting the WebLogic credential configuration and re-running AutoConfig and Preclone on the RUN file system, fs_clone force=yes completed successfully.

    Context: Oracle EBS R12.2 uses WebLogic Server in the application tier. File system cloning involves WebLogic configuration and credentials when configuring the PATCH file system from the RUN file system.

    Environment

    Oracle E-Business Suite: R12.2
    Database: Oracle 19c
    Application tier: Multiple nodes
    RUN file system: fs1
    PATCH file system: fs2
    Clone method: Full file system synchronization

    RUN File System: /u01/oracle//fs1
    PATCH File System: /u01/oracle//fs2

    The Failure

    The adop phase=fs_clone operation reported:

    ERROR: Unable to set edition. Invalid edition type : PATCH

    Followed by:

    adclone.pl did not go through successfully.
    EXIT STATUS: 1

    These messages indicated failure but not the cause. The detailed WebLogic configuration logs revealed the actual exception.

    Actual Failure: WebLogic Password Decryption

    While WLST configured Node Manager credentials on fs2, the logs showed:

    File “…/txkSetAppsLibsTarget.py”, line 705, in setNodeManagerCredentials

    java.lang.AssertionError:
    java.lang.reflect.InvocationTargetException

    The stack trace continued through:

    SecurityConfigurationMBeanImpl.getNodeManagerPassword

    EncryptionServiceException:
    com.rsa.jsafe.JSAFE_PaddingException: Invalid padding

    The failure chain:

    txkSetAppsLibsTarget.py
    -> setNodeManagerCredentials()
    -> getNodeManagerPassword()
    -> WebLogic decrypt()
    -> JSAFE_PaddingException: Invalid padding

    WebLogic was unable to decrypt the Node Manager password during fs2 configuration. This pointed to a WebLogic credential/encryption configuration issue rather than a filesystem capacity or OS resource issue.

    Investigation

    Before pinpointing the WebLogic credential issue, these areas were examined:

    Filesystem Space
    The application filesystem had adequate free space. The FMW Home cloning process had progressed substantially before failing, making a simple disk-space issue unlikely.

    Port Allocation
    The clone successfully acquired all required temporary ports. No port conflicts were identified.

    Operating System
    OS logs showed no evidence of: OOM kills, Java process termination, segmentation faults, core dumps, or hs_err_pid files.

    FMW Home Cloning
    The pasteBinary operation progressed to approximately 90% before stopping. However, the actionable error was the WebLogic decryption failure:

    getNodeManagerPassword
    JSAFE_PaddingException: Invalid padding

    The 90% progress point indicated where the clone stopped, not the root cause.

    Resolution

    After reviewing the issue with a senior DBA, the WebLogic credential configuration was corrected and the RUN file system configuration was refreshed before retrying the clone.

    The following steps resolved the issue:

    1. Update boot.properties
    Update the WebLogic boot.properties file with the appropriate credentials as part of correcting the WebLogic credential configuration.

    2. Restart WebLogic
    Restart WebLogic to load the updated credential configuration.

    3. Run AutoConfig on the RUN File System
    cd $COMMON_TOP/admin/scripts
    ./adautocfg.sh

    This regenerates EBS configuration on the RUN file system.

    4. Run Preclone on the RUN File System
    cd $ADMIN_SCRIPTS_HOME
    perl adpreclone.pl appsTier

    Preclone refreshes the RUN file system configuration used for the subsequent clone operation.

    5. Rerun fs_clone in Force Mode
    adop phase=fs_clone force=yes

    The operation completed successfully.

    Result

    The PATCH file system was successfully rebuilt after the WebLogic credential configuration was corrected.

    adop phase=fs_clone force=yes -> SUCCESS

    Troubleshooting Takeaways

    Don’t Stop at Generic Errors

    Messages such as “adclone.pl did not go through successfully” or “Unable to set edition. Invalid edition type : PATCH” only indicate that the clone failed. Always examine the detailed FSCloneApplyAppsTier and WLST logs to identify the actual component failure.

    WebLogic Stack Traces Matter

    If you encounter this pattern:
    setNodeManagerCredentials
    getNodeManagerPassword
    JSAFE_PaddingException: Invalid padding

    Investigate the WebLogic credential and encryption configuration. Do not assume filesystem space, port allocation, or OS resource exhaustion is responsible.

    Look Beyond the Final FAILED Message

    The most useful error is often several layers below the final adop failure. In this case:

    SecurityConfigurationMBeanImpl.getNodeManagerPassword
    com.rsa.jsafe.JSAFE_PaddingException: Invalid padding

    This identified the WebLogic password decryption step as the failure point.

    The PATCH Edition Message Was Not the Root Cause

    The message “Unable to set edition. Invalid edition type : PATCH” appeared during the investigation but was not the cause of the fs_clone failure. The actionable failure was the WebLogic password decryption error.

    Summary

    The adop phase=fs_clone failure occurred while configuring WebLogic Node Manager credentials during PATCH file system construction. The failure occurred because WebLogic was unable to decrypt the Node Manager password during the Node Manager credential configuration step:

    java.lang.AssertionError:
    java.lang.reflect.InvocationTargetException

    SecurityConfigurationMBeanImpl.getNodeManagerPassword

    com.rsa.jsafe.JSAFE_PaddingException: Invalid padding

    The resolution involved correcting the WebLogic boot.properties configuration, restarting WebLogic, and re-running AutoConfig and Preclone on the RUN file system. A subsequent fs_clone force=yes completed successfully.

    Note: This resolution is based on remediation in a specific environment. JSAFE_PaddingException errors during EBS cloning can have different underlying causes. Always review the detailed WebLogic and WLST logs before applying this remediation to another environment.

    Disclaimer: All environment identifiers, hostnames, IP addresses, and server paths in this article have been masked with generic placeholders to protect production system information and organizational confidentiality. The technical troubleshooting methodology and error signatures remain directly applicable to similar incidents in your environment.

  • Oracle EBS R12.2 Multi-Node Patching in Downtime Mode: Recovering a Failed Slave Node with restart=yes

    Alhamdulillah, sharing another real patching session — this one from an Oracle E-Business Suite R12.2 multi-node environment. While applying a patch in downtime mode, I encountered an interesting situation: the patch completed successfully on the master node but failed on the slave node, while the final ADOP output still showed:

    adop exiting with status = 0 (Success)

    This post walks through the complete sequence: pre-checks, the slave-node failure, investigation, recovery using restart=yes, and the SQL checks used to verify the final patch state.

    Environment

    • Oracle E-Business Suite R12.2.13
    • AD/TXK Delta 15 (ADOP C.Delta.15)
    • Two application nodes: appsnode1 (master/admin node) and webnode1 (slave/web node)
    • Oracle Database 19c on dbnode1
    • Patch mode: downtime (single file system apply; no online patching cutover involved)

    Pre-Checks Before the Apply

    Before starting the patch, I checked three things: the current ADOP session status, application processes on all application nodes, and the current file-system edition.

    From the master node:

    [applmgr@appsnode1]$ adop -status
    Node Name       Node Type  Phase           Status
    --------------- ---------- --------------- ---------------
    appsnode1       master     APPLY           ACTIVE
                               CLEANUP         NOT STARTED
    webnode1        slave      APPLY           ACTIVE
                               CLEANUP         NOT STARTED

    I also confirmed I was operating from the run edition:

    [applmgr@appsnode1]$ echo $FILE_EDITION
    run

    Then I checked for application-tier processes:

    [applmgr@appsnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|FNDCRM|FNDOPP|oacore|forms|WebLogic|NodeManager' | grep -v grep

    There was no output, confirming the expected services were down on the master node.

    Important: In a multi-node environment, repeat these process checks on every application node. Checking only the master node is not sufficient — a condition on a slave node can otherwise go unnoticed until the apply reaches it.

    Starting the Downtime Apply

    The downtime apply was initiated from the master node:

    [applmgr@appsnode1]$ adop phase=apply apply_mode=downtime patches=<patch_number>

    ADOP manages execution across the registered application-tier nodes, so there was no need to manually start ADOP on the slave node.

    During initialization, ETCC reported missing database fixes:

    [WARNING]   ETCC: The following required database fixes have not been applied to node dbnode1:
                ...
                Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

    ETCC warnings should always be reviewed before proceeding. In this case, the apply was allowed to continue based on the maintenance plan, while the outstanding database fixes were documented and tracked separately for a subsequent maintenance activity.

    Master Succeeded, Slave Failed

    The patch application succeeded on the master node but failed on the slave:

    Applying <patch> patch(es) on admin node: [appsnode1].
            txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
        [ERROR]     adop phase=apply failed on Node: "webnode1"
    Summary report for current adop session:
        Node webnode1: Failed
           - Apply status:      Failed
        Node appsnode1: Completed successfully
           - Apply status:      Completed successfully
    adop exiting with status = 0 (Success)

    This is an important operational lesson. Despite the slave-node failure being clearly reported in the summary, the ADOP invocation ended with adop exiting with status = 0 (Success).

    Therefore, relying only on the shell return code or the final ADOP exit line is not sufficient for determining whether every application node completed successfully. The per-node summary must also be reviewed.

    I confirmed the state with:

    adop -status -detail

    which showed:

    Node Name       Node Type  Phase           Status
    --------------- ---------- --------------- ---------------
    appsnode1       master     APPLY           ACTIVE
    webnode1        slave      APPLY           FAILED

    For automated patching this matters even more: scripts should validate the ADOP session and per-node phase status rather than treating $? = 0 alone as proof of a successful multi-node patch.

    Investigating the Slave Node

    I then logged into webnode1. Application services were expected to be completely down, so I checked for NodeManager:

    [applmgr@webnode1]$ ps -ef | grep -i nodemanager | grep -v grep
    applmgr  4148261  1  0 Jun18 ... weblogic.NodeManager -v

    A NodeManager process was still running, and had been since a previous month.

    I checked whether other WebLogic or EBS application processes were present:

    [applmgr@webnode1]$ ps -ef | grep -i weblogic | grep -v grep

    Only the NodeManager process remained. Concurrent Manager processes were also absent:

    [applmgr@webnode1]$ ps -ef | grep FNDLIBR | grep -v grep

    No output was returned.

    Since the environment was in a planned patching outage and no managed servers were running, the stale NodeManager process was stopped:

    [applmgr@webnode1]$ kill 4148261

    I then verified again:

    [applmgr@webnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|oacore|forms|WebLogic|NodeManager' | grep -v grep

    No relevant processes remained.

    A Note About Root Cause

    The presence of the stale NodeManager was an abnormal condition that needed to be corrected before retrying the patch. However, unless the ADOP logs explicitly identify that process as the reason for the failure, it is safer not to state that NodeManager was definitively the root cause.

    What can be established from this incident is:

    1. The slave-node apply failed.
    2. A stale NodeManager process was discovered on that node.
    3. The process was stopped after confirming application services were intended to be down.
    4. The failed ADOP apply subsequently completed successfully with restart=yes.

    For a formal RCA, the relevant ADOP and slave-node logs should be used to establish the exact failure mechanism.

    Also, before terminating any process, verify that:

    • It belongs to the expected EBS environment/file system.
    • No managed servers depend on it.
    • Application services are intended to be down.
    • The action complies with your production change procedure.

    Recovering the Failed Apply with restart=yes

    After cleaning up the slave node, I returned to the master node. The existing failed apply was resumed using:

    [applmgr@appsnode1]$ adop phase=apply patches=<patch_number> apply_mode=downtime restart=yes

    ADOP detected the existing session:

    Checking for existing adop sessions.
    Application tier services are down.
    Continuing with the existing session [Session ID: <n>].

    The apply was then processed successfully:

    Applying <patch> patch(es) on admin node: [appsnode1].
            txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
            txkADOPEvalSrvStatus.pl returned SUCCESS

    The final summary showed both nodes successful:

    Summary report for current adop session:
        Node webnode1: Completed successfully
           - Apply status:      Completed successfully
        Node appsnode1: Completed successfully
           - Apply status:      Completed successfully

    The important point here is that I resumed the existing ADOP session rather than initiating an unrelated fresh patching attempt. The recovery command was executed from the master/admin node, allowing ADOP to coordinate processing on the registered slave node.

    For the exact semantics of restart=yes in downtime mode and multi-node processing, refer to the Oracle E-Business Suite Patching Guide and the relevant My Oracle Support documentation. What I can report from this incident is the observed behaviour above: ADOP continued the existing session and both nodes subsequently reported success.

    Post-Restart Validation

    After the successful rerun, I checked the detailed ADOP status:

    [applmgr@appsnode1]$ adop -status -detail

    The nodes no longer showed a failed apply state.

    I also scanned the latest ADOP logs:

    [applmgr@appsnode1]$ adopscanlog -latest=yes
    Scanning .../log/adop/<session>/ directory ...
    No Errors.

    adopscanlog is a useful quick check after an ADOP phase because it searches the session logs for reported errors that may otherwise be easy to miss.

    SQL Verification: Confirming Patch Registration

    I don’t rely only on the ADOP console output. After patching, I also verify the database-side patch records.

    1. Check AD_BUGS

    SET LINES 200
    COL BUG_NUMBER FORMAT A15
    COL CREATION_DATE FORMAT A22
    SELECT bug_number,
           creation_date
    FROM   ad_bugs
    WHERE  bug_number = '<patch_number>';

    This confirms whether the specified bug/patch number is recorded in AD_BUGS. Because patch contents and registration behaviour can vary, this check should be used together with ADOP status, logs, and the relevant applied-patch records rather than as standalone proof of complete patch application. Running the same query before patching is also useful, because it establishes a clear before-and-after state.

    2. Check Applied Patch Records

    SELECT applied_patch_id,
           patch_name,
           patch_type,
           source_code,
           creation_date,
           last_update_date,
           data_model_done_flag
    FROM   ad_applied_patches
    WHERE  patch_name = '<patch_number>';

    Depending on the patch structure, this helps confirm the corresponding patch-driver records.

    3. Verify AD/TXK Code Levels

    COL ABBREVIATION FORMAT A15
    COL CODELEVEL FORMAT A25
    SELECT abbreviation,
           codelevel
    FROM   ad_trackable_entities
    WHERE  abbreviation IN ('ad','txk')
    ORDER  BY abbreviation;

    For this environment the query returned:

    ABBREVIATION    CODELEVEL
    --------------- -------------------------
    ad              C.15
    txk             C.15

    This check is especially important when applying AD/TXK-related patches or preparing for an EBS Release Update Pack.

    4. Verify the EBS Release

    SELECT release_name
    FROM   fnd_product_groups;

    Expected for this environment: 12.2.13.

    5. Verify Node Registration

    COL NODE_NAME FORMAT A25
    SELECT node_name,
           support_cp,
           support_forms,
           support_web,
           support_admin
    FROM   fnd_nodes
    ORDER  BY node_name;

    This provides a quick sanity check of application-tier node registration and responsibilities.

    6. Verify ADOP Valid Nodes

    SELECT *
    FROM   adop_valid_nodes
    ORDER  BY node_name;

    Both application nodes should appear as expected.

    In this case, the final checks confirmed:

    • The bug/patch number was recorded in AD_BUGS.
    • The corresponding applied-patch records were present.
    • AD and TXK were at the expected C.15 level.
    • EBS remained at Release 12.2.13.
    • Both application nodes were properly registered in FND_NODES.
    • Both nodes appeared correctly in ADOP_VALID_NODES.
    • adopscanlog reported no errors after the successful rerun.

    Together, these checks provide much stronger verification than relying on the final ADOP console message alone.

    Complete the Session with Cleanup

    After confirming the apply completed successfully, the session still showed:

    CLEANUP         NOT STARTED

    The cleanup phase was then run:

    adop phase=cleanup

    After cleanup, perform another status and log review to make sure the session is in the expected final state.

    Key Takeaways

    Don’t rely only on ADOP’s exit status in a multi-node environment. In this incident, ADOP printed status = 0 (Success) even though the per-node summary clearly showed the slave node had failed.

    Check every application node before a downtime apply. A clean process list on the master does not guarantee the slave nodes are equally clean.

    Use the per-node ADOP status as part of your success criteria — adop -status -detail.

    Investigate the failed node before retrying. In this case a stale NodeManager process was discovered and removed before the successful retry. Unless supported by the logs, however, avoid treating correlation as a confirmed root cause.

    Resume the existing failed apply appropriately — after correcting the condition on the slave node, the existing apply was resumed from the master node with restart=yes.

    Use adopscanlog -latest=yes after the rerun.

    Verify patch registration in the database — AD_BUGS, AD_APPLIED_PATCHES, AD_TRACKABLE_ENTITIES, FND_NODES, ADOP_VALID_NODES, FND_PRODUCT_GROUPS.

    Run ETCC and review its findings. If required database fixes are deferred, document and track them for the appropriate maintenance window rather than allowing them to become forgotten technical debt.

    Complete the ADOP cleanup phase — adop phase=cleanup.

    Final Thought

    Multi-node EBS patching adds a layer of validation that is easy to overlook: success on the master node does not automatically mean success across the application tier.

    The most useful lesson from this incident was not the retry command itself, but the validation process around it:

    Check every node → read the ADOP per-node summary → investigate the failed node → resume the existing session → scan the logs → verify the database records → complete cleanup.

    That sequence gives you a far more defensible and auditable patching result than relying on a single Success message at the end of the command.

    I hope this helps anyone troubleshooting a failed application-tier node during Oracle E-Business Suite R12.2 downtime patching. Feel free to leave a comment if you have questions.

    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS R12.2 AutoConfig Template Version Conflict After Patching: How to Preserve Customizations

    Alhamdulillah, sharing another real troubleshooting session — this one from an Oracle E-Business Suite R12.2 multi-node environment. After applying a patch, AutoConfig aborted on the primary application node with a template version conflict. This post walks through the exact error, how I identified which template entries were genuine customizations versus Oracle-shipped content, and the correct way to rebase a custom template so AutoConfig completes cleanly.

    Environment

    • Oracle E-Business Suite R12.2.13, AD/TXK Delta 15
    • Two application nodes: appsnode1 (primary/admin) and webnode1 (secondary web node)
    • Database: Oracle 19c on dbnode1
    • Third-party customization: More4Apps servlet registered in oacore web.xml

    The Error

    After patching, running AutoConfig on the run file system failed immediately:

    [applmgr@appsnode1 scripts]$ sh adautocfg.sh
Enter the APPS user password:
The log file for this session is located at: .../admin/log/<timestamp>/adconfig.log
wlsDomainName: EBS_domain
WLS Domain Name is VALID.
AutoConfig is configuring the Applications environment...
AutoConfig will consider the custom templates if present.
Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

    The log pinpointed the conflicting template:

    [ FND_TOP ]

[ VERSION CONFLICTS INFORMATION ]
Template shipped by oracle is having a version different than the template lying in custom directory.
Template shipped by Oracle : $FND_TOP/admin/template/oacore_web_xml_FMW.tmp (version: 120.17.12020000.33)
Custom template            : $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp (version: 120.17.12020000.31)
Please resolve the differences between the two templates or refer to the Oracle E-Business Suite Setup Guide for further details.

    What happened is straightforward: the patch delivered a newer version of oacore_web_xml_FMW.tmp (120.17.12020000.33), but a custom copy of the older version (120.17.12020000.31) exists under $FND_TOP/admin/template/custom. AutoConfig detects the mismatch and refuses to run, because blindly using the stale custom template would silently discard whatever Oracle changed in the new version.

    Root Cause Analysis: What Is Actually a Customization?

    This is the critical analysis step. Before touching anything, you need to answer one question precisely: which lines in the custom template are genuine customizations, and which are just old Oracle-shipped content?

    First, take a backup and diff the new shipped template against the custom one:

    cd $FND_TOP/admin/template

cp -p custom/oacore_web_xml_FMW.tmp \
      custom/oacore_web_xml_FMW.tmp.pre_patch_25Jul2026

diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

    The diff showed three differences beyond the header:

    1. A m4aServlet servlet and servlet-mapping (More4Apps) present only in the custom template
    2. A SyncServlet servlet and servlet-mapping present only in the custom template
    3. A RequestAuditReport (QP) servlet present only in the new Oracle template

    At first glance it looks like there are two customizations to carry forward: m4aServlet and SyncServlet. This is where a wrong assumption would corrupt the template. To verify, I compared the old shipped template against the new shipped template. In an R12.2 dual file system this is easy — the other file system still had the previous patch level, so both shipped versions were available:

    diff -u \
  <old_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp \
  <new_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp

    Key excerpt from the shipped-vs-shipped diff:

    -   <servlet>
-     <servlet-name>SyncServlet</servlet-name>
-     <servlet-class>oracle.apps.jtf.cac.sync.transport.SyncServlet</servlet-class>
-  </servlet>
...
+     <servlet>
+         <servlet-name>RequestAuditReport</servlet-name>
+         <servlet-class>oracle.apps.qp.servlet.RequestAuditReport</servlet-class>
+    </servlet>

    This settled the question:

    • SyncServlet was Oracle-shipped content in version .31 and is no longer present in the .33 shipped template. It was never a customer customization, so it should NOT be carried forward.
    • RequestAuditReport is new Oracle-shipped content in .33 — it must be retained.
    • m4aServlet was not present in either shipped template version compared — it exists only in the custom directory, which identifies it as the genuine site-specific customization.

    If I had merged the old custom template on top of the new one, I would have reintroduced a servlet that is no longer present in Oracle’s newer shipped template. This is why “just copy your custom entries into the new template” advice found on many forums is dangerous — verify against the shipped versions first.

    The Fix: Rebase the Custom Template

    The correct approach: replace the custom template with the new shipped version, then re-apply only the genuine customization.

    cd $FND_TOP/admin/template

# Rebase custom template on the new shipped version
cp -p oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

# Re-insert the only genuine customization (m4aServlet)
sed -i '/<!-- FND Servlets -->/a\
\
  <servlet>\
    <servlet-name>m4aServlet</servlet-name>\
    <servlet-class>com.more4apps.r12.servlet.XmlServlet</servlet-class>\
  </servlet>\
\
  <servlet-mapping>\
    <servlet-name>m4aServlet</servlet-name>\
    <url-pattern>/m4aServlet/*</url-pattern>\
  </servlet-mapping>\
' custom/oacore_web_xml_FMW.tmp

    This matches the procedure in the Oracle E-Business Suite Setup Guide: when a patch delivers a newer version of a template you have customized, copy the new Oracle template into the custom directory and reapply the still-required customizations to that copy — never the other way around.

    Verify before rerunning AutoConfig — the headers must match, and the only diff must be the customization:

    grep '\$Header' oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

oacore_web_xml_FMW.tmp:        version 120.17.12020000.33
custom/oacore_web_xml_FMW.tmp: version 120.17.12020000.33

diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
--- only the m4aServlet servlet + mapping block should appear ---

    As an additional best-practice validation, Oracle recommends running the AutoConfig configuration-check utility to preview the impact of template changes before the actual AutoConfig run:

    $AD_TOP/bin/adchkcfg.sh contextfile=$CONTEXT_FILE

    Rerun AutoConfig:

    cd $ADMIN_SCRIPTS_HOME
sh adautocfg.sh

...
Configuring templates from all of the product tops...
        Configuring AD_TOP........COMPLETED
        Configuring FND_TOP.......COMPLETED
        ...
AutoConfig completed successfully.

    A Bonus Finding on the Second Node

    While repeating the check on the secondary web node, the header comparison came back clean — custom template already at .33 — but something was off:

    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp
(no output)

grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp_ORIG
61:    <servlet-name>m4aServlet</servlet-name>

    The current custom template matched the newer shipped version, but the m4aServlet customization was missing. Interestingly, the _ORIG backup still contained the customization. This indicated that during an earlier template update, the customization was not carried forward. AutoConfig had been running fine and web.xml was being generated — just without the third-party servlet.

    I re-inserted the m4aServlet block using the same sed command, verified the diff, and ran AutoConfig on that node as well:

    AutoConfig completed successfully.

    This is the real lesson of the post. A template version conflict at least fails loudly. A customization that is accidentally omitted during a template rebase may not — AutoConfig can complete successfully while the generated configuration no longer contains the required customization, and you find out later when the third-party tool stops working after a bounce.

    An Additional Message Observed

    The AutoConfig log in this run also contained:

    ECC not enabled, setting FND_ECC_ENABLED to FALSE
[ FND_ECC_ENABLED ]
INFO : Error updating/creating profile option value.

    In this case Enterprise Command Center is not configured, AutoConfig continued past this message, and the run ultimately exited with status 0. Always evaluate such messages in the context of your own environment rather than assuming every INFO : Error entry can be ignored.

    Post-Fix Checklist

    1. Confirm the customization landed in the generated web.xml (get the target path from the adconfig log and grep for your servlet).
    2. Bounce the application services (at least the oacore managed servers) so the new web.xml is deployed.
    3. Test the customization end to end — for More4Apps, hit the servlet URL from the wizard.
    4. If your patching cycle requires it, regenerate appsutil.zip (perl $AD_TOP/bin/admkappsutil.pl) and refresh the database tier.
    5. Repeat the custom template verification on every application node — as seen above, nodes can drift.

    Key Takeaways

    • Never resolve this error by copying the old custom template over the new shipped one, and never blindly merge either. The Oracle-documented procedure is to copy the new shipped template into the custom directory and re-apply only verified customizations to that copy.
    • Use the dual file system to your advantage: diff old-shipped vs new-shipped to separate Oracle’s changes from your customizations.
    • After any template rebase, diff the shipped and custom templates — the output should contain nothing but your customizations.
    • Audit custom templates on all nodes periodically. A missing customization may not trigger an AutoConfig error, allowing the generated configuration to differ from what you expect.

    I hope this helps someone facing the same conflict. Feel free to leave a comment if you have questions.

    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • EBS 12.2 Rapid Clone dbTechStack Fails with RC-50013 — INS-08101 supportedOSCheck on Oracle Linux 8

    Alhamdulillah, sharing another real troubleshooting session — this one from an EBS 12.2 database tier clone on Oracle Linux 8.10, where adcfgclone.pl dbTechStack failed and the on-screen error pointed in completely the wrong direction.

    Environment

    Oracle EBS 12.2, Oracle Database 19c (19.21) multitenant (CDB/PDB), Oracle Linux 8.10, target server testserver01.

    The Symptom

    perl adcfgclone.pl dbTechStack /u01/EBSDB/19.0.0/appsutil/EBSDB_testserver01.xml

RC-00110: Fatal: Error occurred while relinking of ApplyDBTechStack
ERROR while running Apply...
ERROR: Failed to execute /u01/EBSDB/19.0.0/appsutil/clone/bin/adclone.pl

    The first instinct with RC-00110 is to suspect a relink problem — missing OS packages, a corrupt backup, bad extraction. We verified all of that and ruled it out. The real lesson of this post: the RC error on screen is generic. Follow the log chain.

    The Log Chain

    Step 1 — The ApplyDBTechStack log showed the relink script actually passed:

    adlnkoh.sh completed sucessfully

    but home registration failed:

    Finished OUI CLI cloning for s_db_oh with return code: 1
ouicli.pl  INSTE8_APPLY  1
RC-50013: Fatal: Instantiate driver did not complete successfully.

    Step 2 — ohclone.log pointed one level deeper:

    OUI runinstaller log file - /u01/EBSDB/oraInventory/logs/InstallActions<timestamp>/installActions<timestamp>.log
Found the INFO: Exit Status is -1 in runInstaller log.
OUI CLI cloning returned non-zero.

    Note the timing: runInstaller started and exited within the same second. An installer that dies instantly is not failing a task — it is failing a pre-check.

    Step 3 — The installActions log had the true error:

    [WARNING] [INS-08101] Unexpected error while executing the action at state: 'supportedOSCheck'
SUMMARY: - java.lang.NullPointerException

    Root Cause

    The 19c ORACLE_HOME delivered inside the EBS Rapid Clone stage carries the 19.3 base installer (year 2019). Its OS certification table ends at Oracle Linux 7. On OL8/RHEL8, the lookup for the distribution ID returns null and the installer crashes with a NullPointerException before doing any work — surfacing back up the chain as RC-50013 and RC-00110.

    The Fix

    Set CV_ASSUME_DISTID in the same shell session before rerunning the clone:

    export CV_ASSUME_DISTID=OEL7.8
echo $CV_ASSUME_DISTID
cd /u01/EBSDB/19.0.0/appsutil/clone/bin
perl adcfgclone.pl dbTechStack /u01/EBSDB/19.0.0/appsutil/EBSDB_testserver01.xml

    Result — the rerun completed cleanly, confirmed by these lines in the new ApplyDBTechStack log:

    Finished OUI CLI cloning for s_db_oh with return code: 0
Completed home registration for s_db_oh
Completed Apply...
ApplyDBTechStack Completed Successfully.

    Important: this variable does NOT change your OS or the installed software. It only tells the old installer which certification profile to use for its checks. The value must be one the 19.3-base installer recognizes — OEL7.8 is the value documented in Oracle’s Linux 8 release notes. Setting it to your actual OS version (8.10) defeats the purpose.

    Permanent Fix

    The workaround can also live inside the ORACLE_HOME, in $ORACLE_HOME/cv/admin/cvu_config:

    # Fallback to this distribution id
CV_ASSUME_DISTID=OEL7.8      <-- uncomment and set

    This also explains a common confusion: “my previous clone worked without this!” If the earlier backup was taken from a home where cvu_config already had this set, the fix travelled inside the backup. A backup from an untouched home reintroduces the failure. Check with:

    grep -i CV_ASSUME_DISTID $ORACLE_HOME/cv/admin/cvu_config

    Post-Clone Notes

    After the fix, you will still see FATAL errors like “DB Connection failed” and “Invalid APPS database user credentials” from the AutoConfig phase, plus DB-ETCC connectivity warnings — these are expected at this stage because the database is not restored and opened yet. They resolve once you restore the database and rerun AutoConfig and ETCC.

    Also refresh ETCC from patch 17537119 — the bundled bugfix XML goes stale after 30 days, and older versions do not recognize newer Release Updates such as 19.21.

    Key Takeaways

    RC-00110/RC-50013 are wrappers, not root causes — always walk ApplyDBTechStack log → ohclone.log → installActions log. A runInstaller that exits in under a second failed a pre-check, not the work itself. And on OL8/RHEL8 with any 19.3-base home, CV_ASSUME_DISTID=OEL7.8 belongs in your clone runbook.

    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • ORA-00060 Deadlock Forensics in Oracle EBS: When a New APEX Scheduler Job Meets Workflow

    Alhamdulillah, another interesting production incident to share. This one is a classic example of how a small code change — a single commented-out COMMIT — can stay silent for weeks until a new concurrent workload exposes it as an Oracle deadlock in production. If you are searching for Oracle deadlock troubleshooting or ORA-00060 trace file analysis in an Oracle EBS environment, this walkthrough covers the full investigation from symptom to fix.

    Quick Summary

    • Issue: ORA-00060 deadlocks after deployment of a new APEX DBMS_SCHEDULER job.
    • Impact: Oracle Workflow Background Process intermittently failed, delaying order processing.
    • Root Cause: A previously removed COMMIT caused row locks to be held far longer than intended, allowing concurrent sessions to deadlock.
    • Resolution: Restored the correct transaction boundary, adjusted scheduler timing, and verified with Oracle Support that no product-side issue was involved.
    • Result: No recurrence after deployment.

    The Symptom

    On a production EBS environment (EBS 12.1, Database 12.1.0.2), we started receiving alerts for ORA-00060 in the database alert log:

    ORA-00060: Deadlock detected. More info in file
/u01/app/oracle/diag/rdbms/prod/PROD/trace/PROD_ora_12345.trc

    At the same time, the business reported sales order lines not progressing. The Oracle Workflow Background Process for the OM Order Line workflow (ONT) was erroring intermittently — a visible impact on order processing and a classic Oracle EBS performance troubleshooting scenario.

    A quick note on deadlock behavior: Oracle detects deadlocks automatically and resolves them by rolling back one of the participating statements — the victim — allowing the remaining transaction to continue. Unlike simple blocking, only the offending statement is rolled back, not the victim’s entire transaction. The victim’s session receives ORA-00060 and a trace file is written to disk. That is why the application does not hang forever; instead, you see intermittent errors and trace files accumulating.

    Step 1 – Read the Deadlock Trace

    The trace file is always the starting point of ORA-00060 trace file analysis. The deadlock graph showed two sessions blocking each other on TX enqueues in row-exclusive mode:

    Deadlock graph:
                       ---------Blocker(s)--------  ---------Waiter(s)---------
Resource Name          process session holds waits  process session holds waits
TX-000a0015-0003c2d1        45     312     X             38     127           X
TX-0008001f-0004a1b2        38     127     X             45     312           X

    Both sessions were waiting on enq: TX - row lock contention, each holding a row the other wanted — textbook Oracle row lock contention forming a deadlock cycle. The trace file also identifies which session was chosen as the victim and the exact SQL statement that was rolled back, which makes it the definitive evidence for the investigation.

    Conceptually, the cycle looked like this:

    Workflow Background Process
        │
        │ Locks Row A
        ▼
Reservation Row A
        │
        │ Waiting for Row B
        ▼

Reservation Row B
        ▲
        │ Locked by
        │
APEX Scheduler Job

Workflow waits for APEX.
APEX waits for Workflow.

→ Oracle detects a deadlock (ORA-00060)

    Step 2 – Identify the Two Sessions

    SET LINES 200 PAGES 100
COL SID       FORMAT 99999
COL SERIAL#   FORMAT 999999
COL USERNAME  FORMAT A12
COL PROGRAM   FORMAT A28
COL MODULE    FORMAT A28
COL ACTION    FORMAT A20
COL SQL_ID    FORMAT A14

SELECT s.sid, s.serial#, s.username,
       NVL(s.program,'N/A')  program,
       NVL(s.module,'N/A')   module,
       NVL(s.action,'N/A')   action,
       NVL(s.sql_id,'N/A')   sql_id
FROM   v$session s
WHERE  s.sid IN (312, 127);

    Session 1 was the Workflow Background Process (FNDWFBG, ONT item type) — making this an Oracle Workflow deadlock scenario. Session 2 was more interesting — a DBMS_SCHEDULER job session:

    COL JOB_NAME        FORMAT A25
COL OWNER           FORMAT A12
COL STATE           FORMAT A12
COL REPEAT_INTERVAL FORMAT A40

SELECT owner, job_name, state, repeat_interval
FROM   dba_scheduler_jobs
WHERE  job_name = 'APEX_ORDER_RESERVE';

    A newly deployed APEX-driven scheduler job, running every few minutes, calling a custom reservation package — the second half of an Oracle DBMS_SCHEDULER deadlock pattern.

    Step 3 – Which Rows Were They Fighting Over?

    From the trace file, the “Rows waited on” section gives the object number:

    COL OWNER       FORMAT A10
COL OBJECT_NAME FORMAT A30
COL OBJECT_TYPE FORMAT A12

SELECT owner, object_name, object_type
FROM   dba_objects
WHERE  object_id = &object_id_from_trace;

    Both sessions were colliding on reservation rows — the Workflow process updating them as part of order line progression, and the APEX job updating the same rows through the custom package.

    Step 4 – The Root Cause

    Reviewing the custom reservation package source, we found this:

       UPDATE xx_order_reservations
   SET    status = 'RESERVED'
   WHERE  header_id = p_header_id;

   -- COMMIT;   <==  commented out during a previous change

    The investigation identified that the custom package was holding row locks far longer than intended, because a COMMIT had been removed in an earlier code change. Instead of releasing locks per iteration, the scheduler job session held all of its row locks across the entire loop over order headers. It is worth being precise here: a missing COMMIT by itself does not cause a deadlock — a deadlock requires two sessions acquiring locks on overlapping rows in conflicting order. What the missing COMMIT did was dramatically widen the lock-holding window, so when the new APEX scheduler job started running concurrently with the Workflow Background Process against the same reservation rows, the probability of the two sessions interleaving into a deadlock cycle went from negligible to near-certain.

    This also explains why the package ran in production for weeks without issue — until the APEX job was deployed, nothing else contended for those rows at that frequency. It also explains why testing never caught it: lower environments rarely generate the same level of concurrent activity as production, making lock-contention issues extremely difficult to reproduce before go-live. The extended lock duration was always present; the new concurrent workload exposed it.

    The Fix

    1. Restored the COMMIT at the correct transactional boundary in the custom package (per-iteration, after each header’s reservation update), shrinking the lock-holding window.
    2. Redeployed the package during an approved change window.
    3. Rescheduled the APEX job to avoid peak Workflow Background Process cycles as an additional safety margin.
    4. Raised an SR with Oracle Support to confirm no product-side involvement — confirmed clean; purely custom code.

    No ORA-00060 recurrence since the fix, ما شاء الله.

    Lessons Learned

    • An Oracle deadlock almost always has two contributors: the locking pattern AND the concurrency pattern. Fixing either breaks the cycle, but fix the code defect, not just the schedule.
    • Commented-out COMMITs are silent time bombs. Code review for custom PL/SQL touching EBS transactional tables must treat transaction boundaries as seriously as the DML itself.
    • New scheduler jobs (APEX, DBMS_SCHEDULER, concurrent programs) should be assessed for row-level contention with existing Oracle Workflow and concurrent processing before go-live.
    • The deadlock trace file gives you everything: the sessions, the victim, the SQL, the rows. Start there, not with guesswork.

    Conclusion

    Production incidents often reveal issues that remain hidden during testing, because realistic concurrency is difficult to reproduce in lower environments. This incident reinforced a simple discipline: whenever a new workload is introduced — an APEX scheduler job, a concurrent program, or an integration — review the transaction boundaries of every custom PL/SQL object it touches, and ask what else updates those same rows.

    In Oracle, deadlocks are rarely caused by a single statement — they are caused by the interaction of multiple sessions under concurrency. Understanding transaction boundaries is often the key to solving them.

    Disclaimer: All environment names, hostnames, and identifiers in this post are anonymized. The views expressed are my own.

  • Oracle EBS R12.2 Clone Troubleshooting: RC-50221 Port Pool Conflict and FRM-92101 Forms Session Failed During Startup

    Disclaimer: This article is based on field experience and publicly shareable troubleshooting steps. Oracle MOS notes are referenced for further reading. Customers with active Oracle Support should review the referenced MOS documents for complete guidance.

    Category: Oracle EBS R12.2 | Cloning | Troubleshooting
    Level: Intermediate to Advanced
    Reference: MOS Doc ID 454427.1

    Introduction

    Cloning an Oracle E-Business Suite R12.2 application tier is a common activity — whether for setting up a DEV, TEST, or UAT environment. While the process is well-documented, real-world scenarios often throw up issues that aren’t immediately obvious from the error messages alone.

    In this post, I walk through two common issues encountered during an EBS R12.2 app tier clone using adcfgclone.pl:

    1. RC-50221: Port Pool Not Free — CloneContext failing due to an already-occupied port pool
    2. FRM-92101: Forms Session Failed During Startup — Forms not launching due to missing library dependencies or invalid symlinks

    Both issues are real, well-known, and worth documenting together since they often appear in sequence during a clone.

    Environment Details

    • Oracle EBS Version: R12.2.x
    • OS: Oracle Linux 7 (OL7) / RHEL 7
    • Clone type: Application Tier only (adcfgclone.pl appsTier)
    • Target server: Fresh clone target (non-production)

    Issue 1: RC-50221 — Port Pool Not Free

    Symptom

    When running adcfgclone.pl appsTier, the CloneContext step prompts for a Target System Port Pool and throws the following warning/error:

    Checking the port pool 0
RC-50221: Warning: Port Pool 0 is not free.
Please check logfile $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log for conflicts.
RC-00201: Error: Not a valid port pool number
ERROR: Context creation not completed successfully.

    CloneContext Log Location

    $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log
# Or typically:
/u01/APPLTOP/EBSapps/comn/clone/bin/CloneContext_<timestamp>.log

    Root Cause

    Port pool 0 is already in use by another EBS instance running on the same server, causing CloneContext validation to fail. Port pool 0 generally corresponds to the initially configured EBS port set for that environment. Actual port assignments vary by release, topology, and previous configuration changes.

    ps -ef | grep weblogic | grep AdminServer
lsof -i -P | grep LISTEN | grep <applcrp_user>

    Diagnosis

    Check which port pools are already in use:

    grep -i "port_pool" $CONTEXT_FILE
cat $INST_TOP/admin/out/portpool.lst

    You can also use the following utility to validate port availability before proceeding with the clone:

    perl $AD_TOP/bin/txkValidateRollupPorts.pl \
-contextfile=$CONTEXT_FILE \
-outfile=/tmp/portcheck.txt

    Review /tmp/portcheck.txt for any conflicts before selecting a port pool number.

    Fix

    Target System Port Pool [0-99] : 1
Checking the port pool 1
done: Port Pool 1 is free

    Important Note: Using a different port pool causes Oracle EBS to calculate a different set of ports based on the predefined port pool mapping. The resulting ports are not always a simple increment of one and should be verified in the generated context file after CloneContext completes.

    Always verify the actual assigned ports:
    grep -i "port" $CONTEXT_FILE | grep -v "#"
    cat $INST_TOP/admin/out/portpool.lst

    If your firewall rules, load balancer, or application configuration depend on specific port numbers, plan accordingly or stop the existing instance first to free pool 0.

    Patch File System Port Pool

    For R12.2 dual file system architecture, the Patch file system must use a different port pool than the Run file system. AutoConfig and cloning utilities will calculate a separate set of ports accordingly.

    Patch file system should have different port pool than Run file system.
Target System Port Pool [0-99] : 2

    Issue 2: FRM-92101 — Forms Session Failed During Startup

    Symptom

    FRM-92101: There was a failure in the Forms Server during startup.
Please look into the web-server log file for details.
Java Exception:
oracle.forms.net.ConnectionException: Forms session <1> failed during startup:
no response from runtime process

    What This Means

    The WebLogic Forms server (forms_server1) is running, but the underlying Forms runtime process (frmweb) is failing to start. WebLogic receives no response from it, and the session times out. FRM-92101 can have more than one root cause — the key is to run frmweb directly to identify which one applies.

    Log Analysis

    tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    This indicates that the Forms managed server is up and processing requests, shifting the investigation toward the underlying frmweb runtime process.

    First Diagnostic Step: Run frmweb Directly

    The most effective way to identify the root cause is to source the 10.1.2 environment and run frmweb manually:

    . /u01/APPLTOP/EBSapps/10.1.2/EBS_appserver.env
$ORACLE_HOME/bin/frmweb

    The output will indicate which root cause applies:

    • Undefined symbol errors (e.g. Symbol nnftboot was referenced...not found) → Root Cause 1: Invalid ldflags symlink
    • Missing shared library error (e.g. libXm.so.2: cannot open shared object file) → Root Cause 2: Missing OpenMotif symlink

    Root Cause 1: Invalid ldflags Symlink

    Symptom

    Running frmweb directly produces undefined symbol errors:

    rtld: 0712-001 Symbol nnftboot was referenced from module frmweb(), but a runtime definition of the symbol was not found.
rtld: 0712-001 Symbol ldap_search_s was referenced from module frmweb(), but a runtime definition of the symbol was not found.

    Root Cause

    Investigation revealed that the ldflags symbolic link under $ORACLE_HOME/lib32 was missing or pointing to an incorrect location, causing the Forms executable to fail during startup. This is a known issue after fresh installations or clones and is referenced in MOS Doc ID 454427.1.

    Verify ldflags

    ls -la $ORACLE_HOME/lib32/ldflags

    If the symlink is missing or pointing to a non-existent path, it needs to be corrected to point to $ORACLE_HOME/lib/ldflags.

    Fix

    After correcting the ldflags symbolic link and relinking the Forms components, frmweb started successfully. The relinking is performed using the Forms makefile under the 10.1.2 Oracle Home. Refer to MOS Doc ID 454427.1 for the complete steps applicable to your environment.

    Root Cause 2: Missing OpenMotif Library Symlink

    Symptom

    Running frmweb directly produces a missing shared library error:

    error while loading shared libraries:
libXm.so.2: cannot open shared object file: No such file or directory

    Root Cause

    The Oracle Forms runtime binary frmweb is compiled against libXm.so.2 (OpenMotif 2.x). On newer Linux versions (OL7/OL8/RHEL7/RHEL8), the installed OpenMotif package provides libXm.so.4, but the compatibility symlink libXm.so.2 → libXm.so.4 is missing.

    Verify with ldd

    ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Before fix:
libXm.so.2 => not found
# After fix:
libXm.so.2 => /usr/lib/libXm.so.4 (0x...)

    Option A: Create the Missing Symlink (Quick Fix)

    📌 Note on 32-bit vs 64-bit Libraries: On some Linux platforms the required library may reside under /usr/lib64 rather than /usr/lib. Always verify the actual location before creating symbolic links.

    # Step 1: Find the actual location of libXm.so.4
ls -l /usr/lib/libXm.so*
ls -l /usr/lib64/libXm.so*
# Step 2: Create the symlink (run as root)
# If library is in /usr/lib:
ln -s /usr/lib/libXm.so.4 /usr/lib/libXm.so.2
# If library is in /usr/lib64:
ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.2
# Step 3: Verify
ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Expected: libXm.so.2 => /usr/lib/libXm.so.4 (or /usr/lib64/...)

    Option B: Install openmotif21 (Recommended Clean Fix)

    # Step 1: Check current state
rpm -qa | grep motif
# Step 2: Enable ol7_addons repo (as root)
yum-config-manager --enable ol7_addons
# Step 3: Remove lesstif if present
yum remove lesstif
# Step 4: Install openmotif21
yum install --setopt=obsoletes=0 openmotif21

    Post-Fix: Bounce EBS Services and Validate

    cd $ADMIN_SCRIPTS_HOME
./adstpall.sh apps/<apps_password>
./adstrtal.sh apps/<apps_password>
# Or start Forms server specifically:
./admanagedsrvctl.sh start forms_server1

    After starting the services, verify that frmweb runtime processes are spawning successfully:

    ps -ef | grep frmweb
ps -ef | grep forms

    Then launch a Forms-based responsibility or function to confirm successful startup.

    Conclusion

    Both RC-50221 and FRM-92101 are common issues that can delay Oracle EBS R12.2 cloning activities. While the error messages may initially appear unrelated, a structured troubleshooting approach focusing on log analysis, port validation, and runtime dependency checks can quickly identify the root cause and reduce downtime.

    Summary

    IssueRoot CauseFix
    RC-50221: Port Pool Not FreeAnother EBS instance occupying port pool 0Use port pool 1 (or stop existing instance first)
    FRM-92101: ldflags issueldflags symlink missing or pointing to wrong locationFix ldflags symlink and relink Forms components (MOS Doc ID 454427.1)
    FRM-92101: OpenMotif issuelibXm.so.2 symlink missing after cloneln -s /usr/lib[64]/libXm.so.4 /usr/lib[64]/libXm.so.2 or install openmotif21

    Key Diagnostic Commands Cheat Sheet

    # Check which port pools are in use
grep -i "port_pool" $CONTEXT_FILE
cat $INST_TOP/admin/out/portpool.lst
# Validate port availability
perl $AD_TOP/bin/txkValidateRollupPorts.pl -contextfile=$CONTEXT_FILE -outfile=/tmp/portcheck.txt
# Check running WebLogic processes
ps -ef | grep weblogic | grep -v grep
# Run frmweb manually to see actual error
$ORACLE_HOME/bin/frmweb
# Check frmweb library dependencies
ldd $ORACLE_HOME/bin/frmweb | grep Xm
# Check ldflags symlink
ls -la $ORACLE_HOME/lib32/ldflags
# Check OpenMotif installation
rpm -qa | grep motif
# Check Forms runtime processes
ps -ef | grep frmweb
# Check Forms server log
tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    Lessons Learned

    • FRM-92101 is often a symptom, not the root cause. The WebLogic log may show the Java exception, but the actual failure is in the native frmweb binary. Always run frmweb directly to see the real error.
    • FRM-92101 has multiple root causes. Check the actual error output from frmweb to distinguish between an ldflags relinking issue and a missing OpenMotif library symlink.
    • Use ldd to diagnose missing shared library dependencies. It quickly identifies unresolved libraries without having to dig through multiple log files.
    • Verify port pool availability before starting a clone. Stop existing instances or choose a different pool to avoid RC-50221 errors.
    • Always validate the resulting ports in the generated context file after CloneContext completes. Port pool mapping is predefined and the assigned ports should be confirmed before proceeding.
    • Newer Linux builds may include libXm.so.4 but not the compatibility symlink required by Oracle Forms. OL7/OL8 installations do not always create libXm.so.2 automatically.
    • Document port assignments after every clone to avoid firewall and load balancer configuration issues down the line.

    Keywords

    • Oracle EBS R12.2 Clone
    • Oracle EBS R12.2 Forms Error
    • RC-50221 Port Pool Not Free
    • RC-50221 CloneContext Failed
    • FRM-92101 Forms Session Failed During Startup
    • FRM-92101 No Response From Runtime Process
    • ldflags symlink Oracle Forms
    • libXm.so.2 not found
    • Oracle Forms Runtime Process
    • adcfgclone.pl appsTier
    • adcfgclone.pl appsTier Error
    • Oracle EBS Clone Troubleshooting
    • Oracle Linux 7 OpenMotif Issue
    • OpenMotif
    • frmweb
    • CloneContext

    References

    • MOS Doc ID 454427.1 — FRM-92101: There was a failure in the Forms Server during startup
    • Oracle EBS R12.2 Cloning Guide

    Related Technologies

    • Oracle E-Business Suite R12.2
    • Oracle Forms
    • Oracle WebLogic Server
    • Oracle Linux 7
    • OpenMotif
    • AutoConfig
    • Rapid Clone

    Syed Anwar Ahmed
    Oracle ACE Associate | Oracle Apps DBA
    Sharing real-world Oracle EBS administration, cloning, patching, and troubleshooting experiences.
    Blog: syedanwarahmedoracle.blog