Category: EBS Administration

Oracle E-Business Suite administration, WebLogic and application tier guides

  • Oracle EBS Workflow Mailer After an SMTP Migration: Why Mail Still Used the Old Relay

    Hostnames, IP addresses, usernames, email addresses, and notification IDs in this post are masked with # characters.

    Alhamdulillah, this investigation started with a simple question: after changing the SMTP server in Oracle E-Business Suite, why was mail still going through the old relay?

    It turned up three separate findings. The EBS Workflow Mailer configuration pointed to the new SMTP server. OS-level Sendmail still used the old relay. Some failed Workflow notifications were associated with a disabled recipient preference, and the reason for that preference is not yet established.

    This post documents the checks, the configuration change, and the evidence collected during validation. It is an interim investigation: the checks still open are listed in section 9.

    Environment

    • Oracle E-Business Suite R12.2
    • Oracle Database 11.2.0.4
    • Linux 7 application tier
    • Sendmail 8.14.7
    • systemd service management

    1. The initial problem

    The Workflow Mailer’s outbound SMTP server had been changed to the new server (####################). Network connectivity to the new server was successful, but the logs showed messages being relayed through the old relay (####################) before the Sendmail configuration change.

    At the same time, two notifications for one user showed MAIL_STATUS = FAILED.

    The first step was to verify the EBS configuration, then trace the actual mail path through the application and operating-system logs.

    2. Finding 1: EBS configuration and OS-level Sendmail did not match

    Verify the Workflow Mailer configuration

    SET LINESIZE 200
    COL parameter_name FORMAT A20
    COL parameter_value FORMAT A30
    COL last_update_date FORMAT A22
    SELECT RPAD(p.parameter_name,20) parameter_name,
    RPAD(NVL(v.parameter_value,'-'),30) parameter_value,
    TO_CHAR(v.last_update_date,'DD-MON-YY HH24:MI:SS')
    last_update_date
    FROM fnd_svc_comp_param_vals v,
    fnd_svc_comp_params_b p,
    fnd_svc_components c
    WHERE v.component_id = c.component_id
    AND v.parameter_id = p.parameter_id
    AND c.component_name = 'Workflow Notification Mailer'
    AND p.parameter_name = 'OUTBOUND_SERVER';

    OUTBOUND_SERVER showed the new SMTP server. The Sendmail log showed a different route:

    sendmail[####]: ##############:
    from=<wfmailer@#######################>,
    msgid=<########.####.#############.JavaMail.wfmailer@#######################>,
    proto=ESMTP, daemon=MTA,
    relay=####################### [###.###.###.###]
    sendmail[####]: ##############:
    to=<##################>, mailer=relay,
    relay=#################### [###.###.###.###],
    dsn=2.0.0, stat=Sent

    The logs confirmed that messages submitted to local Sendmail were forwarded through the old smart host. The sender and the JavaMail.wfmailer message ID point to messages generated by the Workflow Mailer. Oracle’s Workflow documentation describes the Mailer sending notification messages by SMTP to the SMTP server configured for it (see Outbound Notification Mailer Processing, from an older Workflow release), and notes that a UNIX host running Sendmail can serve as that SMTP server (see the Administrator’s Guide). The Mailer’s previous OUTBOUND_SERVER value was not captured, so the logs do not show why it handed these messages to local Sendmail. Further tracing was needed to establish the route used by the affected Workflow notifications.

    Check the Workflow Mailer process start time

    SET LINESIZE 200
    COL queue_name FORMAT A20
    COL status FORMAT A10
    COL os_pid FORMAT A10
    COL started FORMAT A22
    SELECT RPAD(q.concurrent_queue_name,20) queue_name,
    RPAD(NVL(p.process_status_code,'-'),10) status,
    RPAD(NVL(TO_CHAR(p.os_process_id),'-'),10) os_pid,
    TO_CHAR(p.process_start_date,'DD-MON-YY HH24:MI:SS')
    started
    FROM fnd_concurrent_processes p,
    fnd_concurrent_queues q
    WHERE p.concurrent_queue_id = q.concurrent_queue_id
    AND p.queue_application_id = q.application_id
    AND q.concurrent_queue_name = 'WFMLRSVC'
    AND p.process_status_code = 'A';

    The process had started 11 days before the parameter change. An OS-level check with ps -o pid,lstart,cmd -p <os_pid> showed the same start date.

    That raised the possibility that the Workflow Mailer had not been restarted after the change. The previous parameter value was not captured, though, so the start time alone could not show which SMTP server the running process had been using.

    Lesson: After changing Workflow Mailer parameters, check the active service process and follow the real message route in the logs. The database parameter alone doesn’t prove which server handled a message.

    Query note: last_update_date on the fnd_svc_components row is not the time a parameter changed. Use last_update_date on fnd_svc_comp_param_vals, as in the query above.

    3. Finding 2: Some failed notifications were associated with a disabled recipient preference

    Two notifications remained OPEN with MAIL_STATUS = FAILED. The recipient’s notification preference was DISABLED.

    SET LINESIZE 220
    COL recipient_role FORMAT A20
    COL mail_status FORMAT A12
    COL status FORMAT A10
    COL message_name FORMAT A30
    COL email_address FORMAT A30
    COL notif_pref FORMAT A12
    SELECT n.notification_id,
    RPAD(n.recipient_role,20) recipient_role,
    RPAD(NVL(n.mail_status,'-'),12) mail_status,
    RPAD(n.status,10) status,
    RPAD(NVL(n.message_name,'-'),30) message_name,
    RPAD(NVL(r.email_address,'-'),30) email_address,
    RPAD(NVL(r.notification_preference,'-'),12) notif_pref
    FROM wf_notifications n,
    wf_roles r
    WHERE n.notification_id IN (#######,#######)
    AND r.name = n.recipient_role;

    Example output:

    NOTIFICATION_ID RECIPIENT_ROLE MAIL_STATUS STATUS MESSAGE_NAME EMAIL_ADDRESS NOTIF_PREF
    --------------- --------------- ------------ ------- ----------------------- -------------------- ----------
    ####### ###### FAILED OPEN PO_PO_HAS_BEEN_APPROVE ################## DISABLED
    ####### ###### FAILED OPEN PO_REQ_APPROVED ################## DISABLED

    The preference was also found in WF_LOCAL_ROLES and traced to the Workflow MAILTYPE preference in FND_USER_PREFERENCES. Both notifications remained in the user’s Worklist.

    A DISABLED preference does not by itself explain a FAILED status. Oracle documents that when every address for a recipient is invalid and delivery fails, the Mailer sets the notification’s mail status to FAILED, sets the recipient’s preference to DISABLED, and notifies SYSADMIN (see the Oracle EBS technology blog). So DISABLED can be the result of an earlier delivery failure, not only a choice the user made.

    I did not check when or why this preference was set, or whether the failures began before it was set. I also have not yet confirmed whether these two messages ever reached Sendmail. Both checks are listed in section 9.

    4. Finding 3: Historical failures need to be separated from the migration issue

    Before attributing every failure to the migration, I collected a baseline of notification statuses.

    SET LINESIZE 200
    COL mail_status FORMAT A12
    COL cnt FORMAT 99999999
    SELECT RPAD(NVL(mail_status,'-'),12) mail_status,
    COUNT(*) cnt
    FROM wf_notifications
    WHERE mail_status IN ('MAIL','FAILED','ERROR')
    GROUP BY mail_status;
    Mail statusCountObservation
    MAIL0No records in this status
    FAILED156115 associated with disabled preferences, 39 with missing recipient roles, 2 canceled
    ERROR2,4252,255 created on one day for a single administrator role; the rest were older

    The dates showed that these records were not all from the migration period. The counts show association, not cause: a disabled preference can be a cause or a consequence of a failure. Historical records, missing roles, disabled preferences and SMTP delivery issues need separate investigation.

    5. Changing the Sendmail smart host

    The change was limited to the Sendmail configuration. These commands illustrate the procedure used here. Run them only after confirming the approved SMTP target, any maintenance requirements and the rollback plan.

    Pre-change checks

    date
    systemctl status sendmail --no-pager
    mailq
    grep -ni "SMART_HOST" /etc/mail/sendmail.mc
    grep -ni "^DS" /etc/mail/sendmail.cf
    nc -vz -w 5 #################### 25

    Back up the configuration

    cp -p /etc/mail/sendmail.mc \
    /etc/mail/sendmail.mc.bak_$(date +%Y%m%d_%H%M%S)
    cp -p /etc/mail/sendmail.cf \
    /etc/mail/sendmail.cf.bak_$(date +%Y%m%d_%H%M%S)

    Update the smart host

    Edit /etc/mail/sendmail.mc and change only the active SMART_HOST definition:

    define(`SMART_HOST', `####################')dnl

    Regenerate the configuration and verify:

    cd /etc/mail
    make
    grep -ni "^DS" /etc/mail/sendmail.cf

    Expected output:

    101:DS####################

    Restart Sendmail:

    systemctl restart sendmail
    systemctl status sendmail --no-pager

    The service returned to active (running) and was accepting connections.

    The change did not touch /etc/hosts, /etc/mail/access or /etc/hosts.allow. Those files serve other purposes and shouldn’t be edited just to change the smart host, unless a separate requirement justifies it.

    Rollback: restore sendmail.mc from its backup, run make, confirm the DS line, and restart Sendmail.

    6. Verify delivery through the new SMTP server

    A controlled test mail was sent from the application server to an approved test mailbox.

    echo "Test email from application server after SMTP migration" \
    | mail -s "SMTP Migration Test" ####################

    The relevant log entries:

    STARTTLS=client, relay=####################.,
    verify=FAIL, cipher=ECDHE-RSA-AES256-GCM-SHA384
    to=<####################>, mailer=relay,
    relay=####################. [###.###.###.###],
    dsn=2.0.0, stat=Sent

    ما شاء الله, the test message went through the new relay on the first attempt.

    What does stat=Sent mean? The next SMTP server accepted the message for delivery. It does not confirm arrival in the recipient’s inbox. Check the mailbox, and use the SMTP server’s message trace if delivery fails later.

    What does verify=FAIL mean? Sendmail did not verify the server certificate under its current settings, even though the SMTP transaction succeeded. Successful delivery is not proof that certificate validation is healthy, so review it separately with the mail or security team.

    7. What the investigation proved

    Confirmed by configuration and logs:

    • The EBS Workflow Mailer was configured with the new SMTP server.
    • OS-level Sendmail still had the old smart host before the change, and messages submitted to it were forwarded through that host.
    • The Sendmail configuration was updated, regenerated, and restarted successfully.
    • A controlled test message was relayed through the new SMTP server and accepted for delivery.
    • The two example notifications had MAIL_STATUS = FAILED, remained OPEN, and had a recipient preference of DISABLED.

    Not established by these checks:

    • Which SMTP server the running Workflow Mailer process was using internally before it was restarted, and why it handed messages to local Sendmail.
    • Whether the two failed notifications ever reached Sendmail.
    • When and why the recipient’s preference became DISABLED, and whether the failures began before it.
    • The cause of every historical FAILED or ERROR notification.
    • Whether the original affected notifications were delivered after the Sendmail change.
    • Whether the recipient received the test email.
    • The cause of the STARTTLS certificate-verification warning.

    A successful test email validates one segment of the mail path. The original Workflow notification still needs its own test.

    8. Practical lessons

    1. Check both EBS and OS-level SMTP settings. The Workflow Mailer parameter and the Sendmail smart host can differ.
    2. Compare the parameter change time with the process start time. Restart the Workflow Mailer component when the change procedure requires it, and confirm the new start time.
    3. Trace a real message end to end. Use the notification record, the Mailer logs, the Sendmail logs and recipient confirmation.
    4. Don’t assume a failed notification is either an SMTP problem or a preference problem. A disabled preference can be a cause or a consequence of a delivery failure, so check when it was set, along with the recipient role, email address and notification status.
    5. Capture a baseline before changing anything. Old FAILED and ERROR records can have unrelated causes.
    6. Verify the handoff and the final delivery separately. stat=Sent is evidence of SMTP acceptance, not of inbox delivery.
    7. Review certificate verification on its own. Successful delivery doesn’t remove the need to investigate TLS warnings.

    9. Status and remaining checks

    The Sendmail change is complete and the test message was accepted by the new relay. These checks are still open:

    1. Failed notifications: search the Sendmail log for the affected recipient to establish whether the two failed messages ever reached Sendmail.
    2. Preference history: establish when and why the recipient’s MAILTYPE preference became DISABLED (who last updated it, and whether the first FAILED notification came before or after), and whether SYSADMIN received an invalid-address notice.
    3. Workflow Mailer process: verify the active WFMLRSVC process against the parameter change time, and restart the component under the approved change procedure if needed.
    4. End-to-end Workflow test: submit a test notification, then check the Mailer log and the Sendmail log to confirm it was relayed through the new server.
    5. Certificate verification: investigate verify=FAIL separately with the mail or security team.

    Until those checks are done, this post is an interim investigation. It does not claim that the original Workflow notifications have been fixed.

    Conclusion

    Changing the SMTP server in Oracle EBS may not be the whole job. OS-level Sendmail has its own smart-host setting, and it needs checking when the mail logs show messages leaving through an unexpected relay.

    The Sendmail test confirmed that the new relay accepts mail from the application server. The checks in section 9 will show whether the Workflow Mailer path itself works end to end, and why the affected notifications failed.

    Disclaimer: This article is based on a troubleshooting scenario. All hostnames, IP addresses, usernames, email addresses and notification IDs are masked. Validate all commands and configuration changes against your own environment and change-management procedures before using them in production.

  • Oracle EBS R12.2 DualFS RapidClone: FMW Prerequisite Failures, RPMDB Corruption and Stale Inventory

    Alhamdulillah, this post documents the issues hit during an EBS R12.2 QA application-tier refresh and how each was isolated. Server names, paths and identifiers are generic examples.

    Environment (generic)

    ComponentExample
    App serverqa-app-server01
    DB serverqa-db-server01
    EBS environmentQA_EBS
    Application base/u01/apps
    Pairs file/u01/cloning/pairs/qa_apps_pairs.txt
    Central inventory/opt/oraInventory

    Background

    A DualFS clone recreates both fs1 and fs2. Per the cloning procedure used here, the old fs2, inst and FMW_Home were moved aside and removed first:

    cd /u01/apps/
    mv fs2 fs2_old
    rm -rf fs2_old &
    
    cd fs1
    mv inst inst_old
    mv FMW_Home FMW_Home_old
    rm -rf inst_old FMW_Home_old &

    Cleanup steps should follow your own environment’s cloning document.

    Symptom

    adcfgclone.pl failed at the FMW prerequisite stage:

    ERROR: Failed to execute FMW pre-requisite check

    Re-running would not have helped. The prerequisite check reads OS state, so that is where the investigation started.

    cat /etc/os-release
    rpm --verifydb
    rpm -qa | grep -E 'compat-libcap1|compat-libstdc'

    Issue 1: RPM database corruption

    On one app server, rpm --verifydb returned Berkeley DB errors:

    BDB0682 Page ... bad prev_pgno
    BDB0683 Page ... overflow item incomplete
    Packages: BDB0090 DB_VERIFY_BAD

    Package queries also showed rpmdbNextIterator: skipping h# ... and region trailer: BAD.

    RPMDB corruption was identified and the package state was subsequently validated. If the RPM database is unhealthy, package-based prerequisite results can be incomplete or misleading. Running rpm --verifydb early is a useful way to identify this condition before spending time troubleshooting the Oracle prerequisite scripts.

    Issue 2: Missing compatibility packages

    Two packages were missing: compat-libcap1 and compat-libstdc++-33. After confirming repository availability, both were installed with dnf install, then the RPM database was verified again.

    A similar QA server passed the check without this problem, so package state can differ between servers that look identical.

    Issue 3: refhost.xml OL7 block on an OL8 target

    The target was Oracle Linux 8, but the first OS block in refhost.xml was defined for Oracle Linux 7:

    /u01/apps/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/prereq/linux64/refhost.xml
    <VERSION VALUE="7"/>

    Only that block was changed to VALUE="8". A global sed replacement is risky because the file contains multiple OS definitions. Back up the file, identify the applicable block, change only that, and validate the XML.

    Issue 4: Stale Oracle Central Inventory

    Removing FMW_Home from disk does not remove it from the central inventory. Entries such as these were still registered after the directories were gone:

    /u01/apps/fs1/FMW_Home/oracle_common
    /u01/apps/fs1/FMW_Home/webtier
    /u01/apps/fs1/FMW_Home/Oracle_EBS-app1

    Documented cloning step. The cloning document’s inventory cleanup applies to the target database server (check /etc/oraInst.loc, remove the DB home entry from ContentsXML/inventory.xml). It does not cover resetting the application-server inventory, so the app-server reset described here is not part of the standard cloning steps.

    Before any inventory change: confirm the active inventory via /etc/oraInst.loc, preserve a copy, then apply the approved procedure. The existing inventory was kept as a backup. Note that the first mv below is conditional: run it only if /opt/oraInventory_old already exists.

    # Only if /opt/oraInventory_old already exists:
    mv /opt/oraInventory_old /opt/oraInventory_old_backup
    
    mv /opt/oraInventory /opt/oraInventory_old
    mkdir /opt/oraInventory
    chown oracle:oinstall /opt/oraInventory

    Issue 5: EXECUTE_FMW_SYS_PREREQS

    The cloning procedure sets this variable before the DualFS run:

    export EXECUTE_FMW_SYS_PREREQS=false
    echo $EXECUTE_FMW_SYS_PREREQS

    This skips the FMW OS prerequisite execution. It was set here only after the failures above had been investigated, and only because the procedure specifies it. It is not a general substitute for fixing prerequisite failures.

    Issue 6: SSH disconnect during a multi-hour clone

    RapidClone can run for hours, and the procedure calls for a screen session (or VDI). During this exercise the SSH session dropped mid-run.

    screen -S rapidclone
    cd /u01/apps/fs1/EBSapps/comn/clone/bin
    export EXECUTE_FMW_SYS_PREREQS=false
    perl adcfgclone.pl component=appsTier \
    pairsfile=/u01/cloning/pairs/qa_apps_pairs.txt \
    dualfs=yes

    After the disconnect, check before restarting anything:

    screen -ls
    screen -r <session>

    If the original process is still running, a second adcfgclone.pl would only add complications.

    Result

    After the identified OS, package, prerequisite and inventory issues were addressed, the DualFS clone completed and created both fs1 and fs2.

    ما شاء الله

    The same approach worked on the second QA environment. This establishes that the DualFS RapidClone completed, not that the entire QA refresh is finished: post-clone configuration is separate, and the procedure says not to start services at the end of the clone until the required post-clone steps are completed.

    Summary

    FindingLayerAction
    FMW prerequisite failureOS / packagesInvestigated underlying checks
    RPMDB corruptionOSIdentified the corruption and validated the package state after remediation
    Missing compat RPMsOSInstalled required packages
    OL7 block on OL8 targetPrerequisite configCorrected the applicable block only
    Stale FMW homes in inventoryOracle InventoryBacked up the existing inventory, then applied the approved remediation
    Long-running cloneSessionRan inside screen

    Suggested pre-clone checklist

    cat /etc/os-release
    rpm --verifydb
    rpm -qa | grep -E 'compat-libcap1|compat-libstdc'
    cat /etc/oraInst.loc
    ls -ld /u01/apps/fs1 /u01/apps/fs2
    ls -l /u01/apps/fs1/EBSapps/comn/clone/bin/adcfgclone.pl

    Post-clone

    RapidClone success is not the end of the refresh. The cloning procedure continues with custom environment configuration, AutoConfig, WebLogic configuration, Concurrent Manager/MWA node configuration, shared filesystem setup and further AutoConfig runs, followed by application validation.

    Takeaway

    RapidClone is often where infrastructure problems become visible, not where they start. When it fails, check the layers underneath before rerunning:

    OS → RPM database → packages → prerequisite config → Oracle Inventory → filesystem → RapidClone → post-clone configuration

    Examples use generic names. No real server names, IP addresses, domains, company names or screenshots are included.

  • Oracle EBS R12.2 AutoConfig Template Version Conflict After Patching: How to Preserve Customizations

    Alhamdulillah, sharing another real troubleshooting session — this one from an Oracle E-Business Suite R12.2 multi-node environment. After applying a patch, AutoConfig aborted on the primary application node with a template version conflict. This post walks through the exact error, how I identified which template entries were genuine customizations versus Oracle-shipped content, and the correct way to rebase a custom template so AutoConfig completes cleanly.

    Environment

    • Oracle E-Business Suite R12.2.13, AD/TXK Delta 15
    • Two application nodes: appsnode1 (primary/admin) and webnode1 (secondary web node)
    • Database: Oracle 19c on dbnode1
    • Third-party customization: More4Apps servlet registered in oacore web.xml

    The Error

    After patching, running AutoConfig on the run file system failed immediately:

    [applmgr@appsnode1 scripts]$ sh adautocfg.sh
    Enter the APPS user password:
    The log file for this session is located at: .../admin/log/<timestamp>/adconfig.log
    wlsDomainName: EBS_domain
    WLS Domain Name is VALID.
    AutoConfig is configuring the Applications environment...
    AutoConfig will consider the custom templates if present.
    Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

    The log pinpointed the conflicting template:

    [ FND_TOP ]
    
    [ VERSION CONFLICTS INFORMATION ]
    Template shipped by oracle is having a version different than the template lying in custom directory.
    Template shipped by Oracle : $FND_TOP/admin/template/oacore_web_xml_FMW.tmp (version: 120.17.12020000.33)
    Custom template            : $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp (version: 120.17.12020000.31)
    Please resolve the differences between the two templates or refer to the Oracle E-Business Suite Setup Guide for further details.

    What happened is straightforward: the patch delivered a newer version of oacore_web_xml_FMW.tmp (120.17.12020000.33), but a custom copy of the older version (120.17.12020000.31) exists under $FND_TOP/admin/template/custom. AutoConfig detects the mismatch and refuses to run, because blindly using the stale custom template would silently discard whatever Oracle changed in the new version.

    Root Cause Analysis: What Is Actually a Customization?

    This is the critical analysis step. Before touching anything, you need to answer one question precisely: which lines in the custom template are genuine customizations, and which are just old Oracle-shipped content?

    First, take a backup and diff the new shipped template against the custom one:

    cd $FND_TOP/admin/template
    
    cp -p custom/oacore_web_xml_FMW.tmp \
          custom/oacore_web_xml_FMW.tmp.pre_patch_25Jul2026
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp

    The diff showed three differences beyond the header:

    1. A m4aServlet servlet and servlet-mapping (More4Apps) present only in the custom template
    2. A SyncServlet servlet and servlet-mapping present only in the custom template
    3. A RequestAuditReport (QP) servlet present only in the new Oracle template

    At first glance it looks like there are two customizations to carry forward: m4aServlet and SyncServlet. This is where a wrong assumption would corrupt the template. To verify, I compared the old shipped template against the new shipped template. In an R12.2 dual file system this is easy — the other file system still had the previous patch level, so both shipped versions were available:

    diff -u \
      <old_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp \
      <new_fs>/EBSapps/appl/fnd/12.0.0/admin/template/oacore_web_xml_FMW.tmp

    Key excerpt from the shipped-vs-shipped diff:

    -   <servlet>
    -     <servlet-name>SyncServlet</servlet-name>
    -     <servlet-class>oracle.apps.jtf.cac.sync.transport.SyncServlet</servlet-class>
    -  </servlet>
    ...
    +     <servlet>
    +         <servlet-name>RequestAuditReport</servlet-name>
    +         <servlet-class>oracle.apps.qp.servlet.RequestAuditReport</servlet-class>
    +    </servlet>

    This settled the question:

    • SyncServlet was Oracle-shipped content in version .31 and is no longer present in the .33 shipped template. It was never a customer customization, so it should NOT be carried forward.
    • RequestAuditReport is new Oracle-shipped content in .33 — it must be retained.
    • m4aServlet was not present in either shipped template version compared — it exists only in the custom directory, which identifies it as the genuine site-specific customization.

    If I had merged the old custom template on top of the new one, I would have reintroduced a servlet that is no longer present in Oracle’s newer shipped template. This is why “just copy your custom entries into the new template” advice found on many forums is dangerous — verify against the shipped versions first.

    The Fix: Rebase the Custom Template

    The correct approach: replace the custom template with the new shipped version, then re-apply only the genuine customization.

    cd $FND_TOP/admin/template
    
    # Rebase custom template on the new shipped version
    cp -p oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    # Re-insert the only genuine customization (m4aServlet)
    sed -i '/<!-- FND Servlets -->/a\
    \
      <servlet>\
        <servlet-name>m4aServlet</servlet-name>\
        <servlet-class>com.more4apps.r12.servlet.XmlServlet</servlet-class>\
      </servlet>\
    \
      <servlet-mapping>\
        <servlet-name>m4aServlet</servlet-name>\
        <url-pattern>/m4aServlet/*</url-pattern>\
      </servlet-mapping>\
    ' custom/oacore_web_xml_FMW.tmp

    This matches the procedure in the Oracle E-Business Suite Setup Guide: when a patch delivers a newer version of a template you have customized, copy the new Oracle template into the custom directory and reapply the still-required customizations to that copy — never the other way around.

    Verify before rerunning AutoConfig — the headers must match, and the only diff must be the customization:

    grep '\$Header' oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    
    oacore_web_xml_FMW.tmp:        version 120.17.12020000.33
    custom/oacore_web_xml_FMW.tmp: version 120.17.12020000.33
    
    diff -u oacore_web_xml_FMW.tmp custom/oacore_web_xml_FMW.tmp
    --- only the m4aServlet servlet + mapping block should appear ---

    As an additional best-practice validation, Oracle recommends running the AutoConfig configuration-check utility to preview the impact of template changes before the actual AutoConfig run:

    $AD_TOP/bin/adchkcfg.sh contextfile=$CONTEXT_FILE

    Rerun AutoConfig:

    cd $ADMIN_SCRIPTS_HOME
    sh adautocfg.sh
    
    ...
    Configuring templates from all of the product tops...
            Configuring AD_TOP........COMPLETED
            Configuring FND_TOP.......COMPLETED
            ...
    AutoConfig completed successfully.

    A Bonus Finding on the Second Node

    While repeating the check on the secondary web node, the header comparison came back clean — custom template already at .33 — but something was off:

    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp
    (no output)
    
    grep -n 'm4aServlet' $FND_TOP/admin/template/custom/oacore_web_xml_FMW.tmp_ORIG
    61:    <servlet-name>m4aServlet</servlet-name>

    The current custom template matched the newer shipped version, but the m4aServlet customization was missing. Interestingly, the _ORIG backup still contained the customization. This indicated that during an earlier template update, the customization was not carried forward. AutoConfig had been running fine and web.xml was being generated — just without the third-party servlet.

    I re-inserted the m4aServlet block using the same sed command, verified the diff, and ran AutoConfig on that node as well:

    AutoConfig completed successfully.

    This is the real lesson of the post. A template version conflict at least fails loudly. A customization that is accidentally omitted during a template rebase may not — AutoConfig can complete successfully while the generated configuration no longer contains the required customization, and you find out later when the third-party tool stops working after a bounce.

    An Additional Message Observed

    The AutoConfig log in this run also contained:

    ECC not enabled, setting FND_ECC_ENABLED to FALSE
    [ FND_ECC_ENABLED ]
    INFO : Error updating/creating profile option value.

    In this case Enterprise Command Center is not configured, AutoConfig continued past this message, and the run ultimately exited with status 0. Always evaluate such messages in the context of your own environment rather than assuming every INFO : Error entry can be ignored.

    Post-Fix Checklist

    1. Confirm the customization landed in the generated web.xml (get the target path from the adconfig log and grep for your servlet).
    2. Bounce the application services (at least the oacore managed servers) so the new web.xml is deployed.
    3. Test the customization end to end — for More4Apps, hit the servlet URL from the wizard.
    4. If your patching cycle requires it, regenerate appsutil.zip (perl $AD_TOP/bin/admkappsutil.pl) and refresh the database tier.
    5. Repeat the custom template verification on every application node — as seen above, nodes can drift.

    Key Takeaways

    • Never resolve this error by copying the old custom template over the new shipped one, and never blindly merge either. The Oracle-documented procedure is to copy the new shipped template into the custom directory and re-apply only verified customizations to that copy.
    • Use the dual file system to your advantage: diff old-shipped vs new-shipped to separate Oracle’s changes from your customizations.
    • After any template rebase, diff the shipped and custom templates — the output should contain nothing but your customizations.
    • Audit custom templates on all nodes periodically. A missing customization may not trigger an AutoConfig error, allowing the generated configuration to differ from what you expect.

    I hope this helps someone facing the same conflict. Feel free to leave a comment if you have questions.


    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • EBS 12.2 Rapid Clone dbTechStack Fails with RC-50013 — INS-08101 supportedOSCheck on Oracle Linux 8

    Alhamdulillah, sharing another real troubleshooting session — this one from an EBS 12.2 database tier clone on Oracle Linux 8.10, where adcfgclone.pl dbTechStack failed and the on-screen error pointed in completely the wrong direction.

    Environment

    Oracle EBS 12.2, Oracle Database 19c (19.21) multitenant (CDB/PDB), Oracle Linux 8.10, target server testserver01.

    The Symptom

    perl adcfgclone.pl dbTechStack /u01/EBSDB/19.0.0/appsutil/EBSDB_testserver01.xml
    
    RC-00110: Fatal: Error occurred while relinking of ApplyDBTechStack
    ERROR while running Apply...
    ERROR: Failed to execute /u01/EBSDB/19.0.0/appsutil/clone/bin/adclone.pl

    The first instinct with RC-00110 is to suspect a relink problem — missing OS packages, a corrupt backup, bad extraction. We verified all of that and ruled it out. The real lesson of this post: the RC error on screen is generic. Follow the log chain.

    The Log Chain

    Step 1 — The ApplyDBTechStack log showed the relink script actually passed:

    adlnkoh.sh completed sucessfully

    but home registration failed:

    Finished OUI CLI cloning for s_db_oh with return code: 1
    ouicli.pl  INSTE8_APPLY  1
    RC-50013: Fatal: Instantiate driver did not complete successfully.

    Step 2 — ohclone.log pointed one level deeper:

    OUI runinstaller log file - /u01/EBSDB/oraInventory/logs/InstallActions<timestamp>/installActions<timestamp>.log
    Found the INFO: Exit Status is -1 in runInstaller log.
    OUI CLI cloning returned non-zero.

    Note the timing: runInstaller started and exited within the same second. An installer that dies instantly is not failing a task — it is failing a pre-check.

    Step 3 — The installActions log had the true error:

    [WARNING] [INS-08101] Unexpected error while executing the action at state: 'supportedOSCheck'
    SUMMARY: - java.lang.NullPointerException

    Root Cause

    The 19c ORACLE_HOME delivered inside the EBS Rapid Clone stage carries the 19.3 base installer (year 2019). Its OS certification table ends at Oracle Linux 7. On OL8/RHEL8, the lookup for the distribution ID returns null and the installer crashes with a NullPointerException before doing any work — surfacing back up the chain as RC-50013 and RC-00110.

    The Fix

    Set CV_ASSUME_DISTID in the same shell session before rerunning the clone:

    export CV_ASSUME_DISTID=OEL7.8
    echo $CV_ASSUME_DISTID
    cd /u01/EBSDB/19.0.0/appsutil/clone/bin
    perl adcfgclone.pl dbTechStack /u01/EBSDB/19.0.0/appsutil/EBSDB_testserver01.xml

    Result — the rerun completed cleanly, confirmed by these lines in the new ApplyDBTechStack log:

    Finished OUI CLI cloning for s_db_oh with return code: 0
    Completed home registration for s_db_oh
    Completed Apply...
    ApplyDBTechStack Completed Successfully.

    Important: this variable does NOT change your OS or the installed software. It only tells the old installer which certification profile to use for its checks. The value must be one the 19.3-base installer recognizes — OEL7.8 is the value documented in Oracle’s Linux 8 release notes. Setting it to your actual OS version (8.10) defeats the purpose.

    Permanent Fix

    The workaround can also live inside the ORACLE_HOME, in $ORACLE_HOME/cv/admin/cvu_config:

    # Fallback to this distribution id
    CV_ASSUME_DISTID=OEL7.8      <-- uncomment and set

    This also explains a common confusion: “my previous clone worked without this!” If the earlier backup was taken from a home where cvu_config already had this set, the fix travelled inside the backup. A backup from an untouched home reintroduces the failure. Check with:

    grep -i CV_ASSUME_DISTID $ORACLE_HOME/cv/admin/cvu_config

    Post-Clone Notes

    After the fix, you will still see FATAL errors like “DB Connection failed” and “Invalid APPS database user credentials” from the AutoConfig phase, plus DB-ETCC connectivity warnings — these are expected at this stage because the database is not restored and opened yet. They resolve once you restore the database and rerun AutoConfig and ETCC.

    Also refresh ETCC from patch 17537119 — the bundled bugfix XML goes stale after 30 days, and older versions do not recognize newer Release Updates such as 19.21.

    Key Takeaways

    RC-00110/RC-50013 are wrappers, not root causes — always walk ApplyDBTechStack log → ohclone.log → installActions log. A runInstaller that exits in under a second failed a pre-check, not the work itself. And on OL8/RHEL8 with any 19.3-base home, CV_ASSUME_DISTID=OEL7.8 belongs in your clone runbook.


    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS 12.2.4 Upgrade to Oracle Database 19c: Oracle Restart Upgrade, Non-CDB to PDB Conversion, and Lessons Learned

    Alhamdulillah, this post brings together one of the more involved upgrade journeys I have worked through recently — taking an Oracle E-Business Suite 12.2 database stack from 12.1.0.2 all the way to 19c, on a single standalone host running Oracle Restart.

    What made this engagement worth writing up is that it was not a clean, textbook upgrade. The Grid Infrastructure layer presented several unexpected challenges — a chain of CLSRSC failures that had to be cleared before it would move, and once Grid was finally on 19c, the database upgrade and the non-CDB to PDB conversion had their own set of gotchas around ASM registration, password files, and UTL_FILE_DIR.

    I have combined both phases — the Grid Infrastructure upgrade and the database upgrade plus multitenant conversion — into one walkthrough, because in the real world they are not separate stories. The GI upgrade forms the foundation for every subsequent phase. If you only read GI upgrade blogs in isolation, you miss how the pieces connect.

    All hostnames, database names, ASM diskgroup names, and paths below are genericized. The technical substance is exactly as it played out.

    Published: June 2026

    Environment:
    Oracle EBS 12.2.4 | AD/TXK Delta 13 | Oracle Database 19c (19.30)
    (Versions shown are from the environment used in this engagement, not a minimum certified baseline — always confirm your own certified levels.)


    Executive Summary

    This upgrade involved moving an Oracle E-Business Suite 12.2 database environment from Oracle Database 12.1.0.2 to 19c on a standalone Oracle Restart host. The journey consisted of four major phases:

    • Oracle Restart (Grid Infrastructure) upgrade from 12.1.0.2 to 19c
    • Database upgrade from 12.1.0.2 to 19c using DBUA
    • Non-CDB to PDB conversion using Oracle EBS TXK utilities
    • EBS application-tier reconfiguration and validation

    The most time-consuming portions were not the upgrades themselves, but resolving legacy Oracle Restart configuration issues, rebuilding HAS components, handling ASM registration after the GI upgrade, addressing password-file requirements during the PDB conversion, and resolving UTL_FILE_DIR migration challenges.

    This guide covers an end-to-end Oracle EBS 12.2.4 upgrade to Oracle Database 19c, including Oracle Restart upgrade, DBUA upgrade, non-CDB to PDB conversion, AutoConfig remediation, and post-upgrade validation.


    Environment overview

    ComponentDetail
    PlatformSingle standalone Linux host (dbhost) running Oracle Restart / standalone HAS — not full RAC Clusterware
    Apps tierSeparate node (appshost) under the applmgr-style owner
    EBS release12.2.4
    AD levelAD.C.Delta.13
    TXK levelTXK.C.Delta.13
    Source GI12.1.0.2
    Target GI19c (19.30)
    Source DBNon-CDB EBS database, SID ebsdb, on 12.1.0.2
    Target DB19c (19.30), ebsdb plugged in as a lowercase PDB inside CDB EBSCDB
    StorageASM diskgroups +DATA and +RECO
    Old GI home/u01/app/12.1.0.2/grid
    New GI home/u01/app/19.0.0/grid
    Old DB home/u01/app/oracle/product/12.1.0.2/dbhome_1
    New DB home/u01/app/oracle/product/19.0.0/dbhome_1

    The overall sequence was four phases:

    1. Grid Infrastructure 12.1.0.2 → 19c (Oracle Restart upgrade)
    2. Database 12.1.0.2 → 19c via DBUA (-keepEvents)
    3. Non-CDB → PDB conversion (plug ebsdb into EBSCDB)
    4. EBS application-tier cutover (AutoConfig, adop)

    The transformation at a glance

            BEFORE                                AFTER
    
        EBS Apps Tier                         EBS Apps Tier
             |                                     |
    12.1.0.2 Non-CDB (ebsdb)        ===>      PDB ebsdb
             |                                     |
    Oracle Restart 12.1.0.2                  CDB EBSCDB (19c)
             |                                     |
    ASM (+DATA / +RECO)                      Oracle Restart 19c
                                                   |
                                             ASM (+DATA / +RECO)

    The apps tier and ASM storage stay in place; everything between them moves up to 19c, and the standalone non-CDB becomes a pluggable database inside a container.

    A note on certification before you begin

    Do not assume any upgrade path is valid. Before starting, confirm that your EBS code level, AD/TXK delta level, database RU, and interoperability patches are certified per the Oracle EBS 12.2 and Oracle Database 19c certification and interoperability documentation. Skipping this check is the most common reason an upgrade that “worked in the lab” falls apart in the field. Database RU levels and EBS interoperability certifications change frequently, so always verify the latest certification matrix in My Oracle Support before implementation — and treat the 19.30 RU referenced here as the level used in this engagement, not the only supported target.

    Rough timeline (for maintenance-window planning)

    PhaseTypical duration
    GI (Oracle Restart) upgrade30–60 min
    DBUA database upgrade2–4 hrs
    Non-CDB → PDB conversion~15 min
    AutoConfig & validation30–60 min

    These are working estimates from this engagement, not guarantees — database size, invalid-object count, and hardware all move the numbers. Treat the GI and DBUA phases as the long poles when you size the window.


    Phase 1 — Grid Infrastructure 12.1.0.2 → 19c

    The 19c Grid software was already laid down (gold image extracted into the new home), and the OUI installer had paused — as it always does — waiting for rootupgrade.sh to be run as root. That is where everything went sideways.

    The first wall: CLSRSC-324 — Could not open old init.cssd

    Running the upgrade script:

    /u01/app/19.0.0/grid/rootupgrade.sh

    failed almost immediately:

    CLSRSC-324: Could not open old init.cssd
    Died at /u01/app/19.0.0/grid/crs/install/s_crsutils.pm line 1569

    The 19c rootupgrade.sh reads the old configuration from init.cssd to learn where the previous CRS home lived. On this host, /etc/init.d/init.cssd was simply missing — it had been stripped out during earlier deconfig attempts before I picked the task up.

    First I confirmed what the upgrade actually expected as the old home:

    grep -iE "old_crs_home|asm_upgrade|oracle_home" /u01/app/19.0.0/grid/crs/install/crsconfig_params
    OLD_CRS_HOME=/u01/app/12.1.0.2/grid
    ASM_UPGRADE=true

    Then recreated the missing init.cssd pointing at the old home:

    cat > /etc/init.d/init.cssd << 'EOF'
    #!/bin/bash
    ORA_CRS_HOME=/u01/app/12.1.0.2/grid
    export ORA_CRS_HOME
    EOF
    chmod 755 /etc/init.d/init.cssd

    The second wall: a missing OLR pointer

    With init.cssd back, the next thing the script needs is the Oracle Local Registry pointer file, /etc/oracle/olr.loc — also missing.

    A common trap here is to assume the OLR is named <hostname>.olr. It was not. The actual file was local.ocr:

    ls -lrt /u01/app/12.1.0.2/grid/cdata/localhost/
    -rw-r----- 1 oracle dba 503484416 Feb 10 2017 local.ocr

    Always verify OLR integrity before trusting it:

    /u01/app/12.1.0.2/grid/bin/ocrcheck -local
    Device/File integrity check succeeded
    Local registry integrity check succeeded
    Logical corruption check succeeded

    Then recreate the pointer with the correct path:

    mkdir -p /etc/oracle
    cat > /etc/oracle/olr.loc << 'EOF'
    olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/localhost/local.ocr
    crs_home=/u01/app/12.1.0.2/grid
    EOF
    chmod 644 /etc/oracle/olr.loc

    The third major blocker: CLSRSC-348 / CLSRSC-318 — old HAS stack will not stop or start

    With the bootstrap files back, rootupgrade.sh progressed to step 2 (GetOldConfig) and then stopped again:

    CLSRSC-192: Unable to stop Oracle Restart
    CLSRSC-348: The Oracle Restart stack failed to stop

    The upgrade was trying to cleanly stop the old 12.1 HAS stack — but the old stack was not even contactable:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4639: Could not contact Oracle High Availability Services

    So I tried to start it, and hit the classic library-mismatch error:

    /u01/app/12.1.0.2/grid/bin/crsctl start has
    CRS-4652: Failure 3 in clsvswversion for the local node

    Root Cause

    The original 12.1 Oracle Restart stack was not in a healthy state. Previous deconfiguration attempts had removed critical bootstrap files and left HAS only partially configured.

    Because Oracle only upgrades an existing, functioning stack, rootupgrade.sh could not proceed until the old stack was cleanly rebuilt and verified. The fix is to deconfigure it fully, rebuild it on 12.1, confirm it is online — and only then let the 19c upgrade take over.

    Deconfigure the broken old stack:

    cd /u01/app/12.1.0.2/grid/crs/install
    perl roothas.pl -deconfig -force
    CLSRSC-337: Successfully deconfigured Oracle Restart stack

    Set the environment correctly (the LD_LIBRARY_PATH is what fixes the CRS-4652 library failure):

    export ORACLE_HOME=/u01/app/12.1.0.2/grid
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

    Rebuild the old 12.1 HAS stack using the 12.1 perl binary (do not use the system perl — use the one shipped with the old Grid home):

    /u01/app/12.1.0.2/grid/perl/bin/perl \
    -I/u01/app/12.1.0.2/grid/perl/lib \
    -I/u01/app/12.1.0.2/grid/crs/install \
    /u01/app/12.1.0.2/grid/crs/install/roothas.pl

    This reconfigured Oracle Restart successfully on 12.1 — note this is the script rebuilding the old stack, confirmed by:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Verify HAS is genuinely online before going further:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4638: Oracle High Availability Services is online

    Clearing stale checkpoints and finally running the upgrade

    The earlier failed rootupgrade.sh runs leave behind checkpoint files that block any retry — the script thinks it has already partly completed and refuses to start fresh. Clear them:

    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTHAS_STACK
    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTCRS_STACK

    Now re-run the 19c upgrade:

    /u01/app/19.0.0/grid/rootupgrade.sh

    This time it walked cleanly through all 12 steps (10–20 minutes), pinned the node, backed up the OLR at both the 12.1 and 19c homes, and finished with:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Back in the OUI GUI, clicking OK finalized the installer. ما شاء الله — Grid was now on 19c.

    One of the most common post-GI issues: ASM was never registered

    This one is worth its own warning, because it does not surface during the GI upgrade — it surfaces later, when you go to create the CDB and DBCA fails with something like DBT-06604: insufficient free space. The real reason is that ASM was not registered with Oracle Restart, so the tools silently fell back to filesystem storage, which did not have the room.

    A point worth being precise about: depending on the configuration, ASM may already exist and simply not be running — in which case srvctl start asm (or mounting the diskgroups) is all that is needed. Check before you add — running srvctl add asm when ASM is already configured will error, so confirm the current state with srvctl status asm / srvctl config asm first. In this environment ASM was not registered correctly, so it had to be added back:

    # verify first — does Oracle Restart already know about ASM?
    srvctl status asm
    srvctl config asm
    # in this environment it was not registered, so:
    srvctl add asm
    srvctl start asm

    And if the diskgroups are not mounted, mount them explicitly from the ASM instance:

    ALTER DISKGROUP DATA MOUNT;
    ALTER DISKGROUP RECO MOUNT;

    Lesson: after any Oracle Restart GI upgrade, always confirm srvctl status asm before you trust any DBCA/DBUA storage decision.

    Lessons Learned

    • Verify Oracle Restart is healthy before attempting an upgrade.
    • Confirm init.cssd and olr.loc exist and point to valid locations.
    • Clear stale checkpoint files before retrying rootupgrade.sh.
    • Validate ASM registration immediately after the upgrade.

    Phase 2 — Database 12.1.0.2 → 19c via DBUA

    With 19c Grid and ASM healthy, and the 19c CDB (EBSCDB) already created and patched (datapatch + catmgd.sql for the MGDSYS schema), the next phase was upgrading the non-CDB ebsdb in place from 12.1.0.2 to 19c. It stays a non-CDB for now — the PDB conversion is a deliberately separate Phase 3.

    For EBS environments, launch DBUA using the -keepEvents option so that the EBS-required events and hidden (underscore) parameters are preserved during the upgrade. Dropping those silently breaks the apps tier later.

    X11 forwarding across a user switch

    A very common, very annoying blocker: you SSH in as one OS user, then sudo to oracle, and X11 forwarding breaks because DISPLAY and the Xauthority cookie do not carry across the user switch. DBUA is a GUI tool, so this stops you cold.

    The fix is to capture the cookie as your login user and add it under oracle:

    # as the login user
    xauth list # note the cookie line for your display, e.g. dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    # as oracle
    xauth add dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    export DISPLAY=localhost:11.0
    xdpyinfo | head # must return display info, not "unable to open display"

    Tip: do not reuse the runbook author’s DISPLAY value (their own workstation name). It will never work for you — set your own.

    DBUA wizard choices that matter for EBS

    ScreenChoice
    CDB / PDB conversionLeave unchecked — conversion is a separate later phase
    Recovery option“I have my own backup/restore strategy” (RMAN backup already taken) — never leave this blank
    Recompile invalid objectsEnabled (parallel)
    Upgrade timezone dataEnabled
    compatibleLeave at 12.1.0.2 during the upgrade — preserves rollback. Raise it later, deliberately.
    Listener migrationKeep the CDB listener migration checked

    A couple of things that look like errors but are not:

    • The DBUA summary may show db_unique_name where you expect db_name. If your unique name differs from the SID, this is just DBUA displaying the unique name — verify against the live DB and move on.
    • The CDB listener (here on port 1523) must be up before DBUA can pass the network screen. If it is down, start it from the 19c home first: lsnrctl start <listener_name>.

    Wrap the whole thing in screen (or tmux) so an SSH drop does not kill the upgrade mid-flight. If screen is not installed, at minimum use nohup-style protection — a dropped DBUA mid-upgrade is a bad day.

    DBUA for an EBS 12.1.0.2 → 19c database typically runs 2–4 hours, plus 30–60 minutes of recompile/datapatch. Expect some ORA- noise in the logs (recompile warnings); watch for new error patterns, not the routine ones.

    Post-DBUA housekeeping

    Once the upgrade completes and all post-upgrade validation is complete (still as a non-CDB), raise COMPATIBLE to 19.0.0. Note that after COMPATIBLE is increased, rollback to the previous release is no longer possible — so this is a deliberate post-validation step, not an automatic next one:

    -- raise compatibility — this is the point of no return, do it deliberately
    ALTER SYSTEM SET compatible='19.0.0' SCOPE=SPFILE;
    -- compatible only takes effect after a restart
    SHUTDOWN IMMEDIATE;
    STARTUP;

    The exact value depends on your site standards and Oracle’s recommendation for the release — some shops set 19.0.0, others use the fully-qualified 19.0.0.0.0. Both forms are accepted; confirm which one your organization standardizes on before setting it.

    Then:

    • Snapshot a pfile from the spfile.
    • Relocate the spfile into the 19c home’s dbs/.
    • Remove obsolete parameters that 19c rejects — notably sec_case_sensitive_logon and utl_file_dir.
    • Run the XDB migration scripts: dbmsxdbschmig.sql then prvtxdbschmig.plb.
    • Run utlrp.sql (twice) to recompile. In this environment the invalid count dropped from over 12,000 to under 30 across the two passes — normal for an EBS database.

    At this point ebsdb is a healthy 19c non-CDB, ready to be plugged in.

    Lessons Learned

    • Always launch DBUA with -keepEvents for EBS — it preserves the required underscore parameters and event settings.
    • Fix X11 forwarding across the user switch before launching; never reuse the runbook author’s DISPLAY value.
    • Leave compatible at 12.1.0.2 during the upgrade and raise it deliberately afterward to preserve rollback.
    • Treat the db_unique_name-vs-db_name display in the summary as cosmetic, and start the CDB listener before the network screen.
    • Wrap the run in screen/tmux so an SSH drop cannot kill the upgrade mid-flight.

    Phase 3 — Non-CDB to PDB conversion

    This is where ebsdb becomes a lowercase PDB inside EBSCDB. EBS ships a set of txk driver scripts for exactly this path; the lowercase, case-sensitive PDB name is a detail that bites you in every subsequent command (ALTER SESSION SET CONTAINER="ebsdb" — with the quotes).

    Step 1 — Pre-creation tasks

    perl $AD_TOP/patch/115/bin/txkOnPremPrePDBCreationTasks.pl

    This generates the PDB descriptor XML (ebsdb_PDBDesc.xml, the output of DBMS_PDB.DESCRIBE, carrying the <ncdb2pdb>1</ncdb2pdb> flag) and shuts the non-CDB ebsdb down permanently — once you are on this path, there is no casual restart of the old non-CDB.

    Step 2 — Compatibility check

    perl $AD_TOP/patch/115/bin/txkChkPDBCompatability.pl

    This reported 8 violations — and per the runbook rules, all were ignorable: standard SQL-patch warnings and the usual benign messages, no character-set mismatch (a character-set violation is the one you must not ignore). Always read each violation; do not blanket-ignore.

    Step 3 — Create the PDB (plug in)

    perl $AD_TOP/patch/115/bin/txkCreatePDB.pl

    This performs a NOCOPY plug-in, allowing the existing ASM datafiles to remain in place while the database is converted into a PDB (the exact ASM file layout varies by environment and Oracle version), and runs noncdb_to_pdb.sql to finish the conversion. It completed in roughly 14 minutes, the PDB state was saved, and ebsdb showed up as CON_ID 3, READ WRITE. ما شاء الله.

    Step 4 — Post-creation tasks (the messy part)

    perl $AD_TOP/patch/115/bin/txkPostPDBCreationTasks.pl

    This is where the real fights happened. Three blockers, all around password files:

    a) ASM password file pre-staging. The script needs the CDB password file in ASM. Pre-stage it with orapwd from the DB home (not the Grid home) into the ASM path for the CDB, then register it:

    orapwd file='+DATA/EBSCDB/orapwEBSCDB' dbuniquename='EBSCDB' format=12
    srvctl modify database -d EBSCDB -pwfile '+DATA/EBSCDB/orapwEBSCDB'

    b) ORA-65066 on the SYSTEM password. Setting the SYSTEM password inside the new PDB threw ORA-65066 until it was applied across all containers:

    ALTER USER SYSTEM IDENTIFIED BY "<StrongPwd#1>" CONTAINER=ALL;

    c) OPW-00029 complexity failure. orapwd rejected the chosen SYSTEM password with OPW-00029 until it satisfied 19c’s special-character complexity rules. Use a value that genuinely meets the 19c verifier (uppercase + lowercase + digit + special character), e.g. <StrongPwd#1>.

    Step 5 — Post-PDB AutoConfig and the UTL_FILE_DIR trap

    Running AutoConfig on the DB tier after the conversion initially failed with a UTL_FILE_DIR error. The trace pointed at /usr/tmp being a symlink rather than a real directory, which the new UTL_FILE_DIR-replacement mechanism in 19c does not tolerate.

    The fix is to drive the directory configuration explicitly with txkCfgUtlfileDir.pl. First, manually create the directory-list file with real (non-symlink) paths:

    # e.g. /u01/app/oracle/product/19.0.0/dbhome_1/dbs/ebsdb_utlfiledir.txt
    # containing two real paths, e.g.:
    # /u01/app/oracle/ebsdb/temp
    # /u01/app/oracle/ebsdb/plsql

    Then run the script in its two modes:

    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=setUtlFileDir
    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=syncUtlFileDir

    After that, AutoConfig on the DB tier completed cleanly.

    Worth flagging for the longer term: UTL_FILE_DIR is deprecated, and Oracle’s recommended direction is to use database directory objects (CREATE DIRECTORY) for PL/SQL file I/O wherever your EBS code level supports it. The txkCfgUtlfileDir.pl mechanism above is the EBS-supported bridge for this release; treat directory objects as the strategic target.

    Lessons Learned

    • The lowercase, case-sensitive PDB name ("ebsdb" with quotes) propagates into every subsequent ALTER SESSION SET CONTAINER — get it right once and stay consistent.
    • Read every compatibility violation; ignore only the benign ones, and never ignore a character-set mismatch.
    • Budget real time for password files: pre-stage the ASM password file from the DB home, expect ORA-65066 (apply CONTAINER=ALL) and OPW-00029 (19c complexity rules).
    • A symlinked /usr/tmp breaks post-PDB AutoConfig — drive UTL_FILE_DIR explicitly via txkCfgUtlfileDir.pl with real paths.

    Phase 4 — Application-tier cutover (overview)

    With the database tier healthy at 19c as a PDB, the apps-tier work on appshost is the final phase:

    • Run AutoConfig on both filesystems (fs1 and fs2) to point the apps tier at the new 19c PDB connection details.
    • Clean up any stale adop sessions before starting fresh — an abandoned session from days earlier will block a new cycle. Run adop cleanup (cleanup_mode=full) and then a fresh adop phase=prepare.
    • Validate HugePages, the env files, and Configuration Manager.

    I will cover the full apps-tier cutover in detail in a follow-up post, since it deserves its own walkthrough.


    Final Validation Checklist

    After the upgrade and conversion were complete, the following validations were performed:

    ✓ Oracle Restart resources online and managed by srvctl
    ✓ ASM instance and diskgroups mounted
    ✓ CDB and PDB open in READ WRITE mode
    ✓ Database listener services registered correctly
    ✓ AutoConfig completed successfully on database and application tiers
    ✓ Concurrent Managers started successfully
    ✓ FNDSM service validated
    ✓ Internal Monitor running normally
    ✓ Forms and WebLogic services accessible
    ✓ adop phase=prepare completed without errors
    ✓ Invalid objects reviewed and reduced to an acceptable count
    ✓ Application smoke testing completed
    ✓ All EBS services registered in the listener
    ✓ Concurrent Manager processing validated
    ✓ Test concurrent request submitted and completed
    ✓ Forms login verified
    ✓ Workflow Mailer status verified
    ✓ OACORE, FORMS, and OAFM managed servers healthy
    ✓ No critical errors in AutoConfig logs
    ✓ adopscanlog reviewed with no blocking errors
    ✓ Database and ASM resources managed successfully through srvctl

    Quick post-upgrade SQL validation

    A handful of SQL checks confirm the database came back healthy — the component registry, the container/PDB state, and the invalid-object count. Set column formatting first so the output stays readable:

    1. Component registry — every component should report VALID (or UPGRADED) at 19.0.0:

    SET LINESIZE 200 PAGESIZE 100
    COL comp_name FORMAT A45
    COL version FORMAT A18
    COL status FORMAT A12
    SELECT comp_name, version, status
    FROM dba_registry
    ORDER BY comp_name;

    2. Container and PDB state — for a 19c multitenant environment, V$PDBS is the most useful check because it shows OPEN_MODE directly (the EBS PDB should be READ WRITE):

    COL name FORMAT A20
    SELECT con_id, name, open_mode
    FROM v$pdbs
    ORDER BY con_id;

    DBA_PDBS complements it by showing the persisted state — NORMAL for a healthy plugged-in PDB:

    COL pdb_name FORMAT A20
    COL status FORMAT A12
    SELECT pdb_name, status
    FROM dba_pdbs
    ORDER BY pdb_name;

    3. Invalid objects — review the breakdown by owner, then the overall total (an EBS database normally settles to a small count after two utlrp.sql passes):

    COL owner FORMAT A22
    COL object_type FORMAT A22
    SELECT NVL(owner,'(none)') AS owner,
    object_type,
    COUNT(*) AS invalid_count
    FROM dba_objects
    WHERE status = 'INVALID'
    GROUP BY owner, object_type
    ORDER BY invalid_count DESC;
    -- overall total
    SELECT COUNT(*) AS total_invalid
    FROM dba_objects
    WHERE status = 'INVALID';

    Although every environment is different, the upgrade itself proved relatively straightforward once the underlying Oracle Restart configuration issues were corrected. The majority of effort was spent validating and repairing legacy infrastructure assumptions rather than executing the upgrade utilities themselves.

    Key takeaways

    1. A GI upgrade is only as clean as the old stack underneath it. Most CLSRSC-318 / CLSRSC-324 / CLSRSC-348 failures trace back to an old HAS stack that was never properly configured or was half-deconfigured. Rebuild it cleanly on the old version first, then upgrade.
    2. Verify the bootstrap files yourself. Do not assume the OLR is <hostname>.olr — check cdata/localhost/ and verify with ocrcheck -local.
    3. Stale checkpoints block retries silently. Clear ROOTHAS_STACK / ROOTCRS_STACK before re-running rootupgrade.sh.
    4. After an Oracle Restart GI upgrade, register ASM (srvctl add asm / srvctl start asm) before you trust any storage-related tool.
    5. Use dbua -keepEvents for EBS, and fix X11 forwarding across the user switch before you launch.
    6. The PDB conversion’s hardest part is password files — ORA-65066, OPW-00029, and ASM password-file placement. Budget time for it.
    7. Post-PDB AutoConfig + UTL_FILE_DIR: a symlinked /usr/tmp will fail it. Drive the directory list explicitly with txkCfgUtlfileDir.pl using real paths.

    Looking back, the actual upgrade steps were straightforward. The real effort was understanding and repairing the assumptions the upgrade process makes about the existing environment. Missing bootstrap files, partially deconfigured Oracle Restart components, ASM registration gaps, password-file placement, and UTL_FILE_DIR migration issues consumed far more time than the upgrade binaries themselves. Those lessons are often absent from official documentation — which is exactly why I wanted to document the journey end to end.


    References

    • Oracle E-Business Suite Release 12.2 — Interoperability Notes for Oracle Database 19c (My Oracle Support)
    • Oracle Database 19c Upgrade Guide
    • Using UTL_FILE_DIR or Database Directories for PL/SQL File I/O in Oracle E-Business Suite (My Oracle Support)
    • Oracle E-Business Suite Multitenant / Pluggable Database (PDB) conversion documentation
    • Oracle Grid Infrastructure Installation and Upgrade Guide (Oracle Restart)

    (Look up the current My Oracle Support note IDs for your exact code and database levels — they are revised regularly.)

    If this write-up helped you during your own Oracle E-Business Suite upgrade, you’re welcome to follow the blog for more real-world Oracle Database, EBS, ADOP, cloning, upgrade, and troubleshooting guides drawn from production experience.


    Disclaimer: This post reflects my own hands-on experience and is shared for educational purposes only. All hostnames, database names, diskgroup names, and paths have been genericized. Always test thoroughly in a non-production environment and follow Oracle’s official documentation and your organization’s change-control process before applying any of these steps. The views expressed here are my own and do not represent those of any employer or client.

  • Oracle EBS R12.2 Clone Troubleshooting: RC-50221 Port Pool Conflict and FRM-92101 Forms Session Failed During Startup

    Disclaimer: This article is based on field experience and publicly shareable troubleshooting steps. Oracle MOS notes are referenced for further reading. Customers with active Oracle Support should review the referenced MOS documents for complete guidance.

    Category: Oracle EBS R12.2 | Cloning | Troubleshooting
    Level: Intermediate to Advanced
    Reference: MOS Doc ID 454427.1

    Introduction

    Cloning an Oracle E-Business Suite R12.2 application tier is a common activity — whether for setting up a DEV, TEST, or UAT environment. While the process is well-documented, real-world scenarios often throw up issues that aren’t immediately obvious from the error messages alone.

    In this post, I walk through two common issues encountered during an EBS R12.2 app tier clone using adcfgclone.pl:

    1. RC-50221: Port Pool Not Free — CloneContext failing due to an already-occupied port pool
    2. FRM-92101: Forms Session Failed During Startup — Forms not launching due to missing library dependencies or invalid symlinks

    Both issues are real, well-known, and worth documenting together since they often appear in sequence during a clone.


    Environment Details

    • Oracle EBS Version: R12.2.x
    • OS: Oracle Linux 7 (OL7) / RHEL 7
    • Clone type: Application Tier only (adcfgclone.pl appsTier)
    • Target server: Fresh clone target (non-production)

    Issue 1: RC-50221 — Port Pool Not Free

    Symptom

    When running adcfgclone.pl appsTier, the CloneContext step prompts for a Target System Port Pool and throws the following warning/error:

    Checking the port pool 0
    RC-50221: Warning: Port Pool 0 is not free.
    Please check logfile $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log for conflicts.
    RC-00201: Error: Not a valid port pool number
    ERROR: Context creation not completed successfully.

    CloneContext Log Location

    $COMMON_TOP/clone/bin/CloneContext_<timestamp>.log
    # Or typically:
    /u01/APPLTOP/EBSapps/comn/clone/bin/CloneContext_<timestamp>.log

    Root Cause

    Port pool 0 is already in use by another EBS instance running on the same server, causing CloneContext validation to fail. Port pool 0 generally corresponds to the initially configured EBS port set for that environment. Actual port assignments vary by release, topology, and previous configuration changes.

    ps -ef | grep weblogic | grep AdminServer
    lsof -i -P | grep LISTEN | grep <applcrp_user>

    Diagnosis

    Check which port pools are already in use:

    grep -i "port_pool" $CONTEXT_FILE
    cat $INST_TOP/admin/out/portpool.lst

    You can also use the following utility to validate port availability before proceeding with the clone:

    perl $AD_TOP/bin/txkValidateRollupPorts.pl \
    -contextfile=$CONTEXT_FILE \
    -outfile=/tmp/portcheck.txt

    Review /tmp/portcheck.txt for any conflicts before selecting a port pool number.

    Fix

    Target System Port Pool [0-99] : 1
    Checking the port pool 1
    done: Port Pool 1 is free

    Important Note: Using a different port pool causes Oracle EBS to calculate a different set of ports based on the predefined port pool mapping. The resulting ports are not always a simple increment of one and should be verified in the generated context file after CloneContext completes.

    Always verify the actual assigned ports:
    grep -i "port" $CONTEXT_FILE | grep -v "#"
    cat $INST_TOP/admin/out/portpool.lst

    If your firewall rules, load balancer, or application configuration depend on specific port numbers, plan accordingly or stop the existing instance first to free pool 0.

    Patch File System Port Pool

    For R12.2 dual file system architecture, the Patch file system must use a different port pool than the Run file system. AutoConfig and cloning utilities will calculate a separate set of ports accordingly.

    Patch file system should have different port pool than Run file system.
    Target System Port Pool [0-99] : 2

    Issue 2: FRM-92101 — Forms Session Failed During Startup

    Symptom

    FRM-92101: There was a failure in the Forms Server during startup.
    Please look into the web-server log file for details.
    Java Exception:
    oracle.forms.net.ConnectionException: Forms session <1> failed during startup:
    no response from runtime process

    What This Means

    The WebLogic Forms server (forms_server1) is running, but the underlying Forms runtime process (frmweb) is failing to start. WebLogic receives no response from it, and the session times out. FRM-92101 can have more than one root cause — the key is to run frmweb directly to identify which one applies.

    Log Analysis

    tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    This indicates that the Forms managed server is up and processing requests, shifting the investigation toward the underlying frmweb runtime process.

    First Diagnostic Step: Run frmweb Directly

    The most effective way to identify the root cause is to source the 10.1.2 environment and run frmweb manually:

    . /u01/APPLTOP/EBSapps/10.1.2/EBS_appserver.env
    $ORACLE_HOME/bin/frmweb

    The output will indicate which root cause applies:

    • Undefined symbol errors (e.g. Symbol nnftboot was referenced...not found) → Root Cause 1: Invalid ldflags symlink
    • Missing shared library error (e.g. libXm.so.2: cannot open shared object file) → Root Cause 2: Missing OpenMotif symlink

    Root Cause 1: Invalid ldflags Symlink

    Symptom

    Running frmweb directly produces undefined symbol errors:

    rtld: 0712-001 Symbol nnftboot was referenced from module frmweb(), but a runtime definition of the symbol was not found.
    rtld: 0712-001 Symbol ldap_search_s was referenced from module frmweb(), but a runtime definition of the symbol was not found.

    Root Cause

    Investigation revealed that the ldflags symbolic link under $ORACLE_HOME/lib32 was missing or pointing to an incorrect location, causing the Forms executable to fail during startup. This is a known issue after fresh installations or clones and is referenced in MOS Doc ID 454427.1.

    Verify ldflags

    ls -la $ORACLE_HOME/lib32/ldflags

    If the symlink is missing or pointing to a non-existent path, it needs to be corrected to point to $ORACLE_HOME/lib/ldflags.

    Fix

    After correcting the ldflags symbolic link and relinking the Forms components, frmweb started successfully. The relinking is performed using the Forms makefile under the 10.1.2 Oracle Home. Refer to MOS Doc ID 454427.1 for the complete steps applicable to your environment.


    Root Cause 2: Missing OpenMotif Library Symlink

    Symptom

    Running frmweb directly produces a missing shared library error:

    error while loading shared libraries:
    libXm.so.2: cannot open shared object file: No such file or directory

    Root Cause

    The Oracle Forms runtime binary frmweb is compiled against libXm.so.2 (OpenMotif 2.x). On newer Linux versions (OL7/OL8/RHEL7/RHEL8), the installed OpenMotif package provides libXm.so.4, but the compatibility symlink libXm.so.2 → libXm.so.4 is missing.

    Verify with ldd

    ldd $ORACLE_HOME/bin/frmweb | grep Xm
    # Before fix:
    libXm.so.2 => not found
    # After fix:
    libXm.so.2 => /usr/lib/libXm.so.4 (0x...)

    Option A: Create the Missing Symlink (Quick Fix)

    📌 Note on 32-bit vs 64-bit Libraries: On some Linux platforms the required library may reside under /usr/lib64 rather than /usr/lib. Always verify the actual location before creating symbolic links.

    # Step 1: Find the actual location of libXm.so.4
    ls -l /usr/lib/libXm.so*
    ls -l /usr/lib64/libXm.so*
    # Step 2: Create the symlink (run as root)
    # If library is in /usr/lib:
    ln -s /usr/lib/libXm.so.4 /usr/lib/libXm.so.2
    # If library is in /usr/lib64:
    ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.2
    # Step 3: Verify
    ldd $ORACLE_HOME/bin/frmweb | grep Xm
    # Expected: libXm.so.2 => /usr/lib/libXm.so.4 (or /usr/lib64/...)

    Option B: Install openmotif21 (Recommended Clean Fix)

    # Step 1: Check current state
    rpm -qa | grep motif
    # Step 2: Enable ol7_addons repo (as root)
    yum-config-manager --enable ol7_addons
    # Step 3: Remove lesstif if present
    yum remove lesstif
    # Step 4: Install openmotif21
    yum install --setopt=obsoletes=0 openmotif21

    Post-Fix: Bounce EBS Services and Validate

    cd $ADMIN_SCRIPTS_HOME
    ./adstpall.sh apps/<apps_password>
    ./adstrtal.sh apps/<apps_password>
    # Or start Forms server specifically:
    ./admanagedsrvctl.sh start forms_server1

    After starting the services, verify that frmweb runtime processes are spawning successfully:

    ps -ef | grep frmweb
    ps -ef | grep forms

    Then launch a Forms-based responsibility or function to confirm successful startup.


    Conclusion

    Both RC-50221 and FRM-92101 are common issues that can delay Oracle EBS R12.2 cloning activities. While the error messages may initially appear unrelated, a structured troubleshooting approach focusing on log analysis, port validation, and runtime dependency checks can quickly identify the root cause and reduce downtime.


    Summary

    IssueRoot CauseFix
    RC-50221: Port Pool Not FreeAnother EBS instance occupying port pool 0Use port pool 1 (or stop existing instance first)
    FRM-92101: ldflags issueldflags symlink missing or pointing to wrong locationFix ldflags symlink and relink Forms components (MOS Doc ID 454427.1)
    FRM-92101: OpenMotif issuelibXm.so.2 symlink missing after cloneln -s /usr/lib[64]/libXm.so.4 /usr/lib[64]/libXm.so.2 or install openmotif21

    Key Diagnostic Commands Cheat Sheet

    # Check which port pools are in use
    grep -i "port_pool" $CONTEXT_FILE
    cat $INST_TOP/admin/out/portpool.lst
    # Validate port availability
    perl $AD_TOP/bin/txkValidateRollupPorts.pl -contextfile=$CONTEXT_FILE -outfile=/tmp/portcheck.txt
    # Check running WebLogic processes
    ps -ef | grep weblogic | grep -v grep
    # Run frmweb manually to see actual error
    $ORACLE_HOME/bin/frmweb
    # Check frmweb library dependencies
    ldd $ORACLE_HOME/bin/frmweb | grep Xm
    # Check ldflags symlink
    ls -la $ORACLE_HOME/lib32/ldflags
    # Check OpenMotif installation
    rpm -qa | grep motif
    # Check Forms runtime processes
    ps -ef | grep frmweb
    # Check Forms server log
    tail -100 $EBS_DOMAIN_HOME/servers/forms_server1/logs/forms_server1.log

    Lessons Learned

    • FRM-92101 is often a symptom, not the root cause. The WebLogic log may show the Java exception, but the actual failure is in the native frmweb binary. Always run frmweb directly to see the real error.
    • FRM-92101 has multiple root causes. Check the actual error output from frmweb to distinguish between an ldflags relinking issue and a missing OpenMotif library symlink.
    • Use ldd to diagnose missing shared library dependencies. It quickly identifies unresolved libraries without having to dig through multiple log files.
    • Verify port pool availability before starting a clone. Stop existing instances or choose a different pool to avoid RC-50221 errors.
    • Always validate the resulting ports in the generated context file after CloneContext completes. Port pool mapping is predefined and the assigned ports should be confirmed before proceeding.
    • Newer Linux builds may include libXm.so.4 but not the compatibility symlink required by Oracle Forms. OL7/OL8 installations do not always create libXm.so.2 automatically.
    • Document port assignments after every clone to avoid firewall and load balancer configuration issues down the line.

    Keywords

    • Oracle EBS R12.2 Clone
    • Oracle EBS R12.2 Forms Error
    • RC-50221 Port Pool Not Free
    • RC-50221 CloneContext Failed
    • FRM-92101 Forms Session Failed During Startup
    • FRM-92101 No Response From Runtime Process
    • ldflags symlink Oracle Forms
    • libXm.so.2 not found
    • Oracle Forms Runtime Process
    • adcfgclone.pl appsTier
    • adcfgclone.pl appsTier Error
    • Oracle EBS Clone Troubleshooting
    • Oracle Linux 7 OpenMotif Issue
    • OpenMotif
    • frmweb
    • CloneContext

    References

    • MOS Doc ID 454427.1 — FRM-92101: There was a failure in the Forms Server during startup
    • Oracle EBS R12.2 Cloning Guide

    Related Technologies

    • Oracle E-Business Suite R12.2
    • Oracle Forms
    • Oracle WebLogic Server
    • Oracle Linux 7
    • OpenMotif
    • AutoConfig
    • Rapid Clone

    Syed Anwar Ahmed
    Oracle ACE Associate | Oracle Apps DBA
    Sharing real-world Oracle EBS administration, cloning, patching, and troubleshooting experiences.
    Blog: syedanwarahmedoracle.blog

  • Oracle EBS Login Issue: Real-Time Production Incident and Fix (OACORE + WebLogic Filter)


    📌 Introduction

    In Oracle E-Business Suite (EBS) environments, login failures are often perceived as simple application issues. However, in complex architectures, they can originate from multiple interacting layers across the application and middleware stack.

    In this blog, I’ll walk through a real-world production incident where an OACORE JVM issue combined with WebLogic security filtering resulted in complete login inaccessibility.

    This case highlights the importance of analyzing both performance and security layers together when troubleshooting critical application outages.


    ⚠️ Issue Summary

    • Users were unable to access the EBS login page
    • Pages were hanging or not loading
    • WebLogic console reported:
    Connection rejected, filter blocked Socket

    🔍 Initial Observation

    From the application server:

    • Load Average: ~10+

    👉 This indicated:

    • High CPU utilization
    • System under heavy stress
    • Potential JVM thread contention

    🔬 Detailed Analysis

    • One of the OACORE managed server JVMs became unresponsive
    • Long-running threads caused thread pool exhaustion
    • Incoming user requests began queueing

    At the same time:

    • WebLogic connection filter was actively enforcing access rules
    • Legitimate requests were being rejected under stressed conditions

    🧠 Understanding the Components

    OACORE (Application Layer)

    Handles:

    • Login requests
    • Forms processing
    • Core application logic

    If JVM threads are exhausted:
    👉 Requests queue → login hangs


    Oracle WebLogic Server Connection Filter

    EBS environments may use:

    oracle.apps.ad.tools.configuration.wls.filter.EBSConnectionFilterImpl

    This filter:

    • Enforces IP-based access control
    • Overrides default allow rules

    If misconfigured or stressed:
    👉 Legitimate traffic may be blocked


    🎯 Root Cause Analysis (RCA)

    The login issue was not caused by a single failure point, but by a combination of application tier resource exhaustion and restrictive middleware-level access control.

    • High CPU utilization and long-running threads caused one OACORE JVM to become unresponsive
    • Thread pool exhaustion led to request queuing, preventing new login requests from being processed

    Simultaneously:

    • The WebLogic connection filter (EBSConnectionFilterImpl) enforced strict access control policies
    • Under high load conditions, legitimate client requests were rejected with “filter blocked Socket”

    This interaction between performance degradation and security enforcement amplified the impact, resulting in complete login inaccessibility despite partial system availability.


    🛠️ Resolution Approach (Controlled & Safe)

    The resolution approach focused on stabilizing the JVM layer while validating and correcting middleware-level access controls in a controlled manner.


    🔹 Step 1: Identify Unresponsive JVM

    ps -ef | grep oacore

    ✔ Identify JVM with abnormal CPU or stuck behavior


    🔹 Step 2: Handle Stuck JVM (Controlled Action)

    ⚠️ Important Note:

    Forcefully terminating JVM processes should NOT be performed without validation.

    ✔ Recommended Approach:

    • Confirm the process is unresponsive
    • Ensure no critical transactions are running
    • Prefer controlled shutdown where possible

    ✔ Example (Only if fully unresponsive and approved):

    kill -9 <PID>

    👉 Node Manager can restart the JVM automatically after termination


    🔹 Step 3: Rolling Restart of OACORE

    admanagedsrvctl.sh stop oacore_server1
    admanagedsrvctl.sh start oacore_server1

    ✔ Ensures clean JVM state
    ✔ Restores thread pool balance


    🔹 Step 4: Validate WebLogic Connection Filter

    ⚠️ Important Note:

    The WebLogic connection filter is a security control and should NOT be disabled permanently.

    ✔ In this case:

    • Filter behavior was validated as part of troubleshooting
    • Temporary relaxation was used to confirm impact

    ✔ Recommended Approach:

    • Review allowed IP ranges
    • Validate filter configuration
    • Re-enable filter after correction

    The filter was re-enabled after validation and correction of configuration.


    🔹 Step 5: Restart Admin Server (If Required)

    adadminsrvctl.sh stop
    adadminsrvctl.sh start

    ✔ Ensures configuration changes are applied


    ✅ Final Outcome

    • System load reduced from ~10 → ~1.5
    • OACORE JVMs stabilized
    • Login page restored
    • Users successfully accessed the application

    📊 Key Learnings

    1. OACORE JVM issues can manifest as complete application outages due to thread exhaustion and request queuing.
    2. WebLogic connection filters can unintentionally block legitimate traffic, especially under high load conditions.
    3. Multi-layer failures (application + middleware + security) significantly amplify incident impact.
    4. System load metrics (CPU, load average) provide early indicators of JVM stress and should be monitored proactively.

    🧩 Preventive Measures

    • Monitor JVM thread utilization proactively
    • Review connection filter configurations periodically
    • Avoid peak concurrent load spikes
    • Implement alerting for high CPU / load conditions
    • Validate session and request handling patterns

    🔐 Governance Considerations

    • Avoid forceful JVM termination without proper validation
    • Do not disable security controls without understanding impact
    • Follow change management procedures in production
    • Coordinate with application and security teams before changes

    🏁 Conclusion

    This incident demonstrates that login failures in Oracle EBS are not always isolated to a single component but can result from complex interactions across application performance and middleware security layers.

    The combination of JVM resource exhaustion and connection filtering behavior created a compounded failure scenario, leading to complete login disruption.

    A structured, multi-layer troubleshooting approach—focused on performance, configuration, and governance—enabled effective resolution while minimizing risk.

    This reinforces the importance of analyzing both system behavior and security controls together when addressing critical production incidents.


    💡 Pro Tip

    When troubleshooting Oracle EBS login issues, always validate both:

    • JVM health (thread utilization, CPU load)
    • Middleware controls (connection filters, access rules)

    Ignoring either layer can lead to incomplete or misleading diagnosis.

  • Oracle EBS OACORE Server in FAILED_NOT_RESTARTABLE State: Real-Time Issue, RCA and Fix

    In Oracle E-Business Suite (EBS) environments, application tier stability is critical to ensure seamless user experience. However, there are scenarios where managed servers behave unexpectedly and require manual intervention. This post walks through a real-world production issue where an OACORE managed server entered a FAILED_NOT_RESTARTABLE state, its impact, root cause analysis, and how it was resolved.


    Environment Details

    • Oracle E-Business Suite: R12.2.x
    • Application Tier: WebLogic Managed Servers
    • Component Impacted: OACORE Server (oacore_server1)
    • Environment Type: Production

    Problem Statement

    An alert was received indicating oacore_server1 was in FAILED_NOT_RESTARTABLE state. Upon verification, the server was running but Node Manager could not auto-restart it.


    Key Observations

    Despite the OACORE server being in a failed state, the application remained accessible and functional — traffic was being handled by other OACORE servers. This is due to the multi-OACORE architecture with load balancing via OHS/Web tier. However, this creates a hidden risk: load redistribution increases pressure on remaining servers and can lead to cascading failures if not addressed promptly.


    Detailed Analysis

    • Managed server restart attempts failed during initialization
    • Bulk concurrent requests were actively running
    • CPU utilization spiked on the application tier
    • JVM resources were under pressure

    Understanding FAILED_NOT_RESTARTABLE

    In Oracle WebLogic Server, a managed server is marked as FAILED_NOT_RESTARTABLE after repeated unsuccessful restart attempts. This is a protective mechanism designed to prevent unstable restart loops when the server cannot recover successfully.


    Root Cause Analysis

    The OACORE managed server entered FAILED_NOT_RESTARTABLE state due to repeated startup failures following an unclean or resource-constrained shutdown. High CPU utilization and heavy concurrent workload placed JVM resources under pressure, preventing a clean restart cycle. Residual runtime artifacts (such as incomplete shutdown state or resource locks) prevented successful reinitialization, causing WebLogic to mark the server as FAILED_NOT_RESTARTABLE.


    Resolution

    cd $ADMIN_SCRIPTS_HOME
    ./admanagedsrvctl.sh stop oacore_server1
    ./admanagedsrvctl.sh start oacore_server1

    After the controlled restart, the server returned to RUNNING state with all deployments active and the application stable.


    Identify Inactive Forms Sessions

    Inactive sessions holding resources can contribute to JVM pressure. Use this query to identify them safely — do not terminate without proper validation and approvals:

    SELECT s.sid,
           s.serial#,
           s.username,
           s.status,
           s.program,
           s.machine,
           ROUND(s.last_call_et/3600,2) AS hours_inactive
    FROM v$session s
    WHERE s.status = 'INACTIVE'
    AND s.username = 'APPS'
    AND s.program LIKE 'frmweb%'
    AND s.last_call_et > 28800   -- 8 hours
    ORDER BY hours_inactive DESC;

    Reference only — do NOT execute without validation:

    ALTER SYSTEM KILL SESSION 'SID,SERIAL#' IMMEDIATE;

    Automate Session Monitoring

    Use this script to monitor inactive sessions every 8 hours via cron:

    #!/bin/bash
    export ORACLE_SID=your_sid
    export ORACLE_HOME=/path/to/oracle_home
    export PATH=$ORACLE_HOME/bin:$PATH
    
    sqlplus -s / as sysdba <<EOF
    SET LINES 200
    SET PAGES 200
    SELECT COUNT(*) AS inactive_sessions
    FROM v\$session
    WHERE status='INACTIVE'
    AND username='APPS'
    AND program LIKE 'frmweb%'
    AND last_call_et > 28800;
    EXIT;
    EOF
    # Crontab entry - every 8 hours
    0 */8 * * * /path/to/inactive_sessions.sh >> /tmp/inactive_sessions.log

    DBA Quick Commands

    -- Check system load
    top
    uptime
    ps -ef | grep oacore
    
    -- Check running concurrent requests
    SELECT request_id, phase_code, status_code
    FROM fnd_concurrent_requests
    WHERE phase_code = 'R';

    Key Takeaways

    • Application may appear healthy even when an OACORE server fails due to load balancing
    • FAILED_NOT_RESTARTABLE is a protective mechanism, not the root cause itself
    • Resource pressure and restart failures must be analyzed together
    • Controlled and governed actions are critical in production environments
    • Proactive session monitoring via automation helps prevent recurrence

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.