Syed Anwar Ahmed – Oracle DBA Blog

Tag: ADOP

  • Oracle EBS 12.2.4 Upgrade to Oracle Database 19c: Oracle Restart Upgrade, Non-CDB to PDB Conversion, and Lessons Learned

    Alhamdulillah, this post brings together one of the more involved upgrade journeys I have worked through recently — taking an Oracle E-Business Suite 12.2 database stack from 12.1.0.2 all the way to 19c, on a single standalone host running Oracle Restart.

    What made this engagement worth writing up is that it was not a clean, textbook upgrade. The Grid Infrastructure layer presented several unexpected challenges — a chain of CLSRSC failures that had to be cleared before it would move, and once Grid was finally on 19c, the database upgrade and the non-CDB to PDB conversion had their own set of gotchas around ASM registration, password files, and UTL_FILE_DIR.

    I have combined both phases — the Grid Infrastructure upgrade and the database upgrade plus multitenant conversion — into one walkthrough, because in the real world they are not separate stories. The GI upgrade forms the foundation for every subsequent phase. If you only read GI upgrade blogs in isolation, you miss how the pieces connect.

    All hostnames, database names, ASM diskgroup names, and paths below are genericized. The technical substance is exactly as it played out.

    Published: June 2026

    Environment:
    Oracle EBS 12.2.4 | AD/TXK Delta 13 | Oracle Database 19c (19.30)
    (Versions shown are from the environment used in this engagement, not a minimum certified baseline — always confirm your own certified levels.)

    Executive Summary

    This upgrade involved moving an Oracle E-Business Suite 12.2 database environment from Oracle Database 12.1.0.2 to 19c on a standalone Oracle Restart host. The journey consisted of four major phases:

    • Oracle Restart (Grid Infrastructure) upgrade from 12.1.0.2 to 19c
    • Database upgrade from 12.1.0.2 to 19c using DBUA
    • Non-CDB to PDB conversion using Oracle EBS TXK utilities
    • EBS application-tier reconfiguration and validation

    The most time-consuming portions were not the upgrades themselves, but resolving legacy Oracle Restart configuration issues, rebuilding HAS components, handling ASM registration after the GI upgrade, addressing password-file requirements during the PDB conversion, and resolving UTL_FILE_DIR migration challenges.

    This guide covers an end-to-end Oracle EBS 12.2.4 upgrade to Oracle Database 19c, including Oracle Restart upgrade, DBUA upgrade, non-CDB to PDB conversion, AutoConfig remediation, and post-upgrade validation.

    Environment overview

    ComponentDetail
    PlatformSingle standalone Linux host (dbhost) running Oracle Restart / standalone HAS — not full RAC Clusterware
    Apps tierSeparate node (appshost) under the applmgr-style owner
    EBS release12.2.4
    AD levelAD.C.Delta.13
    TXK levelTXK.C.Delta.13
    Source GI12.1.0.2
    Target GI19c (19.30)
    Source DBNon-CDB EBS database, SID ebsdb, on 12.1.0.2
    Target DB19c (19.30), ebsdb plugged in as a lowercase PDB inside CDB EBSCDB
    StorageASM diskgroups +DATA and +RECO
    Old GI home/u01/app/12.1.0.2/grid
    New GI home/u01/app/19.0.0/grid
    Old DB home/u01/app/oracle/product/12.1.0.2/dbhome_1
    New DB home/u01/app/oracle/product/19.0.0/dbhome_1

    The overall sequence was four phases:

    1. Grid Infrastructure 12.1.0.2 → 19c (Oracle Restart upgrade)
    2. Database 12.1.0.2 → 19c via DBUA (-keepEvents)
    3. Non-CDB → PDB conversion (plug ebsdb into EBSCDB)
    4. EBS application-tier cutover (AutoConfig, adop)

    The transformation at a glance

            BEFORE                                AFTER

    EBS Apps Tier                         EBS Apps Tier
         |                                     |
12.1.0.2 Non-CDB (ebsdb)        ===>      PDB ebsdb
         |                                     |
Oracle Restart 12.1.0.2                  CDB EBSCDB (19c)
         |                                     |
ASM (+DATA / +RECO)                      Oracle Restart 19c
                                               |
                                         ASM (+DATA / +RECO)

    The apps tier and ASM storage stay in place; everything between them moves up to 19c, and the standalone non-CDB becomes a pluggable database inside a container.

    A note on certification before you begin

    Do not assume any upgrade path is valid. Before starting, confirm that your EBS code level, AD/TXK delta level, database RU, and interoperability patches are certified per the Oracle EBS 12.2 and Oracle Database 19c certification and interoperability documentation. Skipping this check is the most common reason an upgrade that “worked in the lab” falls apart in the field. Database RU levels and EBS interoperability certifications change frequently, so always verify the latest certification matrix in My Oracle Support before implementation — and treat the 19.30 RU referenced here as the level used in this engagement, not the only supported target.

    Rough timeline (for maintenance-window planning)

    PhaseTypical duration
    GI (Oracle Restart) upgrade30–60 min
    DBUA database upgrade2–4 hrs
    Non-CDB → PDB conversion~15 min
    AutoConfig & validation30–60 min

    These are working estimates from this engagement, not guarantees — database size, invalid-object count, and hardware all move the numbers. Treat the GI and DBUA phases as the long poles when you size the window.

    Phase 1 — Grid Infrastructure 12.1.0.2 → 19c

    The 19c Grid software was already laid down (gold image extracted into the new home), and the OUI installer had paused — as it always does — waiting for rootupgrade.sh to be run as root. That is where everything went sideways.

    The first wall: CLSRSC-324 — Could not open old init.cssd

    Running the upgrade script:

    /u01/app/19.0.0/grid/rootupgrade.sh

    failed almost immediately:

    CLSRSC-324: Could not open old init.cssd
    Died at /u01/app/19.0.0/grid/crs/install/s_crsutils.pm line 1569

    The 19c rootupgrade.sh reads the old configuration from init.cssd to learn where the previous CRS home lived. On this host, /etc/init.d/init.cssd was simply missing — it had been stripped out during earlier deconfig attempts before I picked the task up.

    First I confirmed what the upgrade actually expected as the old home:

    grep -iE "old_crs_home|asm_upgrade|oracle_home" /u01/app/19.0.0/grid/crs/install/crsconfig_params
    OLD_CRS_HOME=/u01/app/12.1.0.2/grid
    ASM_UPGRADE=true

    Then recreated the missing init.cssd pointing at the old home:

    cat > /etc/init.d/init.cssd << 'EOF'
    #!/bin/bash
    ORA_CRS_HOME=/u01/app/12.1.0.2/grid
    export ORA_CRS_HOME
    EOF
    chmod 755 /etc/init.d/init.cssd

    The second wall: a missing OLR pointer

    With init.cssd back, the next thing the script needs is the Oracle Local Registry pointer file, /etc/oracle/olr.loc — also missing.

    A common trap here is to assume the OLR is named <hostname>.olr. It was not. The actual file was local.ocr:

    ls -lrt /u01/app/12.1.0.2/grid/cdata/localhost/
    -rw-r----- 1 oracle dba 503484416 Feb 10 2017 local.ocr

    Always verify OLR integrity before trusting it:

    /u01/app/12.1.0.2/grid/bin/ocrcheck -local
    Device/File integrity check succeeded
    Local registry integrity check succeeded
    Logical corruption check succeeded

    Then recreate the pointer with the correct path:

    mkdir -p /etc/oracle
    cat > /etc/oracle/olr.loc << 'EOF'
    olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/localhost/local.ocr
    crs_home=/u01/app/12.1.0.2/grid
    EOF
    chmod 644 /etc/oracle/olr.loc

    The third major blocker: CLSRSC-348 / CLSRSC-318 — old HAS stack will not stop or start

    With the bootstrap files back, rootupgrade.sh progressed to step 2 (GetOldConfig) and then stopped again:

    CLSRSC-192: Unable to stop Oracle Restart
    CLSRSC-348: The Oracle Restart stack failed to stop

    The upgrade was trying to cleanly stop the old 12.1 HAS stack — but the old stack was not even contactable:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4639: Could not contact Oracle High Availability Services

    So I tried to start it, and hit the classic library-mismatch error:

    /u01/app/12.1.0.2/grid/bin/crsctl start has
    CRS-4652: Failure 3 in clsvswversion for the local node

    Root Cause

    The original 12.1 Oracle Restart stack was not in a healthy state. Previous deconfiguration attempts had removed critical bootstrap files and left HAS only partially configured.

    Because Oracle only upgrades an existing, functioning stack, rootupgrade.sh could not proceed until the old stack was cleanly rebuilt and verified. The fix is to deconfigure it fully, rebuild it on 12.1, confirm it is online — and only then let the 19c upgrade take over.

    Deconfigure the broken old stack:

    cd /u01/app/12.1.0.2/grid/crs/install
    perl roothas.pl -deconfig -force
    CLSRSC-337: Successfully deconfigured Oracle Restart stack

    Set the environment correctly (the LD_LIBRARY_PATH is what fixes the CRS-4652 library failure):

    export ORACLE_HOME=/u01/app/12.1.0.2/grid
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

    Rebuild the old 12.1 HAS stack using the 12.1 perl binary (do not use the system perl — use the one shipped with the old Grid home):

    /u01/app/12.1.0.2/grid/perl/bin/perl \
      -I/u01/app/12.1.0.2/grid/perl/lib \
      -I/u01/app/12.1.0.2/grid/crs/install \
      /u01/app/12.1.0.2/grid/crs/install/roothas.pl

    This reconfigured Oracle Restart successfully on 12.1 — note this is the script rebuilding the old stack, confirmed by:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Verify HAS is genuinely online before going further:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4638: Oracle High Availability Services is online

    Clearing stale checkpoints and finally running the upgrade

    The earlier failed rootupgrade.sh runs leave behind checkpoint files that block any retry — the script thinks it has already partly completed and refuses to start fresh. Clear them:

    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTHAS_STACK
    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTCRS_STACK

    Now re-run the 19c upgrade:

    /u01/app/19.0.0/grid/rootupgrade.sh

    This time it walked cleanly through all 12 steps (10–20 minutes), pinned the node, backed up the OLR at both the 12.1 and 19c homes, and finished with:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Back in the OUI GUI, clicking OK finalized the installer. ما شاء الله — Grid was now on 19c.

    One of the most common post-GI issues: ASM was never registered

    This one is worth its own warning, because it does not surface during the GI upgrade — it surfaces later, when you go to create the CDB and DBCA fails with something like DBT-06604: insufficient free space. The real reason is that ASM was not registered with Oracle Restart, so the tools silently fell back to filesystem storage, which did not have the room.

    A point worth being precise about: depending on the configuration, ASM may already exist and simply not be running — in which case srvctl start asm (or mounting the diskgroups) is all that is needed. Check before you add — running srvctl add asm when ASM is already configured will error, so confirm the current state with srvctl status asm / srvctl config asm first. In this environment ASM was not registered correctly, so it had to be added back:

    # verify first — does Oracle Restart already know about ASM?
    srvctl status asm
    srvctl config asm
    # in this environment it was not registered, so:
    srvctl add asm
    srvctl start asm

    And if the diskgroups are not mounted, mount them explicitly from the ASM instance:

    ALTER DISKGROUP DATA MOUNT;
    ALTER DISKGROUP RECO MOUNT;

    Lesson: after any Oracle Restart GI upgrade, always confirm srvctl status asm before you trust any DBCA/DBUA storage decision.

    Lessons Learned

    • Verify Oracle Restart is healthy before attempting an upgrade.
    • Confirm init.cssd and olr.loc exist and point to valid locations.
    • Clear stale checkpoint files before retrying rootupgrade.sh.
    • Validate ASM registration immediately after the upgrade.

    Phase 2 — Database 12.1.0.2 → 19c via DBUA

    With 19c Grid and ASM healthy, and the 19c CDB (EBSCDB) already created and patched (datapatch + catmgd.sql for the MGDSYS schema), the next phase was upgrading the non-CDB ebsdb in place from 12.1.0.2 to 19c. It stays a non-CDB for now — the PDB conversion is a deliberately separate Phase 3.

    For EBS environments, launch DBUA using the -keepEvents option so that the EBS-required events and hidden (underscore) parameters are preserved during the upgrade. Dropping those silently breaks the apps tier later.

    X11 forwarding across a user switch

    A very common, very annoying blocker: you SSH in as one OS user, then sudo to oracle, and X11 forwarding breaks because DISPLAY and the Xauthority cookie do not carry across the user switch. DBUA is a GUI tool, so this stops you cold.

    The fix is to capture the cookie as your login user and add it under oracle:

    # as the login user
    xauth list        # note the cookie line for your display, e.g. dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    # as oracle
    xauth add dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    export DISPLAY=localhost:11.0
    xdpyinfo | head   # must return display info, not "unable to open display"

    Tip: do not reuse the runbook author’s DISPLAY value (their own workstation name). It will never work for you — set your own.

    DBUA wizard choices that matter for EBS

    ScreenChoice
    CDB / PDB conversionLeave unchecked — conversion is a separate later phase
    Recovery option“I have my own backup/restore strategy” (RMAN backup already taken) — never leave this blank
    Recompile invalid objectsEnabled (parallel)
    Upgrade timezone dataEnabled
    compatibleLeave at 12.1.0.2 during the upgrade — preserves rollback. Raise it later, deliberately.
    Listener migrationKeep the CDB listener migration checked

    A couple of things that look like errors but are not:

    • The DBUA summary may show db_unique_name where you expect db_name. If your unique name differs from the SID, this is just DBUA displaying the unique name — verify against the live DB and move on.
    • The CDB listener (here on port 1523) must be up before DBUA can pass the network screen. If it is down, start it from the 19c home first: lsnrctl start <listener_name>.

    Wrap the whole thing in screen (or tmux) so an SSH drop does not kill the upgrade mid-flight. If screen is not installed, at minimum use nohup-style protection — a dropped DBUA mid-upgrade is a bad day.

    DBUA for an EBS 12.1.0.2 → 19c database typically runs 2–4 hours, plus 30–60 minutes of recompile/datapatch. Expect some ORA- noise in the logs (recompile warnings); watch for new error patterns, not the routine ones.

    Post-DBUA housekeeping

    Once the upgrade completes and all post-upgrade validation is complete (still as a non-CDB), raise COMPATIBLE to 19.0.0. Note that after COMPATIBLE is increased, rollback to the previous release is no longer possible — so this is a deliberate post-validation step, not an automatic next one:

    -- raise compatibility — this is the point of no return, do it deliberately
    ALTER SYSTEM SET compatible='19.0.0' SCOPE=SPFILE;
    -- compatible only takes effect after a restart
    SHUTDOWN IMMEDIATE;
    STARTUP;

    The exact value depends on your site standards and Oracle’s recommendation for the release — some shops set 19.0.0, others use the fully-qualified 19.0.0.0.0. Both forms are accepted; confirm which one your organization standardizes on before setting it.

    Then:

    • Snapshot a pfile from the spfile.
    • Relocate the spfile into the 19c home’s dbs/.
    • Remove obsolete parameters that 19c rejects — notably sec_case_sensitive_logon and utl_file_dir.
    • Run the XDB migration scripts: dbmsxdbschmig.sql then prvtxdbschmig.plb.
    • Run utlrp.sql (twice) to recompile. In this environment the invalid count dropped from over 12,000 to under 30 across the two passes — normal for an EBS database.

    At this point ebsdb is a healthy 19c non-CDB, ready to be plugged in.

    Lessons Learned

    • Always launch DBUA with -keepEvents for EBS — it preserves the required underscore parameters and event settings.
    • Fix X11 forwarding across the user switch before launching; never reuse the runbook author’s DISPLAY value.
    • Leave compatible at 12.1.0.2 during the upgrade and raise it deliberately afterward to preserve rollback.
    • Treat the db_unique_name-vs-db_name display in the summary as cosmetic, and start the CDB listener before the network screen.
    • Wrap the run in screen/tmux so an SSH drop cannot kill the upgrade mid-flight.

    Phase 3 — Non-CDB to PDB conversion

    This is where ebsdb becomes a lowercase PDB inside EBSCDB. EBS ships a set of txk driver scripts for exactly this path; the lowercase, case-sensitive PDB name is a detail that bites you in every subsequent command (ALTER SESSION SET CONTAINER="ebsdb" — with the quotes).

    Step 1 — Pre-creation tasks

    perl $AD_TOP/patch/115/bin/txkOnPremPrePDBCreationTasks.pl

    This generates the PDB descriptor XML (ebsdb_PDBDesc.xml, the output of DBMS_PDB.DESCRIBE, carrying the <ncdb2pdb>1</ncdb2pdb> flag) and shuts the non-CDB ebsdb down permanently — once you are on this path, there is no casual restart of the old non-CDB.

    Step 2 — Compatibility check

    perl $AD_TOP/patch/115/bin/txkChkPDBCompatability.pl

    This reported 8 violations — and per the runbook rules, all were ignorable: standard SQL-patch warnings and the usual benign messages, no character-set mismatch (a character-set violation is the one you must not ignore). Always read each violation; do not blanket-ignore.

    Step 3 — Create the PDB (plug in)

    perl $AD_TOP/patch/115/bin/txkCreatePDB.pl

    This performs a NOCOPY plug-in, allowing the existing ASM datafiles to remain in place while the database is converted into a PDB (the exact ASM file layout varies by environment and Oracle version), and runs noncdb_to_pdb.sql to finish the conversion. It completed in roughly 14 minutes, the PDB state was saved, and ebsdb showed up as CON_ID 3, READ WRITE. ما شاء الله.

    Step 4 — Post-creation tasks (the messy part)

    perl $AD_TOP/patch/115/bin/txkPostPDBCreationTasks.pl

    This is where the real fights happened. Three blockers, all around password files:

    a) ASM password file pre-staging. The script needs the CDB password file in ASM. Pre-stage it with orapwd from the DB home (not the Grid home) into the ASM path for the CDB, then register it:

    orapwd file='+DATA/EBSCDB/orapwEBSCDB' dbuniquename='EBSCDB' format=12
    srvctl modify database -d EBSCDB -pwfile '+DATA/EBSCDB/orapwEBSCDB'

    b) ORA-65066 on the SYSTEM password. Setting the SYSTEM password inside the new PDB threw ORA-65066 until it was applied across all containers:

    ALTER USER SYSTEM IDENTIFIED BY "<StrongPwd#1>" CONTAINER=ALL;

    c) OPW-00029 complexity failure. orapwd rejected the chosen SYSTEM password with OPW-00029 until it satisfied 19c’s special-character complexity rules. Use a value that genuinely meets the 19c verifier (uppercase + lowercase + digit + special character), e.g. <StrongPwd#1>.

    Step 5 — Post-PDB AutoConfig and the UTL_FILE_DIR trap

    Running AutoConfig on the DB tier after the conversion initially failed with a UTL_FILE_DIR error. The trace pointed at /usr/tmp being a symlink rather than a real directory, which the new UTL_FILE_DIR-replacement mechanism in 19c does not tolerate.

    The fix is to drive the directory configuration explicitly with txkCfgUtlfileDir.pl. First, manually create the directory-list file with real (non-symlink) paths:

    # e.g. /u01/app/oracle/product/19.0.0/dbhome_1/dbs/ebsdb_utlfiledir.txt
    # containing two real paths, e.g.:
    #   /u01/app/oracle/ebsdb/temp
    #   /u01/app/oracle/ebsdb/plsql

    Then run the script in its two modes:

    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=setUtlFileDir
    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=syncUtlFileDir

    After that, AutoConfig on the DB tier completed cleanly.

    Worth flagging for the longer term: UTL_FILE_DIR is deprecated, and Oracle’s recommended direction is to use database directory objects (CREATE DIRECTORY) for PL/SQL file I/O wherever your EBS code level supports it. The txkCfgUtlfileDir.pl mechanism above is the EBS-supported bridge for this release; treat directory objects as the strategic target.

    Lessons Learned

    • The lowercase, case-sensitive PDB name ("ebsdb" with quotes) propagates into every subsequent ALTER SESSION SET CONTAINER — get it right once and stay consistent.
    • Read every compatibility violation; ignore only the benign ones, and never ignore a character-set mismatch.
    • Budget real time for password files: pre-stage the ASM password file from the DB home, expect ORA-65066 (apply CONTAINER=ALL) and OPW-00029 (19c complexity rules).
    • A symlinked /usr/tmp breaks post-PDB AutoConfig — drive UTL_FILE_DIR explicitly via txkCfgUtlfileDir.pl with real paths.

    Phase 4 — Application-tier cutover (overview)

    With the database tier healthy at 19c as a PDB, the apps-tier work on appshost is the final phase:

    • Run AutoConfig on both filesystems (fs1 and fs2) to point the apps tier at the new 19c PDB connection details.
    • Clean up any stale adop sessions before starting fresh — an abandoned session from days earlier will block a new cycle. Run adop cleanup (cleanup_mode=full) and then a fresh adop phase=prepare.
    • Validate HugePages, the env files, and Configuration Manager.

    I will cover the full apps-tier cutover in detail in a follow-up post, since it deserves its own walkthrough.

    Final Validation Checklist

    After the upgrade and conversion were complete, the following validations were performed:

    ✓ Oracle Restart resources online and managed by srvctl
    ✓ ASM instance and diskgroups mounted
    ✓ CDB and PDB open in READ WRITE mode
    ✓ Database listener services registered correctly
    ✓ AutoConfig completed successfully on database and application tiers
    ✓ Concurrent Managers started successfully
    ✓ FNDSM service validated
    ✓ Internal Monitor running normally
    ✓ Forms and WebLogic services accessible
    ✓ adop phase=prepare completed without errors
    ✓ Invalid objects reviewed and reduced to an acceptable count
    ✓ Application smoke testing completed
    ✓ All EBS services registered in the listener
    ✓ Concurrent Manager processing validated
    ✓ Test concurrent request submitted and completed
    ✓ Forms login verified
    ✓ Workflow Mailer status verified
    ✓ OACORE, FORMS, and OAFM managed servers healthy
    ✓ No critical errors in AutoConfig logs
    ✓ adopscanlog reviewed with no blocking errors
    ✓ Database and ASM resources managed successfully through srvctl

    Quick post-upgrade SQL validation

    A handful of SQL checks confirm the database came back healthy — the component registry, the container/PDB state, and the invalid-object count. Set column formatting first so the output stays readable:

    1. Component registry — every component should report VALID (or UPGRADED) at 19.0.0:

    SET LINESIZE 200 PAGESIZE 100
    COL comp_name FORMAT A45
    COL version   FORMAT A18
    COL status    FORMAT A12
    SELECT comp_name, version, status
    FROM   dba_registry
    ORDER  BY comp_name;

    2. Container and PDB state — for a 19c multitenant environment, V$PDBS is the most useful check because it shows OPEN_MODE directly (the EBS PDB should be READ WRITE):

    COL name FORMAT A20
    SELECT con_id, name, open_mode
    FROM   v$pdbs
    ORDER  BY con_id;

    DBA_PDBS complements it by showing the persisted state — NORMAL for a healthy plugged-in PDB:

    COL pdb_name FORMAT A20
    COL status   FORMAT A12
    SELECT pdb_name, status
    FROM   dba_pdbs
    ORDER  BY pdb_name;

    3. Invalid objects — review the breakdown by owner, then the overall total (an EBS database normally settles to a small count after two utlrp.sql passes):

    COL owner       FORMAT A22
    COL object_type FORMAT A22
    SELECT NVL(owner,'(none)') AS owner,
           object_type,
           COUNT(*)             AS invalid_count
    FROM   dba_objects
    WHERE  status = 'INVALID'
    GROUP  BY owner, object_type
    ORDER  BY invalid_count DESC;
    -- overall total
    SELECT COUNT(*) AS total_invalid
    FROM   dba_objects
    WHERE  status = 'INVALID';

    Although every environment is different, the upgrade itself proved relatively straightforward once the underlying Oracle Restart configuration issues were corrected. The majority of effort was spent validating and repairing legacy infrastructure assumptions rather than executing the upgrade utilities themselves.

    Key takeaways

    1. A GI upgrade is only as clean as the old stack underneath it. Most CLSRSC-318 / CLSRSC-324 / CLSRSC-348 failures trace back to an old HAS stack that was never properly configured or was half-deconfigured. Rebuild it cleanly on the old version first, then upgrade.
    2. Verify the bootstrap files yourself. Do not assume the OLR is <hostname>.olr — check cdata/localhost/ and verify with ocrcheck -local.
    3. Stale checkpoints block retries silently. Clear ROOTHAS_STACK / ROOTCRS_STACK before re-running rootupgrade.sh.
    4. After an Oracle Restart GI upgrade, register ASM (srvctl add asm / srvctl start asm) before you trust any storage-related tool.
    5. Use dbua -keepEvents for EBS, and fix X11 forwarding across the user switch before you launch.
    6. The PDB conversion’s hardest part is password files — ORA-65066, OPW-00029, and ASM password-file placement. Budget time for it.
    7. Post-PDB AutoConfig + UTL_FILE_DIR: a symlinked /usr/tmp will fail it. Drive the directory list explicitly with txkCfgUtlfileDir.pl using real paths.

    Looking back, the actual upgrade steps were straightforward. The real effort was understanding and repairing the assumptions the upgrade process makes about the existing environment. Missing bootstrap files, partially deconfigured Oracle Restart components, ASM registration gaps, password-file placement, and UTL_FILE_DIR migration issues consumed far more time than the upgrade binaries themselves. Those lessons are often absent from official documentation — which is exactly why I wanted to document the journey end to end.

    References

    • Oracle E-Business Suite Release 12.2 — Interoperability Notes for Oracle Database 19c (My Oracle Support)
    • Oracle Database 19c Upgrade Guide
    • Using UTL_FILE_DIR or Database Directories for PL/SQL File I/O in Oracle E-Business Suite (My Oracle Support)
    • Oracle E-Business Suite Multitenant / Pluggable Database (PDB) conversion documentation
    • Oracle Grid Infrastructure Installation and Upgrade Guide (Oracle Restart)

    (Look up the current My Oracle Support note IDs for your exact code and database levels — they are revised regularly.)

    If this write-up helped you during your own Oracle E-Business Suite upgrade, you’re welcome to follow the blog for more real-world Oracle Database, EBS, ADOP, cloning, upgrade, and troubleshooting guides drawn from production experience.

    Disclaimer: This post reflects my own hands-on experience and is shared for educational purposes only. All hostnames, database names, diskgroup names, and paths have been genericized. Always test thoroughly in a non-production environment and follow Oracle’s official documentation and your organization’s change-control process before applying any of these steps. The views expressed here are my own and do not represent those of any employer or client.

  • Oracle EBS 12.2 — ADOP fs_clone Failure: Failed to Delete FMW_Home (Root Cause & Fix)

    Category: Oracle EBS 12.2  |  Topic: ADOP Patching  |  Difficulty: Intermediate  |  Oracle Support: Search ADOP fs_clone Failed to delete FMW_Home on My Oracle Support

    Introduction

    Oracle EBS 12.2 introduced Online Patching (ADOP), which relies on a dual file system architecture — a Run File System (fs2) where production runs, and a Patch File System (fs1) where patches are applied. The fs_clone phase synchronises fs1 from fs2 at the start of each patching cycle, making fs1 a fresh copy of the production file system.

    One of the most common issues encountered during fs_clone is a failure while trying to delete the FMW_Home directory on the Patch FS. This blog walks through a real production scenario on a 2-node RAC database with 4 application server nodes — covering the exact error, step-by-step diagnostic process, root cause identification, fix applied, and the final successful run with actual timings. All server-specific details have been anonymised.

    This issue applies to Oracle EBS 12.2.x on all platforms. For related Oracle Support articles, search “ADOP fs_clone Failed to delete FMW_Home” on My Oracle Support.

    Environment

    ParameterValue
    ApplicationOracle E-Business Suite 12.2 (2-Node RAC DB + 4 Application Server Nodes)
    ADOP VersionC.Delta.13
    ADOP Session ID129
    Run File System (fs2)/u01/app/fs2 (Production — active)
    Patch File System (fs1)/u01/app/fs1 (Inactive — patching target)
    Shared StorageNFS-mounted shared volume (1.3T, 447G free)
    OS Userapplmgr

    Incident Timeline

    EventTimestampDurationOutcome
    1st run startedApr 11, 2026 17:07:35—time adop phase=fs_clone executed
    1st run failedApr 11, 2026 ~18:41~1h 34mFATAL ERROR — FMW_Home deletion failed
    Diagnosis performedApr 11, 2026 18:45–19:44~59mRoot cause identified — root-owned OHS log files
    Fix applied (mv)Apr 11, 2026 ~19:44SecondsFMW_Home renamed to dated backup as applmgr
    2nd run startedApr 11, 2026 19:45—time adop phase=fs_clone re-executed after fix
    2nd run completedApr 12, 2026 00:49:085h 21m 25sSUCCESS — all 4 app nodes completed ✅
    Total session elapsedApr 11 17:07 → Apr 12 00:497h 46m 33sFull session including failed run + fix + retry

    The Issue

    During time adop phase=fs_clone, the synchronisation process was progressing normally — staging the file system clone, detaching Oracle Homes, removing APPL_TOP and COMM_TOP — until it reached stage 6 (REMOVE-1012-ORACLE-HOME) inside the removeFMWHome() function, where it attempted to delete the FMW_Home directory on the Patch FS and hit a fatal error.

    Error on the Console

    fs_clone/remote_execution_result_level1.xml:
*******FATAL ERROR*******
PROGRAM : (.../fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Apr 11 18:41:40 2026
FUNCTION: main::removeDirectory [ Level 1 ]
ERRORMSG: Failed to delete the directory /u01/app/fs1/FMW_Home.
[UNEXPECTED]fs_clone has failed

    Key Log File: txkADOPPreparePhaseSynchronize.log

    The primary log file is located at:

    $ADOP_LOG_DIR/<session_id>/<timestamp>/fs_clone/<node>/TXK_SYNC_create/
    txkADOPPreparePhaseSynchronize.log

    Inside this log, the clone status progression was clearly visible:

    ========================== Inside getCloneStatus()... ==========================
clone_status             = REMOVE-1012-ORACLE-HOME
clone_status_from_caller = 7
clone_status_from_db     = 6
Removing the directory: /u01/app/fs1/FMW_Home
Failed to delete the directory /u01/app/fs1/FMW_Home.
*******FATAL ERROR*******
FUNCTION: main::removeDirectory [ Level 1 ]
ERRORMSG: Failed to delete the directory /u01/app/fs1/FMW_Home.

    clone_status_from_db = 6 indicates the process had already completed: fs_clone staging, detach of Oracle Homes, removal of APPL_TOP, COMM_TOP, and 10.1.2 Oracle Home. It failed specifically and only while removing FMW_Home.

    ADOP fs_clone Stage Flow

    Stage DBClone StatusDescription
    1STARTEDSession initialised
    2FSCLONESTAGE-DONEFile system staging completed
    3DEREGISTER-ORACLE-HOMESOracle Homes deregistered from inventory
    4REMOVE-APPL-TOPAPPL_TOP removed from Patch FS
    5REMOVE-COMM-TOPCOMM_TOP removed from Patch FS
    6REMOVE-1012-ORACLE-HOMERemoving FMW_Home — ❌ FAILED HERE
    7+(clone proceeds…)Clone fs2 to fs1, re-register homes, config clone

    Diagnostic Steps

    Step 1 — Confirm You Are on the Correct File System

    Most critical check first. The target must be on Patch FS (fs1), never Run FS (fs2):

    echo $FILE_EDITION   # Must show: run
echo $RUN_BASE       # Must show path to fs2
    $ echo $FILE_EDITION
run
$ echo $RUN_BASE
/u01/app/fs2

    ⚠️ If FILE_EDITION shows patch, stop immediately — source the Run FS environment before proceeding.

    Step 2 — Check for Open File Handles

    lsof +D /u01/app/fs1/FMW_Home 2>/dev/null
fuser -cu /u01/app/fs1/FMW_Home 2>&1

    In our case both commands returned empty output — no active process was holding FMW_Home open.

    Step 3 — Identify Root-Owned Files

    find /u01/app/fs1/FMW_Home ! -user applmgr -ls 2>/dev/null

    Output revealed multiple root-owned files under the OHS instance directories:

    drwxr-x---  3 root root 4096 Feb 15 06:48 .../EBS_web_OHS4/auditlogs/OHS
-rw-------  1 root root    0 Feb 15 06:48 .../EBS_web_OHS4/diagnostics/logs/OHS/EBS_web/sec_audit_log
-rw-r-----  1 root root 5670 Feb 15 06:49 .../EBS_web_OHS4/diagnostics/logs/OHS/EBS_web/EBS_web.log
-rw-r-----  1 root root  249 Feb 15 06:49 .../EBS_web_OHS4/diagnostics/logs/OHS/EBS_web/access_log
... (same pattern for EBS_web_OHS2 and EBS_web_OHS3)

    All root-owned files were dated February 15 — nearly 2 months stale. This confirmed they were leftovers from OHS being incorrectly started as root during a previous patching cycle. No active process was involved.

    Step 4 — Attempt Manual Delete to Confirm the Error

    rm -rf /u01/app/fs1/FMW_Home 2>&1 | head -5
    rm: cannot remove '.../EBS_web_OHS4/diagnostics/logs/OHS/EBS_web/sec_audit_log': Permission denied

    This confirmed the issue was purely a file ownership/permission problem — not filesystem corruption or an NFS issue.

    Step 5 — Check Disk Space

    df -h /u01/app/fs1
Filesystem      Size  Used Avail Use% Mounted on
nfs_server:/vol  1.3T  844G  447G  66% /u01

    447GB free — sufficient to retain a backup of FMW_Home by renaming it.

    Root Cause Analysis

    The root cause was OHS (Oracle HTTP Server) being started as root on the Patch File System during a previous patching cycle in February 2026. This created log and audit files owned by root under:

    /u01/app/fs1/FMW_Home/webtier/instances/EBS_web_OHS2/auditlogs/OHS/
/u01/app/fs1/FMW_Home/webtier/instances/EBS_web_OHS2/diagnostics/logs/OHS/EBS_web/
/u01/app/fs1/FMW_Home/webtier/instances/EBS_web_OHS3/  (same structure)
/u01/app/fs1/FMW_Home/webtier/instances/EBS_web_OHS4/  (same structure)

    Since fs_clone runs as applmgr, and applmgr cannot delete files owned by root, the removeDirectory() function in txkADOPPreparePhaseSynchronize.pl failed with Permission Denied — surfaced as a fatal error.

    Why did OHS create root-owned files? If OHS start/stop scripts are executed as root or with sudo (instead of using applmgr-owned wrapper scripts), the resulting log and audit files are created with root ownership and persist on the Patch FS across patching cycles.

    Pre-Action Safety Checklist

    CheckExpectedResult
    FILE_EDITION = runrun✅ PASS
    RUN_BASE points to fs2/u01/app/fs2✅ PASS
    FMW_Home target is on fs1 (Patch FS only)fs1 only✅ PASS
    lsof returns empty (no open handles)Empty✅ PASS
    Root-owned files are stale (no active processes)Stale only✅ PASS
    Sufficient disk space for backup rename> 50GB free✅ PASS
    Production services confirmed running on fs2fs2 up✅ PASS

    Solution — Move FMW_Home as Backup

    The safest approach on production is to move (rename) FMW_Home rather than deleting it. This avoids the need for root access entirely, completes in seconds, and preserves a backup.

    Why mv works even with root-owned files: mv on the same filesystem is a purely atomic rename at the directory level. It does not touch or modify any file contents inside the directory — so applmgr can rename FMW_Home even if files inside are owned by root. This is fundamentally different from rm -rf, which must access and remove each individual file.

    Step 1 — Move FMW_Home as a Dated Backup

    mv /u01/app/fs1/FMW_Home /u01/app/fs1/FMW_Home_$(date +%d%b%Y)_bkp && echo "MOVE SUCCESSFUL"
    MOVE SUCCESSFUL

    Step 2 — Verify FMW_Home Is Gone

    ls -lrt /u01/app/fs1/

    Step 3 — Confirm You Are applmgr Before Retrying

    whoami
# Expected output: applmgr

    ⚠️ Never run adop as root. Always confirm whoami shows applmgr before executing any adop command.

    Step 4 — Retry fs_clone

    time adop phase=fs_clone

    Running fs_clone Safely on Production

    time adop phase=fs_clone on a 2-node RAC with 4 application server nodes takes several hours. Never run it in a plain SSH/PuTTY session that could disconnect. Use one of the following:

    • VNC Session (Best): Network drops have zero impact on the running process.
    • nohup: nohup adop phase=fs_clone > /tmp/fsclone_$(date +%Y%m%d_%H%M%S).log 2>&1 &
    • screen: screen -S fsclone then time adop phase=fs_clone. Detach with Ctrl+A D, reattach with screen -r fsclone.

    Successful Run — 2nd Attempt

    After applying the fix, time adop phase=fs_clone was re-executed. The adopmon output confirmed all 4 application nodes progressing through validation, port blocking, clone steps, and config clone phases without any errors.

    ADOP (C.Delta.13)
Session Id: 129
Command:    status
Node Name   Node Type  Phase        Status     Started               Finished              Elapsed
----------  ---------  -----------  ---------  --------------------  --------------------  -------
app-node1   master     FS_CLONE     COMPLETED  2026/04/11 17:07:35   2026/04/12 00:49:08   7:46:33
app-node2   slave      CONFIG_CLONE COMPLETED  2026/04/11 17:07:36   2026/04/12 01:01:55   7:47:19
app-node3   slave      CONFIG_CLONE COMPLETED  2026/04/11 17:07:36   2026/04/12 01:01:25   7:47:49
app-node4   slave      CONFIG_CLONE COMPLETED  2026/04/11 17:07:36   2026/04/12 01:02:16   7:47:40
File System Synchronization Type: Full
adop exiting with status = 0 (Success)
    Summary report for current adop session:
    Node app-node1:  - Fs_clone status: Completed successfully
    Node app-node2:  - Fs_clone status: Completed successfully
    Node app-node3:  - Fs_clone status: Completed successfully
    Node app-node4:  - Fs_clone status: Completed successfully
adop exiting with status = 0 (Success)
real    321m25.733s   (5 hours 21 minutes 25 seconds)
user     40m1.142s
sys      70m59.804s
    NodeTypeStartedFinishedElapsed
    app-node1MasterApr 11, 2026 17:07:35Apr 12, 2026 00:49:087h 46m 33s
    app-node2SlaveApr 11, 2026 17:07:36Apr 12, 2026 01:01:557h 47m 19s
    app-node3SlaveApr 11, 2026 17:07:36Apr 12, 2026 01:01:257h 47m 49s
    app-node4SlaveApr 11, 2026 17:07:36Apr 12, 2026 01:02:167h 47m 40s

    The 2nd run completed cleanly in 5 hours 21 minutes 25 seconds across all 4 application nodes. File System Synchronization Type: Full.

    Post-Resolution Cleanup

    After a successful fs_clone and full patching cycle, old FMW_Home backups can be removed. Keep the most recent backup until the next patching cycle completes, then clean up older ones as root (since they may contain root-owned files):

    ls -lrt /u01/app/fs1/FMW_Home*
du -sh /u01/app/fs1/FMW_Home*
# Remove old backups as root
sudo rm -rf /u01/app/fs1/FMW_Home_<old_date>_bkp

    Prevention — Avoiding Recurrence

    • Never start OHS as root. Always use applmgr-owned wrapper scripts. Never use sudo or root to run adohs.sh or adadminsrvctl.sh.
    • Post-patching ownership check. After every adop finalize/cutover, run: find /u01/app/fs1 ! -user applmgr -ls 2>/dev/null | head -20
    • Pre-fs_clone health check. Verify no lingering adop sessions, confirm Run FS services are healthy, check disk space, and verify no root-owned files under fs1/FMW_Home before starting.

    Summary

    ItemDetail
    Phaseadop phase=fs_clone
    Failing Functionmain::removeDirectory inside removeFMWHome()
    Clone Stageclone_status_from_db = 6 (REMOVE-1012-ORACLE-HOME)
    Root CauseOHS started as root in a previous cycle — stale root-owned OHS log/audit files blocking applmgr deletion
    Production ImpactNone — fs1 is Patch FS, production ran on fs2 throughout
    Fix Appliedmv FMW_Home to dated backup as applmgr — atomic rename, no root needed, completed in seconds. rm -rf was NOT used.
    1st Run Duration~1h 34m before fatal error (Apr 11 17:07 → 18:41)
    2nd Run Duration5h 21m 25s — completed successfully (Apr 11 19:45 → Apr 12 00:49)
    Total Session Elapsed7h 46m 33s (including failed run, diagnosis, fix, and retry)
    Final Statusadop exiting with status = 0 (Success) — all 4 app nodes completed ✅
    PreventionNever start OHS as root; add post-patching ownership check to runbook
    Oracle SupportSearch “ADOP fs_clone Failed to delete FMW_Home” on My Oracle Support

    Happy Debugging! All server-specific details have been anonymised. The diagnostic commands and fix are generic and applicable to any Oracle EBS 12.2.x environment. If this helped you, feel free to share with the community.

  • When ADOP Remembers Too Much: Fixing Patch Failures Caused by Stale Metadata in Oracle EBS

    During an Oracle E-Business Suite ADOP patching cycle in a multi-node environment, the apply phase failed on one node while completing successfully on others. Despite retries — including downtime mode — the issue persisted, pointing to a deeper inconsistency within the patching framework.

    Symptoms Observed

    • ADOP session status: FAILED
    • Patch applied successfully on some nodes, failed on admin node
    • Repeated failures even with restart=no, abandon=yes, and downtime mode
    • No immediate actionable error from standard logs

    Timeline of Events

    T0 -- Patch execution initiated (ADOP apply phase)
T1 -- Failure observed on admin node
T2 -- Retry using downtime mode -- Failure persists
T3 -- ADOP session review shows inconsistent state
T4 -- Internal metadata tables analyzed
T5 -- Cleanup performed (tables + restart directory)
T6 -- Patch re-executed -- Success across all nodes

    Investigation

    Step 1: Check ADOP Session State

    Query the ADOP session status to understand the current state across all nodes:

    -- Check current ADOP session status
SELECT session_id, node_name, phase, status,
       start_date, end_date
FROM applsys.ad_adop_sessions
ORDER BY start_date DESC;

-- Check apply phase status per node
SELECT s.session_id, n.node_name, p.phase_code,
       p.status, p.start_date, p.end_date
FROM applsys.ad_adop_sessions s,
     applsys.ad_adop_session_phases p,
     applsys.fnd_nodes n
WHERE s.session_id = p.session_id
AND p.node_id = n.node_id
ORDER BY p.start_date DESC;

    The existing session showed status FAILED with the apply phase partially completed — a clear indicator of inconsistent execution state across nodes.

    Step 2: Check adalldefaults.txt

    Reviewed the defaults file for any relevant configuration:

    cat $APPL_TOP/admin/$TWO_TASK/adalldefaults.txt | grep -i missing
-- Key parameter found:
-- MISSING_TRANSLATED_VERSION = No

    Modifying and retrying with this parameter had no impact, confirming the issue was not translation-related.

    Step 3: Check Install Processes Table

    -- Check for stale entries in FND_INSTALL_PROCESSES
SELECT COUNT(*) FROM applsys.fnd_install_processes;

-- View stale entries in detail
SELECT process_status, process_name, last_update_date
FROM applsys.fnd_install_processes
ORDER BY last_update_date DESC;

-- Check AD_DEFERRED_JOBS
SELECT COUNT(*) FROM applsys.ad_deferred_jobs;
SELECT * FROM applsys.ad_deferred_jobs;

    Observation: FND_INSTALL_PROCESSES contained stale entries from the failed session. AD_DEFERRED_JOBS was empty.

    Root Cause

    The failure was caused by stale and inconsistent ADOP metadata tables — specifically APPLSYS.FND_INSTALL_PROCESSES and APPLSYS.AD_DEFERRED_JOBS. ADOP internally relies on these tables to track patch progress checkpoints, deferred job execution, and restart state management. When these tables retain entries from failed or incomplete sessions, ADOP assumes an incorrect execution state, leading to patch reconciliation failure, apply phase breakdown, and node-level inconsistencies.

    Resolution Steps

    Step 1: Backup Critical Tables

    -- Always backup before any cleanup
CREATE TABLE applsys.fnd_install_processes_bak AS
SELECT * FROM applsys.fnd_install_processes;

CREATE TABLE applsys.ad_deferred_jobs_bak AS
SELECT * FROM applsys.ad_deferred_jobs;

-- Verify backups
SELECT COUNT(*) FROM applsys.fnd_install_processes_bak;
SELECT COUNT(*) FROM applsys.ad_deferred_jobs_bak;

    Step 2: Drop Stale Metadata Tables

    Dropping these tables forces ADOP to rebuild clean metadata during the next run:

    DROP TABLE applsys.fnd_install_processes;
DROP TABLE applsys.ad_deferred_jobs;

    Step 3: Reset the Restart Directory

    The restart directory can silently preserve failure states. Back it up and create a fresh one:

    cd $APPL_TOP/admin/$TWO_TASK

-- Backup existing restart directory
mv restart restart_bkp_$(date +%Y%m%d)

-- Create fresh restart directory
mkdir restart

-- Verify
ls -la | grep restart

    Step 4: Re-run the Patch

    adop phase=apply \
     patches=<patch_id> \
     restart=no \
     abandon=yes \
     apply_mode=downtime

    The patch completed successfully across all nodes after the metadata cleanup.

    Before vs After

    ComponentBefore FixAfter Fix
    ADOP SessionFailedSuccessful
    Node ConsistencyPartialFull
    Restart BehaviorStuckClean
    Patch ExecutionIncompleteCompleted

    Key Takeaways

    • ADOP is state-driven — even when logs appear clean, internal metadata drives execution decisions
    • Partial success is a clue — if some nodes succeed and one fails, focus on local metadata, not the patch itself
    • The restart directory matters — it can silently preserve failure states and must be validated before retrying
    • Downtime mode is not a fix-all — even in downtime, ADOP still reads metadata tables; corruption persists unless cleaned
    • Always backup before cleanup — never drop tables without creating a backup first

    When NOT to Use This Approach

    Avoid applying this fix if the issue is caused by missing database patches (ETCC warnings), file system or permission issues, incorrect patch sequencing, or environment misconfiguration. Always validate the root cause before performing any metadata cleanup.

    This scenario highlights a subtle but critical behavior in ADOP — sometimes patch failures are not caused by the patch itself, but by what the system remembers about past attempts. By resetting stale metadata, we allow ADOP to re-evaluate the environment cleanly, leading to successful execution.

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.