Category: Oracle Database

  • Upgrading Oracle Database 19c to Oracle AI Database 26ai with AutoUpgrade: A Home-Lab Run (and Why 19.3 Would Not Install on Oracle Linux 9)

    Alhamdulillah. This is a write-up of a home-lab exercise: upgrading an Oracle Database 19c container database to Oracle AI Database 26ai (23.26.1) with AutoUpgrade. The upgrade itself was the easy part. Getting a 19c source database onto the lab took longer, because the base 19.3 release would not link on Oracle Linux 9. I have kept that part in, because it is the kind of thing that costs an evening if you have not seen it before.

    The idea came from a community post on the same upgrade path (Oracle 19c to 26ai Database Upgrade). Everything below is from my own lab run. Hostnames and IP addresses are masked.

    Lab environment

    ItemValue
    HostWindows desktop, VirtualBox, 64 GB RAM
    Upgrade VMlabhost19, Oracle Linux 8.10, 4 vCPU, 12 GB RAM, 100 GB disk
    SourceOracle Database 19c EE 19.3 (base release, no RU), CDB19 with PDB19
    TargetOracle AI Database 26ai EE 23.26.1 (public download)
    ToolAutoUpgrade 26.7.2 (build 261111)
    Source home/u01/app/oracle/product/19.0.0/dbhome_1
    Target home/u01/app/oracle/product/26ai/dbhome_1

    Only public downloads were used: the 19.3 and 23.26.1 zips from oracle.com and the AutoUpgrade jar. No My Oracle Support account was involved, which matters for the first problem.

    Part 1: Why 19.3 would not install on Oracle Linux 9

    My first plan was to install a 19c home next to the existing 26ai home on an Oracle Linux 9.8 VM. The software-only install failed at the relink stage every time:

    [FATAL] Error in invoking target 'libasmclntsh19.ohso libasmperl19.ohso client_sharedlib' of makefile
    '/u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/ins_rdbms.mk'.

    The installer message is generic. The actual linker error is in $ORACLE_HOME/install/make.log, so that is the first file to check.

    Attempt 1: gcc was not installed

    $ gcc --version | head -1
    bash: gcc: command not found...

    Neither the 26ai nor the 19c preinstall RPM pulled in gcc on this OL9 VM. I installed it, but the relink still failed.

    dnf -y install gcc gcc-c++ make glibc-devel elfutils-libelf-devel ksh

    Attempt 2: libpthread_nonshared.a

    $ grep -n -i -E "error|undefined|cannot find" $ORACLE_HOME/install/make.log
    136:/usr/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a
    ...
    genclntsh: Failed to link libclntshcore.so.19.1

    The 19.3 home ships its own linker stubs. The libpthread.so stub is a linker script that still points at a file Oracle Linux 9 does not have:

    $ cat $ORACLE_HOME/lib/stubs/libpthread.so
    /* GNU ld script
    Use the shared library, but some functions are only in
    the static library, so try that secondarily. */
    OUTPUT_FORMAT(elf64-x86-64)
    GROUP ( libpthread.so.0 /usr/lib64/libpthread_nonshared.a )

    OL9 uses glibc 2.34, where libpthread was merged into libc and libpthread_nonshared.a is gone. The base 19.3 stubs were built for an older glibc.

    Attempts 3 to 5: each stub fix moved the failure

    • Moving all stubs aside gave cannot find -lgcc_s. The link needs the other stubs.
    • Restoring the stubs and removing only libpthread.so gave cannot find -lclntshcore. That one was self-inflicted: an earlier failed genclntsh run had already deleted libclntshcore.so, so the home was damaged.
    • On a fresh unzip with only the libpthread.so stub removed, the client libraries linked, and the failure moved to the Pro*C link with undefined pthread_* symbols.
    /usr/bin/ld: .../lib//libclntsh.so: undefined reference to `pthread_sigmask'
    /usr/bin/ld: .../lib//libclntshcore.so: undefined reference to `pthread_mutexattr_settype'
    /usr/bin/ld: .../lib//libclntsh.so: undefined reference to `pthread_join'
    make[1]: *** [.../precomp/lib/ins_precomp.mk:118: .../precomp/lib/proc] Error 1

    At that point I stopped patching stubs. Oracle’s supported route for 19c on Oracle Linux 9 is a newer Release Update applied at install time (runInstaller -applyRU), and RUs need a support account. Even if a hand-edited 19.3 home had linked in the end, it would not have been a sound base for an upgrade test. I built a separate Oracle Linux 8.10 VM instead, where 19.3 installs without workarounds.

    One lesson from this part: after a relink failure, do not retry in the same home. Wipe it completely, including hidden folders such as .patch_storage, and unzip again.

    Part 2: Building the 19c source on Oracle Linux 8.10

    OS packages and directories, as root:

    dnf -y install oracle-database-preinstall-19c gcc gcc-c++ make glibc-devel elfutils-libelf-devel ksh unzip
    mkdir -p /u01/app/oracle/product/19.0.0/dbhome_1 /u01/app/oracle/product/26ai/dbhome_1 /u01/stage /u01/app/oracle/fra
    chown -R oracle:oinstall /u01
    chmod -R 775 /u01

    I kept one environment file per home so the profile never points at the wrong one. A mistake worth mentioning: the first time, the env file did not exist, $ORACLE_HOME was empty, cd $ORACLE_HOME went to the oracle user’s home directory and the 19c zip started extracting there. Check pwd before every unzip.

    cat > ~/cdb19.env <<'EOF'
    export ORACLE_BASE=/u01/app/oracle
    export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
    export ORACLE_SID=CDB19
    export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:/usr/sbin:/usr/bin:/bin
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib
    EOF
    . ~/cdb19.env
    cd $ORACLE_HOME && pwd

    Software-only install. CV_ASSUME_DISTID=OL7 gets the base 19.3 installer past its OS version check on OL8:

    unzip -q /u01/stage/LINUX.X64_193000_db_home.zip
    export CV_ASSUME_DISTID=OL7
    ./runInstaller -silent -ignorePrereqFailure -waitforcompletion \
    oracle.install.option=INSTALL_DB_SWONLY \
    UNIX_GROUP_NAME=oinstall \
    INVENTORY_LOCATION=/u01/app/oraInventory \
    ORACLE_HOME=$ORACLE_HOME \
    ORACLE_BASE=$ORACLE_BASE \
    oracle.install.db.InstallEdition=EE \
    oracle.install.db.OSDBA_GROUP=dba \
    oracle.install.db.OSOPER_GROUP=oper \
    oracle.install.db.OSBACKUPDBA_GROUP=backupdba \
    oracle.install.db.OSDGDBA_GROUP=dgdba \
    oracle.install.db.OSKMDBA_GROUP=kmdba \
    oracle.install.db.OSRACDBA_GROUP=racdba \
    DECLINE_SECURITY_UPDATES=true
    [WARNING] [INS-13014] Target environment does not meet some optional requirements.
    ...
    Successfully Setup Software with warning(s).

    No linking errors on OL8. After orainstRoot.sh and root.sh, I created the listener and the database. ARCHIVELOG mode and a recovery area are needed because AutoUpgrade creates a guaranteed restore point before it starts.

    netca -silent -responsefile $ORACLE_HOME/assistants/netca/netca.rsp
    dbca -silent -createDatabase -templateName General_Purpose.dbc \
    -gdbName CDB19 -sid CDB19 -createAsContainerDatabase true \
    -numberOfPDBs 1 -pdbName PDB19 \
    -sysPassword <password> -systemPassword <password> -pdbAdminPassword <password> \
    -datafileDestination /u01/app/oracle/oradata -characterSet AL32UTF8 \
    -totalMemory 3072 -enableArchive true \
    -recoveryAreaDestination /u01/app/oracle/fra -recoveryAreaSize 20480 \
    -emConfiguration NONE

    Before the upgrade I saved the PDB open state and created a small test table to compare after the upgrade:

    SET LINESIZE 150 PAGESIZE 100
    COL name FORMAT a10
    COL log_mode FORMAT a12
    COL open_mode FORMAT a12
    COL version FORMAT a12
    SELECT d.name, d.log_mode, d.open_mode, i.version FROM v$database d, v$instance i;
    ALTER PLUGGABLE DATABASE ALL SAVE STATE;
    ALTER SESSION SET CONTAINER=PDB19;
    CREATE USER lab IDENTIFIED BY <password> QUOTA UNLIMITED ON users;
    GRANT CREATE SESSION, CREATE TABLE TO lab;
    CREATE TABLE lab.upg_test AS SELECT * FROM dba_objects;
    SELECT COUNT(*) FROM lab.upg_test;
    NAME LOG_MODE OPEN_MODE VERSION
    ---------- ------------ ------------ ------------
    CDB19 ARCHIVELOG READ WRITE 19.0.0.0.0
    COUNT(*)
    ----------
    72356

    Saving the PDB state matters here. The post that inspired this run hit “PDB is MOUNTED and will not be processed” in AutoUpgrade. With the state saved, both PDBs were picked up.

    I took a VirtualBox snapshot at this point, so the upgrade could be repeated from a clean 19c source.

    Part 3: Installing the 26ai home

    # as root
    dnf -y install oracle-ai-database-preinstall-26ai
    # as oracle
    . ~/26ai.env
    cd $ORACLE_HOME && pwd
    unzip -oq /u01/stage/LINUX.X64_2326100_db_home.zip
    ./runInstaller -ignorePrereq -waitforcompletion -silent \
    -responseFile $ORACLE_HOME/install/response/db_install.rsp \
    installOption=INSTALL_DB_SWONLY \
    UNIX_GROUP_NAME=oinstall \
    INVENTORY_LOCATION=/u01/app/oraInventory \
    ORACLE_HOME=$ORACLE_HOME \
    ORACLE_BASE=$ORACLE_BASE \
    installEdition=EE \
    OSDBA=dba OSOPER=oper OSBACKUPDBA=backupdba \
    OSDGDBA=dgdba OSKMDBA=kmdba OSRACDBA=racdba

    Note that the 26ai installer uses shorter parameter names than 19c: installOption instead of oracle.install.option, installEdition, and OSDBA rather than oracle.install.db.OSDBA_GROUP. On OL8 the preinstall package also pulled in compat-openssl10. The install finished with the same optional-prerequisite warning (INS-13014), followed by root.sh.

    Part 4: AutoUpgrade analyze

    The VM had internet access, so I downloaded the latest jar instead of using the copy bundled in the target home, and ran it with the JDK from the 26ai home:

    mkdir -p /u01/app/oracle/autoupgrade/logs
    cd /u01/app/oracle/autoupgrade
    curl -sSL -o autoupgrade.jar https://download.oracle.com/otn-pub/otn_software/autoupgrade.jar
    JAVA=/u01/app/oracle/product/26ai/dbhome_1/jdk/bin/java
    $JAVA -jar autoupgrade.jar -version
    build.version 26.7.2.261111
    build.date 2026/11/11 04:27:02 +0000
    build.supported_target_versions 12.2,18,19,21,23,26
    build.type production

    Configuration file (autoupg.cfg):

    global.global_log_dir=/u01/app/oracle/autoupgrade/logs
    upg1.sid=CDB19
    upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_1
    upg1.target_home=/u01/app/oracle/product/26ai/dbhome_1
    upg1.restoration=YES
    upg1.timezone_upg=YES
    upg1.log_dir=/u01/app/oracle/autoupgrade/logs/CDB19
    • restoration=YES: AutoUpgrade creates a guaranteed restore point before the upgrade.
    • timezone_upg=YES: the time zone file is upgraded as part of the post-upgrade fixups.
    $JAVA -jar autoupgrade.jar -config autoupg.cfg -mode analyze -noconsole
    1 CDB(s) plus 2 PDB(s) will be analyzed
    Job 100 database cdb19
    ...
    Job 100 completed
    ------------------- Final Summary --------------------
    Number of databases [ 1 ]
    Jobs finished [1]
    Jobs failed [0]
    Checks Report
    +------------+------+-----------+---------+-------+-------+
    | Job Number | INFO | RECOMMEND | WARNING | ERROR | Total |
    +------------+------+-----------+---------+-------+-------+
    | 100 | 26 | 16 | 12 | 0 | 54 |
    +------------+------+-----------+---------+-------+-------+

    Zero errors. The 12 warnings were four distinct checks, each reported for CDB$ROOT, PDB$SEED and PDB19:

    WarningStageWhat I did
    Convert traditional audit to unified audit policiesPRECHECKSNothing in the lab. On a client system this needs planning before the upgrade, because traditional auditing does not carry on by default.
    Run outside the automatic maintenance windowsPRECHECKSNothing. The run started in the afternoon and finished well before the default window.
    Upgrade the time zone file (DBMS_DST)POSTCHECKSAUTOFIXUP, covered by timezone_upg=YES
    Recompile objects with timestamp mismatchPOSTCHECKSAUTOFIXUP

    Part 5: Deploy

    I ran deploy under nohup so that a dropped PuTTY session would not kill the job:

    nohup $JAVA -jar autoupgrade.jar -config autoupg.cfg -mode deploy -noconsole > deploy.out 2>&1 &
    tail -8 /u01/app/oracle/autoupgrade/deploy.out

    Progress on this VM:

    TimeStageMessage
    16:49PRECHECKSJob 101 started, 1 CDB plus 2 PDBs
    ~16:52PREFIXUPSExecuting fixups (restore point already created)
    17:00DBUPGRADE0% Upgraded CDB$ROOT
    17:13DBUPGRADE41% Upgraded CDB$ROOT
    18:17POSTFIXUPSExecuting fixups (time zone upgrade, recompile)
    ~18:26POSTUPGRADECreating final SPFILE, job completed

    About 1 hour 37 minutes in total. During POSTFIXUPS the alert log repeatedly reported slow online redo log writes. On a VirtualBox disk under upgrade load that is expected, and it did not affect the result.

    Job 101 completed
    ------------------- Final Summary --------------------
    Number of databases [ 1 ]
    Jobs finished [1]
    Jobs failed [0]
    Jobs restored [0]
    Jobs pending [0]
    ---- Drop GRP at your convenience once you consider it is no longer needed ----
    Drop GRP from CDB19: drop restore point AUTOUPGRADE_9212_CDB19193000

    Part 6: Post-upgrade verification

    AutoUpgrade had already updated /etc/oratab to the 26ai home. I pointed my env file at the new home and checked the database:

    sed -i 's#product/19.0.0/dbhome_1#product/26ai/dbhome_1#' ~/cdb19.env
    . ~/cdb19.env
    grep CDB19 /etc/oratab
    CDB19:/u01/app/oracle/product/26ai/dbhome_1:N
    SET LINESIZE 160 PAGESIZE 100
    COL version_full FORMAT a15
    COL comp_id FORMAT a12
    COL status FORMAT a10
    COL name FORMAT a40
    SELECT version_full FROM v$instance;
    SHOW PDBS
    SELECT comp_id, NVL(version_full,'-') version_full, status FROM dba_registry ORDER BY 1;
    SELECT version FROM v$timezone_file;
    SELECT name FROM v$restore_point;
    SHOW PARAMETER compatible
    ALTER SESSION SET CONTAINER=PDB19;
    SELECT COUNT(*) FROM lab.upg_test;
    SELECT COUNT(*) invalid_objs FROM dba_objects WHERE status='INVALID';
    VERSION_FULL
    ---------------
    23.26.1.0.0
    CON_ID CON_NAME OPEN MODE RESTRICTED
    ---------- ------------------------------ ---------- ----------
    2 PDB$SEED READ ONLY NO
    3 PDB19 READ WRITE NO
    COMP_ID VERSION_FULL STATUS
    ------------ --------------- ----------
    APS 23.26.1.0.0 VALID
    CATALOG 23.26.1.0.0 VALID
    CATJAVA 23.26.1.0.0 VALID
    CATPROC 23.26.1.0.0 VALID
    CONTEXT 23.26.1.0.0 VALID
    DV 23.26.1.0.0 VALID
    JAVAVM 23.26.1.0.0 VALID
    OLS 23.26.1.0.0 VALID
    OWM 23.26.1.0.0 VALID
    RAC 23.26.1.0.0 OPTION OFF
    SDO 23.26.1.0.0 VALID
    XDB 23.26.1.0.0 VALID
    XML 23.26.1.0.0 VALID
    XOQ 23.26.1.0.0 VALID
    VERSION
    ----------
    43
    NAME
    ----------------------------------------
    AUTOUPGRADE_9212_CDB19193000
    NAME TYPE VALUE
    ------------------------------------ ----------- ------------------------------
    compatible string 19.0.0
    COUNT(*)
    ----------
    72356
    INVALID_OBJS
    ------------
    0
    • Version 23.26.1.0.0, PDB19 open READ WRITE.
    • All components VALID. RAC shows OPTION OFF, which is expected on a single instance.
    • Time zone file upgraded to version 43.
    • The test table still has 72,356 rows, and PDB19 has no invalid objects.
    • COMPATIBLE is still 19.0.0, so the restore point could still be used to go back.

    Part 7: Listener, restore point and COMPATIBLE

    The listener was still running from the 19c home. I stopped it, copied the network files to the 26ai home, updated the paths and started it from the new home:

    /u01/app/oracle/product/19.0.0/dbhome_1/bin/lsnrctl stop
    cp /u01/app/oracle/product/19.0.0/dbhome_1/network/admin/{listener.ora,sqlnet.ora,tnsnames.ora} $ORACLE_HOME/network/admin/
    sed -i 's#product/19.0.0/dbhome_1#product/26ai/dbhome_1#g' $ORACLE_HOME/network/admin/*.ora
    lsnrctl start

    The listener showed “supports no services” at first. local_listener was set to the alias LISTENER_CDB19, which dbca had written into tnsnames.ora, so copying that file to the new home mattered. After ALTER SYSTEM REGISTER the services appeared:

    Service "CDB19" has 1 instance(s).
    Instance "CDB19", status READY, has 1 handler(s) for this service...
    Service "CDB19XDB" has 1 instance(s).
    Instance "CDB19", status READY, has 1 handler(s) for this service...
    Service "pdb19" has 1 instance(s).
    Instance "CDB19", status READY, has 1 handler(s) for this service...

    With the checks done, I dropped the restore point so it would stop holding flashback logs in the recovery area:

    DROP RESTORE POINT AUTOUPGRADE_9212_CDB19193000;
    SELECT COUNT(*) FROM v$restore_point;

    I left COMPATIBLE at 19.0.0. Raising it enables the new release’s features, but it cannot be lowered again, which rules out a downgrade. In a real project that decision comes after application testing. In the lab I took a VirtualBox snapshot first, so I can raise it later and still roll back the VM.

    Lessons learned

    • Check the certification of the exact release on the exact OS. “19c on OL9” and “base 19.3 on OL9” are not the same thing.
    • When runInstaller reports a relink FATAL, read $ORACLE_HOME/install/make.log. The installer log only repeats the generic message.
    • After a failed relink, start from a fresh unzip. Retrying in a damaged home produces new errors that have nothing to do with the original cause.
    • Put a time limit on workarounds. After a few rounds of stub fixes, moving to a certified platform was faster and gave a cleaner result.
    • Save the PDB open state before AutoUpgrade so every PDB is processed.
    • Run deploy under nohup (or screen/tmux) and follow deploy.out and status.log.
    • Moving the listener is not part of the AutoUpgrade job. Copy tnsnames.ora too if local_listener uses an alias.
    • Keep the guaranteed restore point until verification is done, and treat COMPATIBLE as a separate, later decision.

    ما شاء الله, a clean result in the end: 0 errors in analyze, 0 failed jobs in deploy, and the test data intact on 26ai. Next in this lab is Oracle APEX on the 26ai database.

  • Oracle 19c October 2026 RU (19.32 → 19.33) on Grid Infrastructure: Recovering from “No space left on device” Mid-opatchauto

    Alhamdulillah, the October 2026 CPU cycle for one of our 19c development databases is complete. Most of it was routine: GI RU, DB RU and OJVM on both nodes, then datapatch and recompilation. The one surprise came on node 2, where opatchauto failed halfway through the DB home with No space left on device. The space had been eaten by more than two years of rotated ASM network listener logs.

    This post walks through the full runbook with the real outputs, the failure, the cleanup, and the opatchauto resume that finished it. Hostnames, database and instance names, Oracle home paths and session IDs have been replaced with generic placeholders.

    Environment

    ClusterOracle Grid Infrastructure 19c, two nodes (node1, node2)
    DatabaseOracle Database 19c Standard Edition 2, non-CDB, managed by clusterware
    From → To19.32.0.0.260721 → 19.33.0.0.261020
    Patch bundleCombo 40121665 = GI RU 39922757 + OJVM RU 39803650
    OPatch12.2.0.1.52

    Placeholders used throughout: $GRID_HOME = Grid Infrastructure home, $ORACLE_HOME = database home, $PATCH_TOP = directory where the combo patch was unzipped, <DB_NAME> / <INSTANCE> = database and instance names.

    The GI RU 39922757 is a bundle of sub-patches. In this run, opatchauto applied 39915792 (Database RU), 39922783 (OCW RU) and 39922802 to the GI home, and only 39915792 and 39922783 to the DB home. Sub-patches 39107825 and 39107855 were skipped on both homes as “not required” or “not applicable to this target type”. That’s expected behaviour, not an error.

    Runbook sequence

    The same patch sequence ran on each node. Node 2 started while node 1 was still on its OJVM and datapatch steps, so the two nodes overlapped.

    1. Pre-patch baseline (database state, invalid objects, SQL patch registry, components)
    2. Client confirms the application is stopped, then shut down the database from node 1
    3. Stop clusterware on both nodes and take TAR backups of the GI and DB homes
    4. Node 1: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply) → start the database → datapatch → utlrp
    5. Node 2, in parallel: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply)
    6. Post-check and compare with the baseline

    Step 1: Pre-patch baseline

    SET LINES 230 PAGES 100
    COL object_name FOR a40
    COL object_type FOR a20
    COL owner       FOR a20
    COL last_ddl    FOR a22
    SELECT object_name, object_type, owner,
           TO_CHAR(last_ddl_time,'DD-MON-YYYY HH24:MI:SS') last_ddl, status
    FROM   dba_objects
    WHERE  status = 'INVALID'
    ORDER  BY owner;
    
    SET LINES 300
    COL action_time FOR a30
    COL action      FOR a15
    COL status      FOR a15
    COL description FOR a60
    SELECT patch_id, action, status, action_time, description
    FROM   dba_registry_sqlpatch
    ORDER  BY action_time;
    
    COL comp_id   FOR a12
    COL comp_name FOR a44 HEA 'Component'
    COL version   FOR a17 HEA 'Version'
    COL status    FOR a17 HEA 'Status'
    SELECT comp_id, comp_name, version, status FROM dba_registry;

    Baseline findings:

    • The database was READ WRITE / PRIMARY.
    • 29 invalid objects, all in two application schemas, most with a LAST_DDL_TIME from 2015 to 2023. Nothing in SYS or SYSTEM. Recording these before you patch saves an argument later about whether patching caused them.
    • The last registry entries were the July cycle: OJVM 39222882 and DB RU 39472050 (19.32), both SUCCESS.
    • All components were VALID or OPTION OFF.

    Step 2: Application stop and database shutdown

    Nothing was touched until the client confirmed the application was stopped. Then, from node 1:

    $ . ~/<DB_NAME>.env
    $ sqlplus / as sysdba
    SQL> shut immediate;
    Database closed.
    Database dismounted.
    ORACLE instance shut down.

    Step 3: Stop clusterware and back up the homes

    On each node, as root from $GRID_HOME/bin:

    # ./crsctl check crs
    # ./crsctl stop has
    ...
    CRS-4133: Oracle High Availability Services has been stopped.

    With the stack down, a TAR backup was taken of both the GI home and the DB home on each node. That’s your fastest way back if a home gets damaged mid-patch. An example of the command form (adjust the target location to wherever you have space):

    # tar -czpf /backup/grid_home_$(hostname -s)_$(date +%Y%m%d).tar.gz -C $GRID_HOME .
    # tar -czpf /backup/db_home_$(hostname -s)_$(date +%Y%m%d).tar.gz   -C $ORACLE_HOME .

    The stack was then started again with crsctl start has ahead of opatchauto. Right after the start, crsctl check crs showed the usual intermediate states for a few minutes:

    # ./crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager
    
    # ./crsctl check crs        (a few minutes later)
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online

    CSS and EVM came up first and CRSD came up last. Give it time and keep checking before you assume something is broken.

    Step 4: Node 1 GI home

    # $GRID_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $GRID_HOME
    
    Performing prepatch operations on CRS - bringing down CRS service on home $GRID_HOME
    CRS service brought down successfully on home $GRID_HOME
    Start applying binary patch on home $GRID_HOME
    Binary patch applied successfully on home $GRID_HOME
    Successfully executed rootadd_rdbms.sh on home $GRID_HOME
    CRS service started successfully on home $GRID_HOME
    
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    Patch: $PATCH_TOP/39922757/39922802
    
    Time taken to complete the session 22 minutes, 51 seconds

    Step 5: Node 1 DB home

    # $ORACLE_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $ORACLE_HOME
    
    Binary patch applied successfully on home $ORACLE_HOME
    ...
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    
    [WARNING] The database instance '<INSTANCE>' from '$ORACLE_HOME', in host 'node1'
    is not running. SQL changes, if any, will not be applied.
    
    Time taken to complete the session 14 minutes, 55 seconds

    The warning is expected here. The database was deliberately left down because OJVM still had to go on, so the SQL changes are applied later by a manual datapatch run.

    Step 6: Node 1 OJVM, plus a small environment gotcha

    The first attempt as oracle returned bash: opatch: command not found. The env file and export PATH lines had been pasted in together with sudo su - oracle, so they never ran in the shell that was actually used. ORACLE_HOME was empty and $ORACLE_HOME/OPatch expanded to /OPatch. The fix was to source the env file again in the current shell and check it:

    $ . ~/<DB_NAME>.env
    $ echo $ORACLE_HOME          # must not be empty
    $ export PATH=$ORACLE_HOME/OPatch:$PATH
    $ which opatch

    Then a conflict check, followed by the apply:

    $ cd $PATCH_TOP/39803650
    $ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    $ opatch apply
    
    OPatch continues with these patches:   39803650
    All checks passed.
    Applying interim patch '39803650' to OH '$ORACLE_HOME'
    Patching component oracle.javavm.server, 19.0.0.0.0...
    Patching component oracle.javavm.server.core, 19.0.0.0.0...
    Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...
    Patching component oracle.rdbms, 19.0.0.0.0...
    Patching component oracle.javavm.client, 19.0.0.0.0...
    Patch 39803650 successfully applied.
    Sub-set patch [39222882] has become inactive due to the application of a super-set patch [39803650].
    Please refer to Doc ID 2161861.1 for any possible further required actions.
    OPatch succeeded.
    
    $ opatch lsinventory | grep 39803650
    Patch  39803650     : applied on Tue Nov 10 21:10:50 2026
    Patch description:  "OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)"

    The “sub-set patch has become inactive” line is OPatch noting that the 19.32 OJVM is superseded by 19.33. datapatch handles the SQL-side rollback of 39222882 in the next step.

    Step 7: Node 1 startup, datapatch and utlrp

    $ sqlplus / as sysdba
    SQL*Plus: Release 19.0.0.0.0 - Production
    Version 19.33.0.0.0
    Connected to an idle instance.
    SQL> startup;
    Database mounted.
    Database opened.
    SQL> exit
    
    $ cd $ORACLE_HOME/OPatch
    $ ./datapatch -verbose
    
    Current state of release update SQL patches:
      Binary registry:
        19.33.0.0.0 Release_Update 261009085306: Installed
      SQL registry:
        Applied 19.32.0.0.0 Release_Update 260705220710 successfully on 31-AUG-26
    
    Installation queue:
      The following interim patches will be rolled back:
        39222882 (OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882))
      Patch 39915792 (Database Release Update : 19.33.0.0.261020 (39915792)):
        Apply from 19.32.0.0.0 Release_Update 260705220710 to 19.33.0.0.0 Release_Update 261009085306
      The following interim patches will be applied:
        39803650 (OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650))
    
    Patch 39222882 rollback: SUCCESS  (no errors)
    Patch 39915792 apply: SUCCESS     (no errors)
    Patch 39803650 apply: SUCCESS     (no errors)
    SQL Patching tool complete

    datapatch needs the database open, so it runs once, from one node, after startup. Here that was node 1, with the 19.33 binaries (DB RU and OJVM) already in place while node 2 was still being patched. It doesn’t need to be repeated on node 2. Invalid objects were then recompiled from the same node:

    $ cd $ORACLE_HOME/rdbms/admin
    $ sqlplus / as sysdba
    SQL> @utlrp.sql

    Step 8: Node 2 GI home (in parallel)

    Node 2’s GI patch started right after node 1’s DB home finished, and ran alongside node 1’s OJVM and datapatch work. The command was the same as on node 1. It completed successfully in 27 minutes 57 seconds with the same three sub-patches applied.

    Step 9: Node 2 DB home fails with “No space left on device”

    Start applying binary patch on home $ORACLE_HOME
    Failed while applying binary patches on home $ORACLE_HOME
    
    ==Following patches FAILED in apply:
    Patch: $PATCH_TOP/39922757/39915792
    Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException:
    ApplySession failed in system modification phase... 'java.io.IOException: No space left on device'
    
    Patch: $PATCH_TOP/39922757/39922783
    Reason: ... 'java.io.IOException: No space left on device'
    
    After fixing the cause of failure Run opatchauto resume
    
    OPATCHAUTO-68061: The orchestration engine failed.
    OPatchAuto failed.
     opatchauto failed with error code 42

    The filesystem holding both homes and ORACLE_BASE was at 97%:

    # df -h /u01
    Filesystem    Size  Used Avail Use% Mounted on
    <oracle_fs>   130G  126G  4.5G  97% /u01

    Node 1 had patched without any trouble, so free space on node 2 hadn’t been checked separately. That’s the first lesson.

    Finding what was using the space

    Drilling down with du -sh * led straight into the listener ADR:

    $ORACLE_BASE                                  35G
      diag                                        26G
        tnslsnr                                   20G
          <node2>/asmnet1lsnr_asm                 15G
            alert                                9.4G
            trace                                5.3G
          <node2>/listener_scan1                3.2G

    Inside asmnet1lsnr_asm there were 32 rotated XML alert files of about 300 MB each (log_1.xml to log_32.xml) and 32 rotated text logs of about 175 MB each (asmnet1lsnr_asm_1.log to _32.log). The oldest went back to March 2024. Nothing had ever purged them.

    Cleanup

    Only the rotated, numbered files were removed. The current log.xml and asmnet1lsnr_asm.log that the listener was writing to were left in place.

    # cd $ORACLE_BASE/diag/tnslsnr/<node2>/asmnet1lsnr_asm/alert
    # rm -f log_*.xml
    # df -h /u01
    <oracle_fs>   130G  117G   14G  90% /u01
    
    # cd ../trace
    # rm -f asmnet1lsnr_asm_*.log
    # df -h /u01
    <oracle_fs>   130G  111G   20G  86% /u01

    That took free space from 4.5 GB to 20 GB. A cleaner and more repeatable option is to purge through ADRCI rather than rm. This is a recommendation, not what was run in this window:

    $ adrci
    adrci> show homes
    adrci> set home diag/tnslsnr/<node2>/asmnet1lsnr_asm
    adrci> purge -age 43200 -type alert     # older than 30 days (minutes)
    adrci> purge -age 43200 -type trace

    Rotated listener_scan1 logs (about 2.9 GB, back to 2024) are still there and are on the housekeeping list.

    opatchauto resume

    Running opatchauto resume from root’s shell returned command not found, even from inside the OPatch directory, because OPatch isn’t in root’s PATH. Use the full path or ./:

    # cd $ORACLE_HOME/OPatch
    # ./opatchauto resume
    
    Resuming existing session with id <session_id>
    Start applying binary patch on home $ORACLE_HOME
    Binary patch applied successfully on home $ORACLE_HOME
    Successfully executed rootadd_rdbms.sh on home $ORACLE_HOME
    Postpatch operation completed successfully on home $ORACLE_HOME
    
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    
    Time taken to complete the session 10 minutes, 28 seconds

    opatchauto picked up the saved session and carried on from the failed binary step. There was no need to roll back or start over.

    Step 10: Node 2 OJVM

    The same prereq and opatch apply from the 39803650 directory as on node 1, with the same output: Patch 39803650 successfully applied and the same sub-set patch notice for 39222882. No datapatch was needed on node 2, because the SQL changes had already been applied to the database from node 1 in Step 7.

    Step 11: Post-check against the baseline

    $ $ORACLE_HOME/OPatch/opatch lspatches
    39803650;OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
    39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
    39915792;Database Release Update : 19.33.0.0.261020 (39915792)
    OPatch succeeded.

    SQL patch registry, last rows:

      PATCH_ID ACTION          STATUS          ACTION_TIME                    DESCRIPTION
    ---------- --------------- --------------- ------------------------------ ----------------------------------------------------
      39222882 APPLY           SUCCESS         31-AUG-26 09.02.59.285596 PM   OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
      39472050 APPLY           SUCCESS         31-AUG-26 09.03.55.262038 PM   Database Release Update : 19.32.0.0.260721 (39472050)
      39222882 ROLLBACK        SUCCESS         10-NOV-26 09.18.28.051415 PM   OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
      39803650 APPLY           SUCCESS         10-NOV-26 09.18.38.798776 PM   OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
      39915792 APPLY           SUCCESS         10-NOV-26 09.19.41.446625 PM   Database Release Update : 19.33.0.0.261020 (39915792)
    • Invalid objects: 29 before patching, no rows selected after utlrp.
    • All dba_registry components were VALID or OPTION OFF, the same as the baseline.
    • SQL*Plus now reports Version 19.33.0.0.0.
    • Crontab entries were re-enabled and the application team was asked to validate.

    Two rows that sometimes cause confusion: OCW 39922783 shows in lspatches but not in dba_registry_sqlpatch, because it has no SQL component. The 19.32 DB RU 39472050 has no ROLLBACK row, because a newer RU is applied on top of the older one rather than rolling it back.

    Timeline

    Clock times are from the opatchauto, opatch and datapatch outputs, and show where the two nodes overlapped.

    StartNode 1Node 2Result
    20:21GI home (22m 51s)—Success
    20:44DB home (14m 55s)—Success (SQL deferred)
    21:01—GI home (27m 57s)Success
    21:05OJVM opatch applyGI home runningSuccess
    21:15Startup + datapatch (done 21:19), then utlrpGI home runningSuccess
    21:29—DB home, first attempt (10m 56s)Failed: no space
    21:40—Log cleanup4.5 GB → 20 GB free
    21:47—opatchauto resume (10m 28s)Success
    22:08—OJVM opatch applySuccess

    Lessons learned

    • Check free space on every node, not just the first. Node 1 was fine and node 2 was at 97% on the same mount. A quick df -h on the homes filesystem of each node belongs in the precheck, especially when you patch the nodes in parallel and won’t see node 2’s problem until it’s mid-apply.
    • Look at the listener ADR homes, not only the database ones. Most housekeeping scripts cover the RDBMS alert log and trace files. Here the ASM network listener alone had 15 GB of rotated files. Add the tnslsnr homes to a scheduled ADRCI purge.
    • Don’t panic when opatchauto fails mid-apply. Fix the cause and run opatchauto resume from the same home with the full path. The session ID is kept and it continues from the failed step. The TAR backups from Step 3 were there if it hadn’t.
    • Source the environment in the shell you actually use. Pasting the env lines together with sudo su - oracle leaves ORACLE_HOME empty. Run echo $ORACLE_HOME before trusting which opatch.
    • Take a baseline first. Recording the 29 application-schema invalids before patching, then running utlrp after datapatch, made the post-check comparison simple.

    ما شاء الله, one more cycle closed cleanly in spite of the space issue. If you run 19c on Grid Infrastructure, check your listener diag directories before the next CPU window.

  • AHF 26.9 Upgrade on RAC: Fixing a Remote Node SSH Failure

    Alhamdulillah. I recently upgraded Oracle AHF from 26.5.1.0.0 to 26.9.0.0.0 on a two-node RAC. Node 1 upgraded cleanly. Node 2 failed because the installer couldn’t authenticate over SSH as root. Here’s what happened and how I finished the job with a local upgrade.

    Hostnames and client-specific details are omitted.

    Environment

    • Platform: Linux x86-64
    • Setup: two-node Oracle RAC
    • Initial AHF version: 26.5.1.0.0
    • Target AHF version: 26.9.0.0.0
    • AHF home: /opt/oracle.ahf
    • AHF data directory: <oracle_base>/oracle.ahf/data

    The paths in the examples below are placeholders. Replace them with your actual installer and staging directories. Run the signature check and ahf_setup from the extracted installer directory, and run each command with the privileges the target paths require.

    The Problem

    I launched the cluster-wide installer from Node 1:

    cd /path/to/ahf_installer
    ./ahf_setup

    Node 1 upgraded fine. When it moved on to Node 2, the installer kept asking for the root password, then failed:

    [ERROR] : [node2] Failed to Upgrade AHF. Exit Status : 255

    The summary showed Node 1 on 26.9.0.0.0 and Node 2 still on 26.5.1.0.0. The installer still printed a general success message at the end, even though the cluster was only half upgraded.

    Lesson: read the per-node summary. Don’t trust the final success line.

    The Fix: Local Upgrade on Node 2

    Step 1: Copy and verify the package

    I copied the installer ZIP from Node 1 to Node 2 using an approved transfer method, then checked its integrity:

    unzip -t AHF-LINUX_v26.9.0.zip
    sha256sum AHF-LINUX_v26.9.0.zip

    The SHA-256 matched the checksum from the source node.

    Step 2: Extract and verify the signature

    mkdir -p /path/to/ahf_stage/26.9.0
    
    unzip -o AHF-LINUX_v26.9.0.zip \
      -d /path/to/ahf_stage/26.9.0
    
    cd /path/to/ahf_stage/26.9.0
    
    openssl dgst -sha256 \
      -verify ./oracle-tfa.pub \
      -signature ./ahf_setup.dat \
      ./ahf_setup

    Expected output:

    Verified OK

    Step 3: Run the local upgrade

    Oracle’s upgrade guide says to run ahf_setup as root. My account had authorized sudo privileges, so I ran the installer locally with sudo:

    sudo ./ahf_setup -local

    The installer detected the existing AHF installation and prompted for confirmation. The local upgrade completed successfully. Its final summary reported both Node 2 and Node 1 at AHF version 26.9.0.0.0 with status UPGRADED. I then verified the version and TFA status from both nodes.

    Step 4: Verify on both nodes

    sudo /opt/oracle.ahf/bin/tfactl version
    sudo /opt/oracle.ahf/bin/tfactl status

    Results on both nodes:

    • AHF version: 26.9.0.0.0
    • Matching build ID: 260900020260911121902
    • TFA status: RUNNING
    • Inventory status: COMPLETE
    • No version mismatch between nodes

    Observed Failure

    The installer couldn’t authenticate as root over SSH to the remote node. Oracle documents secure daemon socket communication for upgrading an existing AHF cluster; in this case, the installer nevertheless prompted for root SSH authentication.

    My logs show the immediate failure: the root SSH authentication attempt failed. They don’t show why the installer took that path or why the authentication failed, and I didn’t dig into it further.

    Best Practices

    • Confirm the target version and compatibility before you start.
    • Verify the installer checksum and signature before running it.
    • Run the installer as root, or with sudo if your account is authorized.
    • Confirm the required privileges and cluster communication prerequisites before upgrading. If unexpected SSH authentication prompts occur, investigate the issue rather than changing SSH settings without approval.
    • If remote authentication fails, use the documented local-upgrade procedure. Don’t make unapproved SSH or security changes.
    • Check the per-node summary and TFA status on every node.
    • Keep the installer logs and record the final version and build ID for change management.

    Conclusion

    ما شاء الله, both RAC nodes are now on AHF 26.9.0.0.0. The takeaway: a success message from the installer doesn’t guarantee every node was upgraded. Check each node’s status and confirm TFA is running before you close the change.

    Always follow the Oracle documentation and your organization’s change-management process for production work.

    References

  • Oracle 19c Data Guard: Recovering UNNAMED Datafiles on a Multitenant Physical Standby

    Alhamdulillah.

    Oracle Data Guard physical standbys keep redo synchronized automatically — until the primary adds a datafile that the standby can’t place on disk. When STANDBY_FILE_MANAGEMENT=MANUAL, the standby refuses to guess where the new file should live. Managed Standby Recovery stops instead, leaving the control file ahead of the actual datafiles and the standby parked at a consistent SCN.

    This post walks through a real 19c multitenant incident: MRP0 hit ORA-01274 after a new datafile couldn’t be added, recovery stalled, and the fix required creating the missing file inside the correct pluggable database container, then resuming and verifying redo apply. In this incident, the missing datafile belonged to PROD. Creating the file from within the owning PDB resolved the placeholder, after which Redo Apply could be restarted from CDB$ROOT.

    Failure Diagnosis

    The alert log during redo apply:
    Recovery of Online Redo Log: Thread 1 Group 4 Seq 105902 Reading mem 0
    Mem# 0: /uora/PROD/db/product/19.0.0/dbs/broken3
    2025-05-12T08:49:56.027319-04:00
    PROD(3):File #433 added to control file as ‘UNNAMED00433’ because
    PROD(3):the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
    PROD(3):The file should be manually created to continue.
    PR00 (PID:2796228): MRP0: Background Media Recovery terminated with error 1274
    2025-05-12T08:49:56.092976-04:00
    Errors in file /uora/PROD/db/diag/rdbms/proddr/cdbPROD/trace/cdbPROD_pr00_2796228.trc:
    ORA-01274: cannot add data file that was originally created as
    ‘/uora/PROD/data/proddata/users02.dbf’
    PR00 (PID:2796228): Managed Standby Recovery not using Real Time Apply
    Recovery interrupted!
    2025-05-12T08:49:57.933554-04:00
    Recovery stopped due to failure in applying recovery marker (opcode 17.30).
    Datafiles are recovered to a consistent state at change 36278361292
    but controlfile could be ahead of datafiles.
    Stopping change tracking


    Three things happening in sequence:

    1. **Control file registered a file that doesn’t exist.** The redo stream carried a datafile creation from the primary. The standby’s control file logged it — but because STANDBY_FILE_MANAGEMENT=MANUAL, the standby recorded it as UNNAMED00433 instead of creating it.

    2. **Recovery stopped on the recovery marker.** Opcode 17.30 is the file operation that couldn’t be satisfied. The standby parked at SCN 36278361292 — a point where datafiles are internally consistent, but the control file believes file 433 exists when it doesn’t.

    3. **MRP0 exited.** The managed standby process shut down cleanly. The alert log shows “not using Real Time Apply,” indicating recovery was interrupted and the standby could not continue applying the current redo stream.

    **Root cause:** Configuration, not corruption. STANDBY_FILE_MANAGEMENT=MANUAL means every new primary datafile must be created manually on the standby. With AUTO and proper DB_FILE_NAME_CONVERT / PDB_FILE_NAME_CONVERT setup, new files appear automatically. This environment chose MANUAL, so a routine tablespace addition on the primary became a recovery outage on the standby.

    ## Recovery Workflow

    MRP0 stops → Query UNNAMED files → Switch to PDB → Create datafile → Resume MRP → Verify

    ## Step 1 — Locate the UNNAMED File

    Confirm the standby’s role and find the placeholder. Run from CDB$ROOT:


    SET ECHO OFF PAGESIZE 60 LINESIZE 120 TRIMSPOOL ON
    COL NAME FORMAT A50
    SQL> SELECT name, open_mode, database_role FROM v$database;
    NAME OPEN_MODE DATABASE_ROLE
    ——— ——————– —————-
    CDBSTBY MOUNTED PHYSICAL STANDBY
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    433 /u01/app/oracle/product/19.0.0/dbhome_1/dbs/UNNAMED00433


    File number is 433. Which PDB owns it? Use CON_ID:


    COL CON_ID FORMAT 9999
    COL FILE# FORMAT 9999
    COL NAME FORMAT A50
    SQL> SELECT con_id, file#, name FROM V$DATAFILE WHERE file# = 433;
    CON_ID FILE# NAME
    ———- —— ————————————————–
    3 433 /u01/app/oracle/product/19.0.0/dbhome_1/dbs/UNNAMED00433


    CON_ID = 3 is PROD. That’s where the fix must run.

    ## Step 2 — PDB Recovery: Create the File in the Right Container

    From CDB$ROOT, the command appeared to succeed:


    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00433’
    AS ‘/uora/PROD/data/proddrdata/users02.dbf’;
    Database altered.
    “`

    But checking from the PDB showed the placeholder still registered:


    SQL> ALTER SESSION SET CONTAINER = PROD;
    Session altered.
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    433 /uora/PROD/db/product/19.0.0/dbs/UNNAMED00433


    After switching to the PROD container and reissuing the command:


    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00433’
    AS ‘/uora/PROD/data/proddrdata/users02.dbf’;
    Database altered.
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    no rows selected


    **Important:** Validate this behavior against Oracle 19c documentation and test in non-production before applying to your standby.

    ## Step 3 — Redo Apply Resumption


    SQL> ALTER SESSION SET CONTAINER = CDB$ROOT;
    Session altered.
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE DISCONNECT FROM SESSION;
    Database altered.
    SQL> SELECT PROCESS, STATUS, THREAD#, SEQUENCE# FROM V$MANAGED_STANDBY;
    PROCESS STATUS THREAD# SEQUENCE#
    ———- ————— ——- ———
    MRP0 APPLYING_LOG 1 105902
    ARCH CLOSING 1 105905
    RFS IDLE 1 105906


    MRP0 is back in APPLYING_LOG.

    ## Step 4 — Expect More: The Second UNNAMED File

    Recovery resumed and encountered another placeholder:

    “`sql
    SQL> ALTER SESSION SET CONTAINER = PROD;
    SQL> SELECT FILE#, NAME FROM V$DATAFILE WHERE NAME LIKE ‘%UNNAMED%’;
    FILE# NAME
    ———- ————————————————–
    435 /uora/PROD/db/product/19.0.0/dbs/UNNAMED00435
    SQL> ALTER DATABASE CREATE DATAFILE ‘UNNAMED00435’
    AS ‘/uora/PROD/data/proddrdata/transaction_table_16.dbf’;
    Database altered.


    Expect a loop. Every datafile added on the primary after the gap will appear as another UNNAMED entry. Grep the alert log for the full backlog first:


    grep -i ‘unnamed’ alert_cdbstby.log | head -20
    grep ‘ORA-01274’ alert_cdbstby.log | wc -l


    ## Recovery Verification

    Once placeholders are gone, resume recovery:


    ALTER SESSION SET CONTAINER = CDB$ROOT;
    ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE DISCONNECT FROM SESSION;
    “`

    Verify:


    SELECT con_id, file#, name FROM V$DATAFILE WHERE name LIKE ‘%UNNAMED%’;
    SELECT process, status, thread#, sequence# FROM V$MANAGED_STANDBY WHERE process LIKE ‘MRP%’;
    SELECT * FROM V$DATAGUARD_STATS;


    Clean results: zero UNNAMED rows, MRP0 in APPLYING_LOG, apply lag returning to normal.

    ## Alternative: Restore from Backup

    If redo is unavailable or purged:

    1. Identify primary backup with missing file
    2. Restore to standby location
    3. Register with control file via RMAN
    4. Resume recovery

    Follow Oracle RMAN and Data Guard documentation for your configuration.

    ## Lessons Learned

    **Decide STANDBY_FILE_MANAGEMENT deliberately.** MANUAL works for locked-down environments. AUTO prevents this class of manual intervention when file naming and storage support it. ما شاء الله — the infrastructure, when well-designed, removes recovery work.

    **Session container context can affect recovery steps.** In this incident, CREATE DATAFILE from CDB$ROOT appeared to succeed but didn’t resolve the placeholder until run from PROD PDB. Validate against Oracle 19c documentation and test in non-production.

    **Expect a loop.** Every datafile added after the gap arrives as another UNNAMED entry. Grep the alert log for full backlog first.

    **Verify before closing.** Confirm zero UNNAMED files, MRP0 running, settled lag before closing incident.

    ## Conclusion

    ORA-01274 on a physical standby is a configuration boundary, not data loss. This incident resolved via CREATE DATAFILE in the PROD PDB, followed by managed recovery restart from CDB$ROOT. Before applying: Verify steps against Oracle 19c documentation, test in non-production, verify standby status and lag.

  • Oracle EBS 12.2.4 Upgrade to Oracle Database 19c: Oracle Restart Upgrade, Non-CDB to PDB Conversion, and Lessons Learned

    Alhamdulillah, this post brings together one of the more involved upgrade journeys I have worked through recently — taking an Oracle E-Business Suite 12.2 database stack from 12.1.0.2 all the way to 19c, on a single standalone host running Oracle Restart.

    What made this engagement worth writing up is that it was not a clean, textbook upgrade. The Grid Infrastructure layer presented several unexpected challenges — a chain of CLSRSC failures that had to be cleared before it would move, and once Grid was finally on 19c, the database upgrade and the non-CDB to PDB conversion had their own set of gotchas around ASM registration, password files, and UTL_FILE_DIR.

    I have combined both phases — the Grid Infrastructure upgrade and the database upgrade plus multitenant conversion — into one walkthrough, because in the real world they are not separate stories. The GI upgrade forms the foundation for every subsequent phase. If you only read GI upgrade blogs in isolation, you miss how the pieces connect.

    All hostnames, database names, ASM diskgroup names, and paths below are genericized. The technical substance is exactly as it played out.

    Published: June 2026

    Environment:
    Oracle EBS 12.2.4 | AD/TXK Delta 13 | Oracle Database 19c (19.30)
    (Versions shown are from the environment used in this engagement, not a minimum certified baseline — always confirm your own certified levels.)


    Executive Summary

    This upgrade involved moving an Oracle E-Business Suite 12.2 database environment from Oracle Database 12.1.0.2 to 19c on a standalone Oracle Restart host. The journey consisted of four major phases:

    • Oracle Restart (Grid Infrastructure) upgrade from 12.1.0.2 to 19c
    • Database upgrade from 12.1.0.2 to 19c using DBUA
    • Non-CDB to PDB conversion using Oracle EBS TXK utilities
    • EBS application-tier reconfiguration and validation

    The most time-consuming portions were not the upgrades themselves, but resolving legacy Oracle Restart configuration issues, rebuilding HAS components, handling ASM registration after the GI upgrade, addressing password-file requirements during the PDB conversion, and resolving UTL_FILE_DIR migration challenges.

    This guide covers an end-to-end Oracle EBS 12.2.4 upgrade to Oracle Database 19c, including Oracle Restart upgrade, DBUA upgrade, non-CDB to PDB conversion, AutoConfig remediation, and post-upgrade validation.


    Environment overview

    ComponentDetail
    PlatformSingle standalone Linux host (dbhost) running Oracle Restart / standalone HAS — not full RAC Clusterware
    Apps tierSeparate node (appshost) under the applmgr-style owner
    EBS release12.2.4
    AD levelAD.C.Delta.13
    TXK levelTXK.C.Delta.13
    Source GI12.1.0.2
    Target GI19c (19.30)
    Source DBNon-CDB EBS database, SID ebsdb, on 12.1.0.2
    Target DB19c (19.30), ebsdb plugged in as a lowercase PDB inside CDB EBSCDB
    StorageASM diskgroups +DATA and +RECO
    Old GI home/u01/app/12.1.0.2/grid
    New GI home/u01/app/19.0.0/grid
    Old DB home/u01/app/oracle/product/12.1.0.2/dbhome_1
    New DB home/u01/app/oracle/product/19.0.0/dbhome_1

    The overall sequence was four phases:

    1. Grid Infrastructure 12.1.0.2 → 19c (Oracle Restart upgrade)
    2. Database 12.1.0.2 → 19c via DBUA (-keepEvents)
    3. Non-CDB → PDB conversion (plug ebsdb into EBSCDB)
    4. EBS application-tier cutover (AutoConfig, adop)

    The transformation at a glance

            BEFORE                                AFTER
    
        EBS Apps Tier                         EBS Apps Tier
             |                                     |
    12.1.0.2 Non-CDB (ebsdb)        ===>      PDB ebsdb
             |                                     |
    Oracle Restart 12.1.0.2                  CDB EBSCDB (19c)
             |                                     |
    ASM (+DATA / +RECO)                      Oracle Restart 19c
                                                   |
                                             ASM (+DATA / +RECO)

    The apps tier and ASM storage stay in place; everything between them moves up to 19c, and the standalone non-CDB becomes a pluggable database inside a container.

    A note on certification before you begin

    Do not assume any upgrade path is valid. Before starting, confirm that your EBS code level, AD/TXK delta level, database RU, and interoperability patches are certified per the Oracle EBS 12.2 and Oracle Database 19c certification and interoperability documentation. Skipping this check is the most common reason an upgrade that “worked in the lab” falls apart in the field. Database RU levels and EBS interoperability certifications change frequently, so always verify the latest certification matrix in My Oracle Support before implementation — and treat the 19.30 RU referenced here as the level used in this engagement, not the only supported target.

    Rough timeline (for maintenance-window planning)

    PhaseTypical duration
    GI (Oracle Restart) upgrade30–60 min
    DBUA database upgrade2–4 hrs
    Non-CDB → PDB conversion~15 min
    AutoConfig & validation30–60 min

    These are working estimates from this engagement, not guarantees — database size, invalid-object count, and hardware all move the numbers. Treat the GI and DBUA phases as the long poles when you size the window.


    Phase 1 — Grid Infrastructure 12.1.0.2 → 19c

    The 19c Grid software was already laid down (gold image extracted into the new home), and the OUI installer had paused — as it always does — waiting for rootupgrade.sh to be run as root. That is where everything went sideways.

    The first wall: CLSRSC-324 — Could not open old init.cssd

    Running the upgrade script:

    /u01/app/19.0.0/grid/rootupgrade.sh

    failed almost immediately:

    CLSRSC-324: Could not open old init.cssd
    Died at /u01/app/19.0.0/grid/crs/install/s_crsutils.pm line 1569

    The 19c rootupgrade.sh reads the old configuration from init.cssd to learn where the previous CRS home lived. On this host, /etc/init.d/init.cssd was simply missing — it had been stripped out during earlier deconfig attempts before I picked the task up.

    First I confirmed what the upgrade actually expected as the old home:

    grep -iE "old_crs_home|asm_upgrade|oracle_home" /u01/app/19.0.0/grid/crs/install/crsconfig_params
    OLD_CRS_HOME=/u01/app/12.1.0.2/grid
    ASM_UPGRADE=true

    Then recreated the missing init.cssd pointing at the old home:

    cat > /etc/init.d/init.cssd << 'EOF'
    #!/bin/bash
    ORA_CRS_HOME=/u01/app/12.1.0.2/grid
    export ORA_CRS_HOME
    EOF
    chmod 755 /etc/init.d/init.cssd

    The second wall: a missing OLR pointer

    With init.cssd back, the next thing the script needs is the Oracle Local Registry pointer file, /etc/oracle/olr.loc — also missing.

    A common trap here is to assume the OLR is named <hostname>.olr. It was not. The actual file was local.ocr:

    ls -lrt /u01/app/12.1.0.2/grid/cdata/localhost/
    -rw-r----- 1 oracle dba 503484416 Feb 10 2017 local.ocr

    Always verify OLR integrity before trusting it:

    /u01/app/12.1.0.2/grid/bin/ocrcheck -local
    Device/File integrity check succeeded
    Local registry integrity check succeeded
    Logical corruption check succeeded

    Then recreate the pointer with the correct path:

    mkdir -p /etc/oracle
    cat > /etc/oracle/olr.loc << 'EOF'
    olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/localhost/local.ocr
    crs_home=/u01/app/12.1.0.2/grid
    EOF
    chmod 644 /etc/oracle/olr.loc

    The third major blocker: CLSRSC-348 / CLSRSC-318 — old HAS stack will not stop or start

    With the bootstrap files back, rootupgrade.sh progressed to step 2 (GetOldConfig) and then stopped again:

    CLSRSC-192: Unable to stop Oracle Restart
    CLSRSC-348: The Oracle Restart stack failed to stop

    The upgrade was trying to cleanly stop the old 12.1 HAS stack — but the old stack was not even contactable:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4639: Could not contact Oracle High Availability Services

    So I tried to start it, and hit the classic library-mismatch error:

    /u01/app/12.1.0.2/grid/bin/crsctl start has
    CRS-4652: Failure 3 in clsvswversion for the local node

    Root Cause

    The original 12.1 Oracle Restart stack was not in a healthy state. Previous deconfiguration attempts had removed critical bootstrap files and left HAS only partially configured.

    Because Oracle only upgrades an existing, functioning stack, rootupgrade.sh could not proceed until the old stack was cleanly rebuilt and verified. The fix is to deconfigure it fully, rebuild it on 12.1, confirm it is online — and only then let the 19c upgrade take over.

    Deconfigure the broken old stack:

    cd /u01/app/12.1.0.2/grid/crs/install
    perl roothas.pl -deconfig -force
    CLSRSC-337: Successfully deconfigured Oracle Restart stack

    Set the environment correctly (the LD_LIBRARY_PATH is what fixes the CRS-4652 library failure):

    export ORACLE_HOME=/u01/app/12.1.0.2/grid
    export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

    Rebuild the old 12.1 HAS stack using the 12.1 perl binary (do not use the system perl — use the one shipped with the old Grid home):

    /u01/app/12.1.0.2/grid/perl/bin/perl \
    -I/u01/app/12.1.0.2/grid/perl/lib \
    -I/u01/app/12.1.0.2/grid/crs/install \
    /u01/app/12.1.0.2/grid/crs/install/roothas.pl

    This reconfigured Oracle Restart successfully on 12.1 — note this is the script rebuilding the old stack, confirmed by:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Verify HAS is genuinely online before going further:

    /u01/app/12.1.0.2/grid/bin/crsctl check has
    CRS-4638: Oracle High Availability Services is online

    Clearing stale checkpoints and finally running the upgrade

    The earlier failed rootupgrade.sh runs leave behind checkpoint files that block any retry — the script thinks it has already partly completed and refuses to start fresh. Clear them:

    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTHAS_STACK
    rm -f /u01/app/19.0.0/grid/checkpoints/ROOTCRS_STACK

    Now re-run the 19c upgrade:

    /u01/app/19.0.0/grid/rootupgrade.sh

    This time it walked cleanly through all 12 steps (10–20 minutes), pinned the node, backed up the OLR at both the 12.1 and 19c homes, and finished with:

    CRS-4123: Oracle High Availability Services has been started.
    CLSRSC-327: Successfully configured Oracle Restart for a standalone server

    Back in the OUI GUI, clicking OK finalized the installer. ما شاء الله — Grid was now on 19c.

    One of the most common post-GI issues: ASM was never registered

    This one is worth its own warning, because it does not surface during the GI upgrade — it surfaces later, when you go to create the CDB and DBCA fails with something like DBT-06604: insufficient free space. The real reason is that ASM was not registered with Oracle Restart, so the tools silently fell back to filesystem storage, which did not have the room.

    A point worth being precise about: depending on the configuration, ASM may already exist and simply not be running — in which case srvctl start asm (or mounting the diskgroups) is all that is needed. Check before you add — running srvctl add asm when ASM is already configured will error, so confirm the current state with srvctl status asm / srvctl config asm first. In this environment ASM was not registered correctly, so it had to be added back:

    # verify first — does Oracle Restart already know about ASM?
    srvctl status asm
    srvctl config asm
    # in this environment it was not registered, so:
    srvctl add asm
    srvctl start asm

    And if the diskgroups are not mounted, mount them explicitly from the ASM instance:

    ALTER DISKGROUP DATA MOUNT;
    ALTER DISKGROUP RECO MOUNT;

    Lesson: after any Oracle Restart GI upgrade, always confirm srvctl status asm before you trust any DBCA/DBUA storage decision.

    Lessons Learned

    • Verify Oracle Restart is healthy before attempting an upgrade.
    • Confirm init.cssd and olr.loc exist and point to valid locations.
    • Clear stale checkpoint files before retrying rootupgrade.sh.
    • Validate ASM registration immediately after the upgrade.

    Phase 2 — Database 12.1.0.2 → 19c via DBUA

    With 19c Grid and ASM healthy, and the 19c CDB (EBSCDB) already created and patched (datapatch + catmgd.sql for the MGDSYS schema), the next phase was upgrading the non-CDB ebsdb in place from 12.1.0.2 to 19c. It stays a non-CDB for now — the PDB conversion is a deliberately separate Phase 3.

    For EBS environments, launch DBUA using the -keepEvents option so that the EBS-required events and hidden (underscore) parameters are preserved during the upgrade. Dropping those silently breaks the apps tier later.

    X11 forwarding across a user switch

    A very common, very annoying blocker: you SSH in as one OS user, then sudo to oracle, and X11 forwarding breaks because DISPLAY and the Xauthority cookie do not carry across the user switch. DBUA is a GUI tool, so this stops you cold.

    The fix is to capture the cookie as your login user and add it under oracle:

    # as the login user
    xauth list # note the cookie line for your display, e.g. dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    # as oracle
    xauth add dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
    export DISPLAY=localhost:11.0
    xdpyinfo | head # must return display info, not "unable to open display"

    Tip: do not reuse the runbook author’s DISPLAY value (their own workstation name). It will never work for you — set your own.

    DBUA wizard choices that matter for EBS

    ScreenChoice
    CDB / PDB conversionLeave unchecked — conversion is a separate later phase
    Recovery option“I have my own backup/restore strategy” (RMAN backup already taken) — never leave this blank
    Recompile invalid objectsEnabled (parallel)
    Upgrade timezone dataEnabled
    compatibleLeave at 12.1.0.2 during the upgrade — preserves rollback. Raise it later, deliberately.
    Listener migrationKeep the CDB listener migration checked

    A couple of things that look like errors but are not:

    • The DBUA summary may show db_unique_name where you expect db_name. If your unique name differs from the SID, this is just DBUA displaying the unique name — verify against the live DB and move on.
    • The CDB listener (here on port 1523) must be up before DBUA can pass the network screen. If it is down, start it from the 19c home first: lsnrctl start <listener_name>.

    Wrap the whole thing in screen (or tmux) so an SSH drop does not kill the upgrade mid-flight. If screen is not installed, at minimum use nohup-style protection — a dropped DBUA mid-upgrade is a bad day.

    DBUA for an EBS 12.1.0.2 → 19c database typically runs 2–4 hours, plus 30–60 minutes of recompile/datapatch. Expect some ORA- noise in the logs (recompile warnings); watch for new error patterns, not the routine ones.

    Post-DBUA housekeeping

    Once the upgrade completes and all post-upgrade validation is complete (still as a non-CDB), raise COMPATIBLE to 19.0.0. Note that after COMPATIBLE is increased, rollback to the previous release is no longer possible — so this is a deliberate post-validation step, not an automatic next one:

    -- raise compatibility — this is the point of no return, do it deliberately
    ALTER SYSTEM SET compatible='19.0.0' SCOPE=SPFILE;
    -- compatible only takes effect after a restart
    SHUTDOWN IMMEDIATE;
    STARTUP;

    The exact value depends on your site standards and Oracle’s recommendation for the release — some shops set 19.0.0, others use the fully-qualified 19.0.0.0.0. Both forms are accepted; confirm which one your organization standardizes on before setting it.

    Then:

    • Snapshot a pfile from the spfile.
    • Relocate the spfile into the 19c home’s dbs/.
    • Remove obsolete parameters that 19c rejects — notably sec_case_sensitive_logon and utl_file_dir.
    • Run the XDB migration scripts: dbmsxdbschmig.sql then prvtxdbschmig.plb.
    • Run utlrp.sql (twice) to recompile. In this environment the invalid count dropped from over 12,000 to under 30 across the two passes — normal for an EBS database.

    At this point ebsdb is a healthy 19c non-CDB, ready to be plugged in.

    Lessons Learned

    • Always launch DBUA with -keepEvents for EBS — it preserves the required underscore parameters and event settings.
    • Fix X11 forwarding across the user switch before launching; never reuse the runbook author’s DISPLAY value.
    • Leave compatible at 12.1.0.2 during the upgrade and raise it deliberately afterward to preserve rollback.
    • Treat the db_unique_name-vs-db_name display in the summary as cosmetic, and start the CDB listener before the network screen.
    • Wrap the run in screen/tmux so an SSH drop cannot kill the upgrade mid-flight.

    Phase 3 — Non-CDB to PDB conversion

    This is where ebsdb becomes a lowercase PDB inside EBSCDB. EBS ships a set of txk driver scripts for exactly this path; the lowercase, case-sensitive PDB name is a detail that bites you in every subsequent command (ALTER SESSION SET CONTAINER="ebsdb" — with the quotes).

    Step 1 — Pre-creation tasks

    perl $AD_TOP/patch/115/bin/txkOnPremPrePDBCreationTasks.pl

    This generates the PDB descriptor XML (ebsdb_PDBDesc.xml, the output of DBMS_PDB.DESCRIBE, carrying the <ncdb2pdb>1</ncdb2pdb> flag) and shuts the non-CDB ebsdb down permanently — once you are on this path, there is no casual restart of the old non-CDB.

    Step 2 — Compatibility check

    perl $AD_TOP/patch/115/bin/txkChkPDBCompatability.pl

    This reported 8 violations — and per the runbook rules, all were ignorable: standard SQL-patch warnings and the usual benign messages, no character-set mismatch (a character-set violation is the one you must not ignore). Always read each violation; do not blanket-ignore.

    Step 3 — Create the PDB (plug in)

    perl $AD_TOP/patch/115/bin/txkCreatePDB.pl

    This performs a NOCOPY plug-in, allowing the existing ASM datafiles to remain in place while the database is converted into a PDB (the exact ASM file layout varies by environment and Oracle version), and runs noncdb_to_pdb.sql to finish the conversion. It completed in roughly 14 minutes, the PDB state was saved, and ebsdb showed up as CON_ID 3, READ WRITE. ما شاء الله.

    Step 4 — Post-creation tasks (the messy part)

    perl $AD_TOP/patch/115/bin/txkPostPDBCreationTasks.pl

    This is where the real fights happened. Three blockers, all around password files:

    a) ASM password file pre-staging. The script needs the CDB password file in ASM. Pre-stage it with orapwd from the DB home (not the Grid home) into the ASM path for the CDB, then register it:

    orapwd file='+DATA/EBSCDB/orapwEBSCDB' dbuniquename='EBSCDB' format=12
    srvctl modify database -d EBSCDB -pwfile '+DATA/EBSCDB/orapwEBSCDB'

    b) ORA-65066 on the SYSTEM password. Setting the SYSTEM password inside the new PDB threw ORA-65066 until it was applied across all containers:

    ALTER USER SYSTEM IDENTIFIED BY "<StrongPwd#1>" CONTAINER=ALL;

    c) OPW-00029 complexity failure. orapwd rejected the chosen SYSTEM password with OPW-00029 until it satisfied 19c’s special-character complexity rules. Use a value that genuinely meets the 19c verifier (uppercase + lowercase + digit + special character), e.g. <StrongPwd#1>.

    Step 5 — Post-PDB AutoConfig and the UTL_FILE_DIR trap

    Running AutoConfig on the DB tier after the conversion initially failed with a UTL_FILE_DIR error. The trace pointed at /usr/tmp being a symlink rather than a real directory, which the new UTL_FILE_DIR-replacement mechanism in 19c does not tolerate.

    The fix is to drive the directory configuration explicitly with txkCfgUtlfileDir.pl. First, manually create the directory-list file with real (non-symlink) paths:

    # e.g. /u01/app/oracle/product/19.0.0/dbhome_1/dbs/ebsdb_utlfiledir.txt
    # containing two real paths, e.g.:
    # /u01/app/oracle/ebsdb/temp
    # /u01/app/oracle/ebsdb/plsql

    Then run the script in its two modes:

    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=setUtlFileDir
    perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=syncUtlFileDir

    After that, AutoConfig on the DB tier completed cleanly.

    Worth flagging for the longer term: UTL_FILE_DIR is deprecated, and Oracle’s recommended direction is to use database directory objects (CREATE DIRECTORY) for PL/SQL file I/O wherever your EBS code level supports it. The txkCfgUtlfileDir.pl mechanism above is the EBS-supported bridge for this release; treat directory objects as the strategic target.

    Lessons Learned

    • The lowercase, case-sensitive PDB name ("ebsdb" with quotes) propagates into every subsequent ALTER SESSION SET CONTAINER — get it right once and stay consistent.
    • Read every compatibility violation; ignore only the benign ones, and never ignore a character-set mismatch.
    • Budget real time for password files: pre-stage the ASM password file from the DB home, expect ORA-65066 (apply CONTAINER=ALL) and OPW-00029 (19c complexity rules).
    • A symlinked /usr/tmp breaks post-PDB AutoConfig — drive UTL_FILE_DIR explicitly via txkCfgUtlfileDir.pl with real paths.

    Phase 4 — Application-tier cutover (overview)

    With the database tier healthy at 19c as a PDB, the apps-tier work on appshost is the final phase:

    • Run AutoConfig on both filesystems (fs1 and fs2) to point the apps tier at the new 19c PDB connection details.
    • Clean up any stale adop sessions before starting fresh — an abandoned session from days earlier will block a new cycle. Run adop cleanup (cleanup_mode=full) and then a fresh adop phase=prepare.
    • Validate HugePages, the env files, and Configuration Manager.

    I will cover the full apps-tier cutover in detail in a follow-up post, since it deserves its own walkthrough.


    Final Validation Checklist

    After the upgrade and conversion were complete, the following validations were performed:

    ✓ Oracle Restart resources online and managed by srvctl
    ✓ ASM instance and diskgroups mounted
    ✓ CDB and PDB open in READ WRITE mode
    ✓ Database listener services registered correctly
    ✓ AutoConfig completed successfully on database and application tiers
    ✓ Concurrent Managers started successfully
    ✓ FNDSM service validated
    ✓ Internal Monitor running normally
    ✓ Forms and WebLogic services accessible
    ✓ adop phase=prepare completed without errors
    ✓ Invalid objects reviewed and reduced to an acceptable count
    ✓ Application smoke testing completed
    ✓ All EBS services registered in the listener
    ✓ Concurrent Manager processing validated
    ✓ Test concurrent request submitted and completed
    ✓ Forms login verified
    ✓ Workflow Mailer status verified
    ✓ OACORE, FORMS, and OAFM managed servers healthy
    ✓ No critical errors in AutoConfig logs
    ✓ adopscanlog reviewed with no blocking errors
    ✓ Database and ASM resources managed successfully through srvctl

    Quick post-upgrade SQL validation

    A handful of SQL checks confirm the database came back healthy — the component registry, the container/PDB state, and the invalid-object count. Set column formatting first so the output stays readable:

    1. Component registry — every component should report VALID (or UPGRADED) at 19.0.0:

    SET LINESIZE 200 PAGESIZE 100
    COL comp_name FORMAT A45
    COL version FORMAT A18
    COL status FORMAT A12
    SELECT comp_name, version, status
    FROM dba_registry
    ORDER BY comp_name;

    2. Container and PDB state — for a 19c multitenant environment, V$PDBS is the most useful check because it shows OPEN_MODE directly (the EBS PDB should be READ WRITE):

    COL name FORMAT A20
    SELECT con_id, name, open_mode
    FROM v$pdbs
    ORDER BY con_id;

    DBA_PDBS complements it by showing the persisted state — NORMAL for a healthy plugged-in PDB:

    COL pdb_name FORMAT A20
    COL status FORMAT A12
    SELECT pdb_name, status
    FROM dba_pdbs
    ORDER BY pdb_name;

    3. Invalid objects — review the breakdown by owner, then the overall total (an EBS database normally settles to a small count after two utlrp.sql passes):

    COL owner FORMAT A22
    COL object_type FORMAT A22
    SELECT NVL(owner,'(none)') AS owner,
    object_type,
    COUNT(*) AS invalid_count
    FROM dba_objects
    WHERE status = 'INVALID'
    GROUP BY owner, object_type
    ORDER BY invalid_count DESC;
    -- overall total
    SELECT COUNT(*) AS total_invalid
    FROM dba_objects
    WHERE status = 'INVALID';

    Although every environment is different, the upgrade itself proved relatively straightforward once the underlying Oracle Restart configuration issues were corrected. The majority of effort was spent validating and repairing legacy infrastructure assumptions rather than executing the upgrade utilities themselves.

    Key takeaways

    1. A GI upgrade is only as clean as the old stack underneath it. Most CLSRSC-318 / CLSRSC-324 / CLSRSC-348 failures trace back to an old HAS stack that was never properly configured or was half-deconfigured. Rebuild it cleanly on the old version first, then upgrade.
    2. Verify the bootstrap files yourself. Do not assume the OLR is <hostname>.olr — check cdata/localhost/ and verify with ocrcheck -local.
    3. Stale checkpoints block retries silently. Clear ROOTHAS_STACK / ROOTCRS_STACK before re-running rootupgrade.sh.
    4. After an Oracle Restart GI upgrade, register ASM (srvctl add asm / srvctl start asm) before you trust any storage-related tool.
    5. Use dbua -keepEvents for EBS, and fix X11 forwarding across the user switch before you launch.
    6. The PDB conversion’s hardest part is password files — ORA-65066, OPW-00029, and ASM password-file placement. Budget time for it.
    7. Post-PDB AutoConfig + UTL_FILE_DIR: a symlinked /usr/tmp will fail it. Drive the directory list explicitly with txkCfgUtlfileDir.pl using real paths.

    Looking back, the actual upgrade steps were straightforward. The real effort was understanding and repairing the assumptions the upgrade process makes about the existing environment. Missing bootstrap files, partially deconfigured Oracle Restart components, ASM registration gaps, password-file placement, and UTL_FILE_DIR migration issues consumed far more time than the upgrade binaries themselves. Those lessons are often absent from official documentation — which is exactly why I wanted to document the journey end to end.


    References

    • Oracle E-Business Suite Release 12.2 — Interoperability Notes for Oracle Database 19c (My Oracle Support)
    • Oracle Database 19c Upgrade Guide
    • Using UTL_FILE_DIR or Database Directories for PL/SQL File I/O in Oracle E-Business Suite (My Oracle Support)
    • Oracle E-Business Suite Multitenant / Pluggable Database (PDB) conversion documentation
    • Oracle Grid Infrastructure Installation and Upgrade Guide (Oracle Restart)

    (Look up the current My Oracle Support note IDs for your exact code and database levels — they are revised regularly.)

    If this write-up helped you during your own Oracle E-Business Suite upgrade, you’re welcome to follow the blog for more real-world Oracle Database, EBS, ADOP, cloning, upgrade, and troubleshooting guides drawn from production experience.


    Disclaimer: This post reflects my own hands-on experience and is shared for educational purposes only. All hostnames, database names, diskgroup names, and paths have been genericized. Always test thoroughly in a non-production environment and follow Oracle’s official documentation and your organization’s change-control process before applying any of these steps. The views expressed here are my own and do not represent those of any employer or client.