Tag: 19c

Alhamdulillah, this post brings together one of the more involved upgrade journeys I have worked through recently — taking an Oracle E-Business Suite 12.2 database stack from 12.1.0.2 all the way to 19c, on a single standalone host running Oracle Restart.

What made this engagement worth writing up is that it was not a clean, textbook upgrade. The Grid Infrastructure layer presented several unexpected challenges — a chain of CLSRSC failures that had to be cleared before it would move, and once Grid was finally on 19c, the database upgrade and the non-CDB to PDB conversion had their own set of gotchas around ASM registration, password files, and UTL_FILE_DIR.

I have combined both phases — the Grid Infrastructure upgrade and the database upgrade plus multitenant conversion — into one walkthrough, because in the real world they are not separate stories. The GI upgrade forms the foundation for every subsequent phase. If you only read GI upgrade blogs in isolation, you miss how the pieces connect.

All hostnames, database names, ASM diskgroup names, and paths below are genericized. The technical substance is exactly as it played out.

Published: June 2026

Environment:
Oracle EBS 12.2.4 | AD/TXK Delta 13 | Oracle Database 19c (19.30)
(Versions shown are from the environment used in this engagement, not a minimum certified baseline — always confirm your own certified levels.)

Executive Summary

This upgrade involved moving an Oracle E-Business Suite 12.2 database environment from Oracle Database 12.1.0.2 to 19c on a standalone Oracle Restart host. The journey consisted of four major phases:

• Oracle Restart (Grid Infrastructure) upgrade from 12.1.0.2 to 19c
• Database upgrade from 12.1.0.2 to 19c using DBUA
• Non-CDB to PDB conversion using Oracle EBS TXK utilities
• EBS application-tier reconfiguration and validation

The most time-consuming portions were not the upgrades themselves, but resolving legacy Oracle Restart configuration issues, rebuilding HAS components, handling ASM registration after the GI upgrade, addressing password-file requirements during the PDB conversion, and resolving UTL_FILE_DIR migration challenges.

This guide covers an end-to-end Oracle EBS 12.2.4 upgrade to Oracle Database 19c, including Oracle Restart upgrade, DBUA upgrade, non-CDB to PDB conversion, AutoConfig remediation, and post-upgrade validation.

Environment overview

The overall sequence was four phases:

1. Grid Infrastructure 12.1.0.2 → 19c (Oracle Restart upgrade)
2. Database 12.1.0.2 → 19c via DBUA (-keepEvents)
3. Non-CDB → PDB conversion (plug ebsdb into EBSCDB)
4. EBS application-tier cutover (AutoConfig, adop)

The transformation at a glance

        BEFORE                                AFTER

    EBS Apps Tier                         EBS Apps Tier
         |                                     |
12.1.0.2 Non-CDB (ebsdb)        ===>      PDB ebsdb
         |                                     |
Oracle Restart 12.1.0.2                  CDB EBSCDB (19c)
         |                                     |
ASM (+DATA / +RECO)                      Oracle Restart 19c
                                               |
                                         ASM (+DATA / +RECO)

The apps tier and ASM storage stay in place; everything between them moves up to 19c, and the standalone non-CDB becomes a pluggable database inside a container.

A note on certification before you begin

Do not assume any upgrade path is valid. Before starting, confirm that your EBS code level, AD/TXK delta level, database RU, and interoperability patches are certified per the Oracle EBS 12.2 and Oracle Database 19c certification and interoperability documentation. Skipping this check is the most common reason an upgrade that “worked in the lab” falls apart in the field. Database RU levels and EBS interoperability certifications change frequently, so always verify the latest certification matrix in My Oracle Support before implementation — and treat the 19.30 RU referenced here as the level used in this engagement, not the only supported target.

Rough timeline (for maintenance-window planning)

These are working estimates from this engagement, not guarantees — database size, invalid-object count, and hardware all move the numbers. Treat the GI and DBUA phases as the long poles when you size the window.

Phase 1 — Grid Infrastructure 12.1.0.2 → 19c

The 19c Grid software was already laid down (gold image extracted into the new home), and the OUI installer had paused — as it always does — waiting for rootupgrade.sh to be run as root. That is where everything went sideways.

The first wall: CLSRSC-324 — Could not open old init.cssd

Running the upgrade script:

/u01/app/19.0.0/grid/rootupgrade.sh

failed almost immediately:

CLSRSC-324: Could not open old init.cssd
Died at /u01/app/19.0.0/grid/crs/install/s_crsutils.pm line 1569

The 19c rootupgrade.sh reads the old configuration from init.cssd to learn where the previous CRS home lived. On this host, /etc/init.d/init.cssd was simply missing — it had been stripped out during earlier deconfig attempts before I picked the task up.

First I confirmed what the upgrade actually expected as the old home:

grep -iE "old_crs_home|asm_upgrade|oracle_home" /u01/app/19.0.0/grid/crs/install/crsconfig_params

OLD_CRS_HOME=/u01/app/12.1.0.2/grid
ASM_UPGRADE=true

Then recreated the missing init.cssd pointing at the old home:

cat > /etc/init.d/init.cssd << 'EOF'
#!/bin/bash
ORA_CRS_HOME=/u01/app/12.1.0.2/grid
export ORA_CRS_HOME
EOF
chmod 755 /etc/init.d/init.cssd

The second wall: a missing OLR pointer

With init.cssd back, the next thing the script needs is the Oracle Local Registry pointer file, /etc/oracle/olr.loc — also missing.

A common trap here is to assume the OLR is named <hostname>.olr. It was not. The actual file was local.ocr:

ls -lrt /u01/app/12.1.0.2/grid/cdata/localhost/

-rw-r----- 1 oracle dba 503484416 Feb 10 2017 local.ocr

Always verify OLR integrity before trusting it:

/u01/app/12.1.0.2/grid/bin/ocrcheck -local

Device/File integrity check succeeded
Local registry integrity check succeeded
Logical corruption check succeeded

Then recreate the pointer with the correct path:

mkdir -p /etc/oracle
cat > /etc/oracle/olr.loc << 'EOF'
olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/localhost/local.ocr
crs_home=/u01/app/12.1.0.2/grid
EOF
chmod 644 /etc/oracle/olr.loc

The third major blocker: CLSRSC-348 / CLSRSC-318 — old HAS stack will not stop or start

With the bootstrap files back, rootupgrade.sh progressed to step 2 (GetOldConfig) and then stopped again:

CLSRSC-192: Unable to stop Oracle Restart
CLSRSC-348: The Oracle Restart stack failed to stop

The upgrade was trying to cleanly stop the old 12.1 HAS stack — but the old stack was not even contactable:

/u01/app/12.1.0.2/grid/bin/crsctl check has

CRS-4639: Could not contact Oracle High Availability Services

So I tried to start it, and hit the classic library-mismatch error:

/u01/app/12.1.0.2/grid/bin/crsctl start has

CRS-4652: Failure 3 in clsvswversion for the local node

Root Cause

The original 12.1 Oracle Restart stack was not in a healthy state. Previous deconfiguration attempts had removed critical bootstrap files and left HAS only partially configured.

Because Oracle only upgrades an existing, functioning stack, rootupgrade.sh could not proceed until the old stack was cleanly rebuilt and verified. The fix is to deconfigure it fully, rebuild it on 12.1, confirm it is online — and only then let the 19c upgrade take over.

Deconfigure the broken old stack:

cd /u01/app/12.1.0.2/grid/crs/install
perl roothas.pl -deconfig -force

CLSRSC-337: Successfully deconfigured Oracle Restart stack

Set the environment correctly (the LD_LIBRARY_PATH is what fixes the CRS-4652 library failure):

export ORACLE_HOME=/u01/app/12.1.0.2/grid
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

Rebuild the old 12.1 HAS stack using the 12.1 perl binary (do not use the system perl — use the one shipped with the old Grid home):

/u01/app/12.1.0.2/grid/perl/bin/perl \
  -I/u01/app/12.1.0.2/grid/perl/lib \
  -I/u01/app/12.1.0.2/grid/crs/install \
  /u01/app/12.1.0.2/grid/crs/install/roothas.pl

This reconfigured Oracle Restart successfully on 12.1 — note this is the script rebuilding the old stack, confirmed by:

CRS-4123: Oracle High Availability Services has been started.
CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Verify HAS is genuinely online before going further:

/u01/app/12.1.0.2/grid/bin/crsctl check has

CRS-4638: Oracle High Availability Services is online

Clearing stale checkpoints and finally running the upgrade

The earlier failed rootupgrade.sh runs leave behind checkpoint files that block any retry — the script thinks it has already partly completed and refuses to start fresh. Clear them:

rm -f /u01/app/19.0.0/grid/checkpoints/ROOTHAS_STACK
rm -f /u01/app/19.0.0/grid/checkpoints/ROOTCRS_STACK

Now re-run the 19c upgrade:

/u01/app/19.0.0/grid/rootupgrade.sh

This time it walked cleanly through all 12 steps (10–20 minutes), pinned the node, backed up the OLR at both the 12.1 and 19c homes, and finished with:

CRS-4123: Oracle High Availability Services has been started.
CLSRSC-327: Successfully configured Oracle Restart for a standalone server

Back in the OUI GUI, clicking OK finalized the installer. ما شاء الله — Grid was now on 19c.

One of the most common post-GI issues: ASM was never registered

This one is worth its own warning, because it does not surface during the GI upgrade — it surfaces later, when you go to create the CDB and DBCA fails with something like DBT-06604: insufficient free space. The real reason is that ASM was not registered with Oracle Restart, so the tools silently fell back to filesystem storage, which did not have the room.

A point worth being precise about: depending on the configuration, ASM may already exist and simply not be running — in which case srvctl start asm (or mounting the diskgroups) is all that is needed. Check before you add — running srvctl add asm when ASM is already configured will error, so confirm the current state with srvctl status asm / srvctl config asm first. In this environment ASM was not registered correctly, so it had to be added back:

# verify first — does Oracle Restart already know about ASM?
srvctl status asm
srvctl config asm
# in this environment it was not registered, so:
srvctl add asm
srvctl start asm

And if the diskgroups are not mounted, mount them explicitly from the ASM instance:

ALTER DISKGROUP DATA MOUNT;
ALTER DISKGROUP RECO MOUNT;

Lesson: after any Oracle Restart GI upgrade, always confirm srvctl status asm before you trust any DBCA/DBUA storage decision.

Lessons Learned

• Verify Oracle Restart is healthy before attempting an upgrade.
• Confirm init.cssd and olr.loc exist and point to valid locations.
• Clear stale checkpoint files before retrying rootupgrade.sh.
• Validate ASM registration immediately after the upgrade.

Phase 2 — Database 12.1.0.2 → 19c via DBUA

With 19c Grid and ASM healthy, and the 19c CDB (EBSCDB) already created and patched (datapatch + catmgd.sql for the MGDSYS schema), the next phase was upgrading the non-CDB ebsdb in place from 12.1.0.2 to 19c. It stays a non-CDB for now — the PDB conversion is a deliberately separate Phase 3.

For EBS environments, launch DBUA using the -keepEvents option so that the EBS-required events and hidden (underscore) parameters are preserved during the upgrade. Dropping those silently breaks the apps tier later.

X11 forwarding across a user switch

A very common, very annoying blocker: you SSH in as one OS user, then sudo to oracle, and X11 forwarding breaks because DISPLAY and the Xauthority cookie do not carry across the user switch. DBUA is a GUI tool, so this stops you cold.

The fix is to capture the cookie as your login user and add it under oracle:

# as the login user
xauth list        # note the cookie line for your display, e.g. dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
# as oracle
xauth add dbhost/unix:11 MIT-MAGIC-COOKIE-1 <cookie>
export DISPLAY=localhost:11.0
xdpyinfo | head   # must return display info, not "unable to open display"

Tip: do not reuse the runbook author’s DISPLAY value (their own workstation name). It will never work for you — set your own.

DBUA wizard choices that matter for EBS

A couple of things that look like errors but are not:

• The DBUA summary may show db_unique_name where you expect db_name. If your unique name differs from the SID, this is just DBUA displaying the unique name — verify against the live DB and move on.
• The CDB listener (here on port 1523) must be up before DBUA can pass the network screen. If it is down, start it from the 19c home first: lsnrctl start <listener_name>.

Wrap the whole thing in screen (or tmux) so an SSH drop does not kill the upgrade mid-flight. If screen is not installed, at minimum use nohup-style protection — a dropped DBUA mid-upgrade is a bad day.

DBUA for an EBS 12.1.0.2 → 19c database typically runs 2–4 hours, plus 30–60 minutes of recompile/datapatch. Expect some ORA- noise in the logs (recompile warnings); watch for new error patterns, not the routine ones.

Post-DBUA housekeeping

Once the upgrade completes and all post-upgrade validation is complete (still as a non-CDB), raise COMPATIBLE to 19.0.0. Note that after COMPATIBLE is increased, rollback to the previous release is no longer possible — so this is a deliberate post-validation step, not an automatic next one:

-- raise compatibility — this is the point of no return, do it deliberately
ALTER SYSTEM SET compatible='19.0.0' SCOPE=SPFILE;
-- compatible only takes effect after a restart
SHUTDOWN IMMEDIATE;
STARTUP;

The exact value depends on your site standards and Oracle’s recommendation for the release — some shops set 19.0.0, others use the fully-qualified 19.0.0.0.0. Both forms are accepted; confirm which one your organization standardizes on before setting it.

Then:

• Snapshot a pfile from the spfile.
• Relocate the spfile into the 19c home’s dbs/.
• Remove obsolete parameters that 19c rejects — notably sec_case_sensitive_logon and utl_file_dir.
• Run the XDB migration scripts: dbmsxdbschmig.sql then prvtxdbschmig.plb.
• Run utlrp.sql (twice) to recompile. In this environment the invalid count dropped from over 12,000 to under 30 across the two passes — normal for an EBS database.

At this point ebsdb is a healthy 19c non-CDB, ready to be plugged in.

Lessons Learned

• Always launch DBUA with -keepEvents for EBS — it preserves the required underscore parameters and event settings.
• Fix X11 forwarding across the user switch before launching; never reuse the runbook author’s DISPLAY value.
• Leave compatible at 12.1.0.2 during the upgrade and raise it deliberately afterward to preserve rollback.
• Treat the db_unique_name-vs-db_name display in the summary as cosmetic, and start the CDB listener before the network screen.
• Wrap the run in screen/tmux so an SSH drop cannot kill the upgrade mid-flight.

Phase 3 — Non-CDB to PDB conversion

This is where ebsdb becomes a lowercase PDB inside EBSCDB. EBS ships a set of txk driver scripts for exactly this path; the lowercase, case-sensitive PDB name is a detail that bites you in every subsequent command (ALTER SESSION SET CONTAINER="ebsdb" — with the quotes).

Step 1 — Pre-creation tasks

perl $AD_TOP/patch/115/bin/txkOnPremPrePDBCreationTasks.pl

This generates the PDB descriptor XML (ebsdb_PDBDesc.xml, the output of DBMS_PDB.DESCRIBE, carrying the <ncdb2pdb>1</ncdb2pdb> flag) and shuts the non-CDB ebsdb down permanently — once you are on this path, there is no casual restart of the old non-CDB.

Step 2 — Compatibility check

perl $AD_TOP/patch/115/bin/txkChkPDBCompatability.pl

This reported 8 violations — and per the runbook rules, all were ignorable: standard SQL-patch warnings and the usual benign messages, no character-set mismatch (a character-set violation is the one you must not ignore). Always read each violation; do not blanket-ignore.

Step 3 — Create the PDB (plug in)

perl $AD_TOP/patch/115/bin/txkCreatePDB.pl

This performs a NOCOPY plug-in, allowing the existing ASM datafiles to remain in place while the database is converted into a PDB (the exact ASM file layout varies by environment and Oracle version), and runs noncdb_to_pdb.sql to finish the conversion. It completed in roughly 14 minutes, the PDB state was saved, and ebsdb showed up as CON_ID 3, READ WRITE. ما شاء الله.

Step 4 — Post-creation tasks (the messy part)

perl $AD_TOP/patch/115/bin/txkPostPDBCreationTasks.pl

This is where the real fights happened. Three blockers, all around password files:

a) ASM password file pre-staging. The script needs the CDB password file in ASM. Pre-stage it with orapwd from the DB home (not the Grid home) into the ASM path for the CDB, then register it:

orapwd file='+DATA/EBSCDB/orapwEBSCDB' dbuniquename='EBSCDB' format=12
srvctl modify database -d EBSCDB -pwfile '+DATA/EBSCDB/orapwEBSCDB'

b) ORA-65066 on the SYSTEM password. Setting the SYSTEM password inside the new PDB threw ORA-65066 until it was applied across all containers:

ALTER USER SYSTEM IDENTIFIED BY "<StrongPwd#1>" CONTAINER=ALL;

c) OPW-00029 complexity failure. orapwd rejected the chosen SYSTEM password with OPW-00029 until it satisfied 19c’s special-character complexity rules. Use a value that genuinely meets the 19c verifier (uppercase + lowercase + digit + special character), e.g. <StrongPwd#1>.

Step 5 — Post-PDB AutoConfig and the UTL_FILE_DIR trap

Running AutoConfig on the DB tier after the conversion initially failed with a UTL_FILE_DIR error. The trace pointed at /usr/tmp being a symlink rather than a real directory, which the new UTL_FILE_DIR-replacement mechanism in 19c does not tolerate.

The fix is to drive the directory configuration explicitly with txkCfgUtlfileDir.pl. First, manually create the directory-list file with real (non-symlink) paths:

# e.g. /u01/app/oracle/product/19.0.0/dbhome_1/dbs/ebsdb_utlfiledir.txt
# containing two real paths, e.g.:
#   /u01/app/oracle/ebsdb/temp
#   /u01/app/oracle/ebsdb/plsql

Then run the script in its two modes:

perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=setUtlFileDir
perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl -contextfile=<DB_CONTEXT> -mode=syncUtlFileDir

After that, AutoConfig on the DB tier completed cleanly.

Worth flagging for the longer term: UTL_FILE_DIR is deprecated, and Oracle’s recommended direction is to use database directory objects (CREATE DIRECTORY) for PL/SQL file I/O wherever your EBS code level supports it. The txkCfgUtlfileDir.pl mechanism above is the EBS-supported bridge for this release; treat directory objects as the strategic target.

Lessons Learned

• The lowercase, case-sensitive PDB name ("ebsdb" with quotes) propagates into every subsequent ALTER SESSION SET CONTAINER — get it right once and stay consistent.
• Read every compatibility violation; ignore only the benign ones, and never ignore a character-set mismatch.
• Budget real time for password files: pre-stage the ASM password file from the DB home, expect ORA-65066 (apply CONTAINER=ALL) and OPW-00029 (19c complexity rules).
• A symlinked /usr/tmp breaks post-PDB AutoConfig — drive UTL_FILE_DIR explicitly via txkCfgUtlfileDir.pl with real paths.

Phase 4 — Application-tier cutover (overview)

With the database tier healthy at 19c as a PDB, the apps-tier work on appshost is the final phase:

• Run AutoConfig on both filesystems (fs1 and fs2) to point the apps tier at the new 19c PDB connection details.
• Clean up any stale adop sessions before starting fresh — an abandoned session from days earlier will block a new cycle. Run adop cleanup (cleanup_mode=full) and then a fresh adop phase=prepare.
• Validate HugePages, the env files, and Configuration Manager.

I will cover the full apps-tier cutover in detail in a follow-up post, since it deserves its own walkthrough.

Final Validation Checklist

After the upgrade and conversion were complete, the following validations were performed:

✓ Oracle Restart resources online and managed by srvctl
✓ ASM instance and diskgroups mounted
✓ CDB and PDB open in READ WRITE mode
✓ Database listener services registered correctly
✓ AutoConfig completed successfully on database and application tiers
✓ Concurrent Managers started successfully
✓ FNDSM service validated
✓ Internal Monitor running normally
✓ Forms and WebLogic services accessible
✓ adop phase=prepare completed without errors
✓ Invalid objects reviewed and reduced to an acceptable count
✓ Application smoke testing completed
✓ All EBS services registered in the listener
✓ Concurrent Manager processing validated
✓ Test concurrent request submitted and completed
✓ Forms login verified
✓ Workflow Mailer status verified
✓ OACORE, FORMS, and OAFM managed servers healthy
✓ No critical errors in AutoConfig logs
✓ adopscanlog reviewed with no blocking errors
✓ Database and ASM resources managed successfully through srvctl

SET LINESIZE 200 PAGESIZE 100
COL comp_name FORMAT A45
COL version   FORMAT A18
COL status    FORMAT A12
SELECT comp_name, version, status
FROM   dba_registry
ORDER  BY comp_name;

COL name FORMAT A20
SELECT con_id, name, open_mode
FROM   v$pdbs
ORDER  BY con_id;

COL pdb_name FORMAT A20
COL status   FORMAT A12
SELECT pdb_name, status
FROM   dba_pdbs
ORDER  BY pdb_name;

COL owner       FORMAT A22
COL object_type FORMAT A22
SELECT NVL(owner,'(none)') AS owner,
       object_type,
       COUNT(*)             AS invalid_count
FROM   dba_objects
WHERE  status = 'INVALID'
GROUP  BY owner, object_type
ORDER  BY invalid_count DESC;
-- overall total
SELECT COUNT(*) AS total_invalid
FROM   dba_objects
WHERE  status = 'INVALID';

Although every environment is different, the upgrade itself proved relatively straightforward once the underlying Oracle Restart configuration issues were corrected. The majority of effort was spent validating and repairing legacy infrastructure assumptions rather than executing the upgrade utilities themselves.

Key takeaways

1. A GI upgrade is only as clean as the old stack underneath it. Most CLSRSC-318 / CLSRSC-324 / CLSRSC-348 failures trace back to an old HAS stack that was never properly configured or was half-deconfigured. Rebuild it cleanly on the old version first, then upgrade.
2. Verify the bootstrap files yourself. Do not assume the OLR is <hostname>.olr — check cdata/localhost/ and verify with ocrcheck -local.
3. Stale checkpoints block retries silently. Clear ROOTHAS_STACK / ROOTCRS_STACK before re-running rootupgrade.sh.
4. After an Oracle Restart GI upgrade, register ASM (srvctl add asm / srvctl start asm) before you trust any storage-related tool.
5. Use dbua -keepEvents for EBS, and fix X11 forwarding across the user switch before you launch.
6. The PDB conversion’s hardest part is password files — ORA-65066, OPW-00029, and ASM password-file placement. Budget time for it.
7. Post-PDB AutoConfig + UTL_FILE_DIR: a symlinked /usr/tmp will fail it. Drive the directory list explicitly with txkCfgUtlfileDir.pl using real paths.

Looking back, the actual upgrade steps were straightforward. The real effort was understanding and repairing the assumptions the upgrade process makes about the existing environment. Missing bootstrap files, partially deconfigured Oracle Restart components, ASM registration gaps, password-file placement, and UTL_FILE_DIR migration issues consumed far more time than the upgrade binaries themselves. Those lessons are often absent from official documentation — which is exactly why I wanted to document the journey end to end.

References

• Oracle E-Business Suite Release 12.2 — Interoperability Notes for Oracle Database 19c (My Oracle Support)
• Oracle Database 19c Upgrade Guide
• Using UTL_FILE_DIR or Database Directories for PL/SQL File I/O in Oracle E-Business Suite (My Oracle Support)
• Oracle E-Business Suite Multitenant / Pluggable Database (PDB) conversion documentation
• Oracle Grid Infrastructure Installation and Upgrade Guide (Oracle Restart)

(Look up the current My Oracle Support note IDs for your exact code and database levels — they are revised regularly.)

If this write-up helped you during your own Oracle E-Business Suite upgrade, you’re welcome to follow the blog for more real-world Oracle Database, EBS, ADOP, cloning, upgrade, and troubleshooting guides drawn from production experience.

Disclaimer: This post reflects my own hands-on experience and is shared for educational purposes only. All hostnames, database names, diskgroup names, and paths have been genericized. Always test thoroughly in a non-production environment and follow Oracle’s official documentation and your organization’s change-control process before applying any of these steps. The views expressed here are my own and do not represent those of any employer or client.