Alhamdulillah. This is a write-up of a home-lab exercise: upgrading an Oracle Database 19c container database to Oracle AI Database 26ai (23.26.1) with AutoUpgrade. The upgrade itself was the easy part. Getting a 19c source database onto the lab took longer, because the base 19.3 release would not link on Oracle Linux 9. I have kept that part in, because it is the kind of thing that costs an evening if you have not seen it before.
The idea came from a community post on the same upgrade path (Oracle 19c to 26ai Database Upgrade). Everything below is from my own lab run. Hostnames and IP addresses are masked.
Lab environment
| Item | Value |
|---|---|
| Host | Windows desktop, VirtualBox, 64 GB RAM |
| Upgrade VM | labhost19, Oracle Linux 8.10, 4 vCPU, 12 GB RAM, 100 GB disk |
| Source | Oracle Database 19c EE 19.3 (base release, no RU), CDB19 with PDB19 |
| Target | Oracle AI Database 26ai EE 23.26.1 (public download) |
| Tool | AutoUpgrade 26.7.2 (build 261111) |
| Source home | /u01/app/oracle/product/19.0.0/dbhome_1 |
| Target home | /u01/app/oracle/product/26ai/dbhome_1 |
Only public downloads were used: the 19.3 and 23.26.1 zips from oracle.com and the AutoUpgrade jar. No My Oracle Support account was involved, which matters for the first problem.
Part 1: Why 19.3 would not install on Oracle Linux 9
My first plan was to install a 19c home next to the existing 26ai home on an Oracle Linux 9.8 VM. The software-only install failed at the relink stage every time:
[FATAL] Error in invoking target 'libasmclntsh19.ohso libasmperl19.ohso client_sharedlib' of makefile'/u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/ins_rdbms.mk'.
The installer message is generic. The actual linker error is in $ORACLE_HOME/install/make.log, so that is the first file to check.
Attempt 1: gcc was not installed
$ gcc --version | head -1bash: gcc: command not found...
Neither the 26ai nor the 19c preinstall RPM pulled in gcc on this OL9 VM. I installed it, but the relink still failed.
dnf -y install gcc gcc-c++ make glibc-devel elfutils-libelf-devel ksh
Attempt 2: libpthread_nonshared.a
$ grep -n -i -E "error|undefined|cannot find" $ORACLE_HOME/install/make.log136:/usr/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a...genclntsh: Failed to link libclntshcore.so.19.1
The 19.3 home ships its own linker stubs. The libpthread.so stub is a linker script that still points at a file Oracle Linux 9 does not have:
$ cat $ORACLE_HOME/lib/stubs/libpthread.so/* GNU ld script Use the shared library, but some functions are only in the static library, so try that secondarily. */OUTPUT_FORMAT(elf64-x86-64)GROUP ( libpthread.so.0 /usr/lib64/libpthread_nonshared.a )
OL9 uses glibc 2.34, where libpthread was merged into libc and libpthread_nonshared.a is gone. The base 19.3 stubs were built for an older glibc.
Attempts 3 to 5: each stub fix moved the failure
- Moving all stubs aside gave
cannot find -lgcc_s. The link needs the other stubs. - Restoring the stubs and removing only
libpthread.sogavecannot find -lclntshcore. That one was self-inflicted: an earlier failedgenclntshrun had already deletedlibclntshcore.so, so the home was damaged. - On a fresh unzip with only the
libpthread.sostub removed, the client libraries linked, and the failure moved to the Pro*C link with undefinedpthread_*symbols.
/usr/bin/ld: .../lib//libclntsh.so: undefined reference to `pthread_sigmask'/usr/bin/ld: .../lib//libclntshcore.so: undefined reference to `pthread_mutexattr_settype'/usr/bin/ld: .../lib//libclntsh.so: undefined reference to `pthread_join'make[1]: *** [.../precomp/lib/ins_precomp.mk:118: .../precomp/lib/proc] Error 1
At that point I stopped patching stubs. Oracle’s supported route for 19c on Oracle Linux 9 is a newer Release Update applied at install time (runInstaller -applyRU), and RUs need a support account. Even if a hand-edited 19.3 home had linked in the end, it would not have been a sound base for an upgrade test. I built a separate Oracle Linux 8.10 VM instead, where 19.3 installs without workarounds.
One lesson from this part: after a relink failure, do not retry in the same home. Wipe it completely, including hidden folders such as .patch_storage, and unzip again.
Part 2: Building the 19c source on Oracle Linux 8.10
OS packages and directories, as root:
dnf -y install oracle-database-preinstall-19c gcc gcc-c++ make glibc-devel elfutils-libelf-devel ksh unzipmkdir -p /u01/app/oracle/product/19.0.0/dbhome_1 /u01/app/oracle/product/26ai/dbhome_1 /u01/stage /u01/app/oracle/frachown -R oracle:oinstall /u01chmod -R 775 /u01
I kept one environment file per home so the profile never points at the wrong one. A mistake worth mentioning: the first time, the env file did not exist, $ORACLE_HOME was empty, cd $ORACLE_HOME went to the oracle user’s home directory and the 19c zip started extracting there. Check pwd before every unzip.
cat > ~/cdb19.env <<'EOF'export ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1export ORACLE_SID=CDB19export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:/usr/sbin:/usr/bin:/binexport LD_LIBRARY_PATH=$ORACLE_HOME/libEOF. ~/cdb19.envcd $ORACLE_HOME && pwd
Software-only install. CV_ASSUME_DISTID=OL7 gets the base 19.3 installer past its OS version check on OL8:
unzip -q /u01/stage/LINUX.X64_193000_db_home.zipexport CV_ASSUME_DISTID=OL7./runInstaller -silent -ignorePrereqFailure -waitforcompletion \ oracle.install.option=INSTALL_DB_SWONLY \ UNIX_GROUP_NAME=oinstall \ INVENTORY_LOCATION=/u01/app/oraInventory \ ORACLE_HOME=$ORACLE_HOME \ ORACLE_BASE=$ORACLE_BASE \ oracle.install.db.InstallEdition=EE \ oracle.install.db.OSDBA_GROUP=dba \ oracle.install.db.OSOPER_GROUP=oper \ oracle.install.db.OSBACKUPDBA_GROUP=backupdba \ oracle.install.db.OSDGDBA_GROUP=dgdba \ oracle.install.db.OSKMDBA_GROUP=kmdba \ oracle.install.db.OSRACDBA_GROUP=racdba \ DECLINE_SECURITY_UPDATES=true
[WARNING] [INS-13014] Target environment does not meet some optional requirements....Successfully Setup Software with warning(s).
No linking errors on OL8. After orainstRoot.sh and root.sh, I created the listener and the database. ARCHIVELOG mode and a recovery area are needed because AutoUpgrade creates a guaranteed restore point before it starts.
netca -silent -responsefile $ORACLE_HOME/assistants/netca/netca.rspdbca -silent -createDatabase -templateName General_Purpose.dbc \ -gdbName CDB19 -sid CDB19 -createAsContainerDatabase true \ -numberOfPDBs 1 -pdbName PDB19 \ -sysPassword <password> -systemPassword <password> -pdbAdminPassword <password> \ -datafileDestination /u01/app/oracle/oradata -characterSet AL32UTF8 \ -totalMemory 3072 -enableArchive true \ -recoveryAreaDestination /u01/app/oracle/fra -recoveryAreaSize 20480 \ -emConfiguration NONE
Before the upgrade I saved the PDB open state and created a small test table to compare after the upgrade:
SET LINESIZE 150 PAGESIZE 100COL name FORMAT a10COL log_mode FORMAT a12COL open_mode FORMAT a12COL version FORMAT a12SELECT d.name, d.log_mode, d.open_mode, i.version FROM v$database d, v$instance i;ALTER PLUGGABLE DATABASE ALL SAVE STATE;ALTER SESSION SET CONTAINER=PDB19;CREATE USER lab IDENTIFIED BY <password> QUOTA UNLIMITED ON users;GRANT CREATE SESSION, CREATE TABLE TO lab;CREATE TABLE lab.upg_test AS SELECT * FROM dba_objects;SELECT COUNT(*) FROM lab.upg_test;
NAME LOG_MODE OPEN_MODE VERSION---------- ------------ ------------ ------------CDB19 ARCHIVELOG READ WRITE 19.0.0.0.0 COUNT(*)---------- 72356
Saving the PDB state matters here. The post that inspired this run hit “PDB is MOUNTED and will not be processed” in AutoUpgrade. With the state saved, both PDBs were picked up.
I took a VirtualBox snapshot at this point, so the upgrade could be repeated from a clean 19c source.
Part 3: Installing the 26ai home
# as rootdnf -y install oracle-ai-database-preinstall-26ai# as oracle. ~/26ai.envcd $ORACLE_HOME && pwdunzip -oq /u01/stage/LINUX.X64_2326100_db_home.zip./runInstaller -ignorePrereq -waitforcompletion -silent \ -responseFile $ORACLE_HOME/install/response/db_install.rsp \ installOption=INSTALL_DB_SWONLY \ UNIX_GROUP_NAME=oinstall \ INVENTORY_LOCATION=/u01/app/oraInventory \ ORACLE_HOME=$ORACLE_HOME \ ORACLE_BASE=$ORACLE_BASE \ installEdition=EE \ OSDBA=dba OSOPER=oper OSBACKUPDBA=backupdba \ OSDGDBA=dgdba OSKMDBA=kmdba OSRACDBA=racdba
Note that the 26ai installer uses shorter parameter names than 19c: installOption instead of oracle.install.option, installEdition, and OSDBA rather than oracle.install.db.OSDBA_GROUP. On OL8 the preinstall package also pulled in compat-openssl10. The install finished with the same optional-prerequisite warning (INS-13014), followed by root.sh.
Part 4: AutoUpgrade analyze
The VM had internet access, so I downloaded the latest jar instead of using the copy bundled in the target home, and ran it with the JDK from the 26ai home:
mkdir -p /u01/app/oracle/autoupgrade/logscd /u01/app/oracle/autoupgradecurl -sSL -o autoupgrade.jar https://download.oracle.com/otn-pub/otn_software/autoupgrade.jarJAVA=/u01/app/oracle/product/26ai/dbhome_1/jdk/bin/java$JAVA -jar autoupgrade.jar -version
build.version 26.7.2.261111build.date 2026/11/11 04:27:02 +0000build.supported_target_versions 12.2,18,19,21,23,26build.type production
Configuration file (autoupg.cfg):
global.global_log_dir=/u01/app/oracle/autoupgrade/logsupg1.sid=CDB19upg1.source_home=/u01/app/oracle/product/19.0.0/dbhome_1upg1.target_home=/u01/app/oracle/product/26ai/dbhome_1upg1.restoration=YESupg1.timezone_upg=YESupg1.log_dir=/u01/app/oracle/autoupgrade/logs/CDB19
restoration=YES: AutoUpgrade creates a guaranteed restore point before the upgrade.timezone_upg=YES: the time zone file is upgraded as part of the post-upgrade fixups.
$JAVA -jar autoupgrade.jar -config autoupg.cfg -mode analyze -noconsole
1 CDB(s) plus 2 PDB(s) will be analyzedJob 100 database cdb19...Job 100 completed------------------- Final Summary --------------------Number of databases [ 1 ]Jobs finished [1]Jobs failed [0]Checks Report+------------+------+-----------+---------+-------+-------+| Job Number | INFO | RECOMMEND | WARNING | ERROR | Total |+------------+------+-----------+---------+-------+-------+| 100 | 26 | 16 | 12 | 0 | 54 |+------------+------+-----------+---------+-------+-------+
Zero errors. The 12 warnings were four distinct checks, each reported for CDB$ROOT, PDB$SEED and PDB19:
| Warning | Stage | What I did |
|---|---|---|
| Convert traditional audit to unified audit policies | PRECHECKS | Nothing in the lab. On a client system this needs planning before the upgrade, because traditional auditing does not carry on by default. |
| Run outside the automatic maintenance windows | PRECHECKS | Nothing. The run started in the afternoon and finished well before the default window. |
| Upgrade the time zone file (DBMS_DST) | POSTCHECKS | AUTOFIXUP, covered by timezone_upg=YES |
| Recompile objects with timestamp mismatch | POSTCHECKS | AUTOFIXUP |
Part 5: Deploy
I ran deploy under nohup so that a dropped PuTTY session would not kill the job:
nohup $JAVA -jar autoupgrade.jar -config autoupg.cfg -mode deploy -noconsole > deploy.out 2>&1 &tail -8 /u01/app/oracle/autoupgrade/deploy.out
Progress on this VM:
| Time | Stage | Message |
|---|---|---|
| 16:49 | PRECHECKS | Job 101 started, 1 CDB plus 2 PDBs |
| ~16:52 | PREFIXUPS | Executing fixups (restore point already created) |
| 17:00 | DBUPGRADE | 0% Upgraded CDB$ROOT |
| 17:13 | DBUPGRADE | 41% Upgraded CDB$ROOT |
| 18:17 | POSTFIXUPS | Executing fixups (time zone upgrade, recompile) |
| ~18:26 | POSTUPGRADE | Creating final SPFILE, job completed |
About 1 hour 37 minutes in total. During POSTFIXUPS the alert log repeatedly reported slow online redo log writes. On a VirtualBox disk under upgrade load that is expected, and it did not affect the result.
Job 101 completed------------------- Final Summary --------------------Number of databases [ 1 ]Jobs finished [1]Jobs failed [0]Jobs restored [0]Jobs pending [0]---- Drop GRP at your convenience once you consider it is no longer needed ----Drop GRP from CDB19: drop restore point AUTOUPGRADE_9212_CDB19193000
Part 6: Post-upgrade verification
AutoUpgrade had already updated /etc/oratab to the 26ai home. I pointed my env file at the new home and checked the database:
sed -i 's#product/19.0.0/dbhome_1#product/26ai/dbhome_1#' ~/cdb19.env. ~/cdb19.envgrep CDB19 /etc/oratabCDB19:/u01/app/oracle/product/26ai/dbhome_1:N
SET LINESIZE 160 PAGESIZE 100COL version_full FORMAT a15COL comp_id FORMAT a12COL status FORMAT a10COL name FORMAT a40SELECT version_full FROM v$instance;SHOW PDBSSELECT comp_id, NVL(version_full,'-') version_full, status FROM dba_registry ORDER BY 1;SELECT version FROM v$timezone_file;SELECT name FROM v$restore_point;SHOW PARAMETER compatibleALTER SESSION SET CONTAINER=PDB19;SELECT COUNT(*) FROM lab.upg_test;SELECT COUNT(*) invalid_objs FROM dba_objects WHERE status='INVALID';
VERSION_FULL---------------23.26.1.0.0 CON_ID CON_NAME OPEN MODE RESTRICTED---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 PDB19 READ WRITE NOCOMP_ID VERSION_FULL STATUS------------ --------------- ----------APS 23.26.1.0.0 VALIDCATALOG 23.26.1.0.0 VALIDCATJAVA 23.26.1.0.0 VALIDCATPROC 23.26.1.0.0 VALIDCONTEXT 23.26.1.0.0 VALIDDV 23.26.1.0.0 VALIDJAVAVM 23.26.1.0.0 VALIDOLS 23.26.1.0.0 VALIDOWM 23.26.1.0.0 VALIDRAC 23.26.1.0.0 OPTION OFFSDO 23.26.1.0.0 VALIDXDB 23.26.1.0.0 VALIDXML 23.26.1.0.0 VALIDXOQ 23.26.1.0.0 VALID VERSION---------- 43NAME----------------------------------------AUTOUPGRADE_9212_CDB19193000NAME TYPE VALUE------------------------------------ ----------- ------------------------------compatible string 19.0.0 COUNT(*)---------- 72356INVALID_OBJS------------ 0
- Version 23.26.1.0.0, PDB19 open READ WRITE.
- All components VALID. RAC shows OPTION OFF, which is expected on a single instance.
- Time zone file upgraded to version 43.
- The test table still has 72,356 rows, and PDB19 has no invalid objects.
COMPATIBLEis still 19.0.0, so the restore point could still be used to go back.
Part 7: Listener, restore point and COMPATIBLE
The listener was still running from the 19c home. I stopped it, copied the network files to the 26ai home, updated the paths and started it from the new home:
/u01/app/oracle/product/19.0.0/dbhome_1/bin/lsnrctl stopcp /u01/app/oracle/product/19.0.0/dbhome_1/network/admin/{listener.ora,sqlnet.ora,tnsnames.ora} $ORACLE_HOME/network/admin/sed -i 's#product/19.0.0/dbhome_1#product/26ai/dbhome_1#g' $ORACLE_HOME/network/admin/*.oralsnrctl start
The listener showed “supports no services” at first. local_listener was set to the alias LISTENER_CDB19, which dbca had written into tnsnames.ora, so copying that file to the new home mattered. After ALTER SYSTEM REGISTER the services appeared:
Service "CDB19" has 1 instance(s). Instance "CDB19", status READY, has 1 handler(s) for this service...Service "CDB19XDB" has 1 instance(s). Instance "CDB19", status READY, has 1 handler(s) for this service...Service "pdb19" has 1 instance(s). Instance "CDB19", status READY, has 1 handler(s) for this service...
With the checks done, I dropped the restore point so it would stop holding flashback logs in the recovery area:
DROP RESTORE POINT AUTOUPGRADE_9212_CDB19193000;SELECT COUNT(*) FROM v$restore_point;
I left COMPATIBLE at 19.0.0. Raising it enables the new release’s features, but it cannot be lowered again, which rules out a downgrade. In a real project that decision comes after application testing. In the lab I took a VirtualBox snapshot first, so I can raise it later and still roll back the VM.
Lessons learned
- Check the certification of the exact release on the exact OS. “19c on OL9” and “base 19.3 on OL9” are not the same thing.
- When
runInstallerreports a relink FATAL, read$ORACLE_HOME/install/make.log. The installer log only repeats the generic message. - After a failed relink, start from a fresh unzip. Retrying in a damaged home produces new errors that have nothing to do with the original cause.
- Put a time limit on workarounds. After a few rounds of stub fixes, moving to a certified platform was faster and gave a cleaner result.
- Save the PDB open state before AutoUpgrade so every PDB is processed.
- Run deploy under
nohup(orscreen/tmux) and followdeploy.outandstatus.log. - Moving the listener is not part of the AutoUpgrade job. Copy
tnsnames.oratoo iflocal_listeneruses an alias. - Keep the guaranteed restore point until verification is done, and treat
COMPATIBLEas a separate, later decision.
ما شاء الله, a clean result in the end: 0 errors in analyze, 0 failed jobs in deploy, and the test data intact on 26ai. Next in this lab is Oracle APEX on the 26ai database.