Alhamdulillah, the October 2026 CPU cycle for one of our 19c development databases is complete. Most of it was routine: GI RU, DB RU and OJVM on both nodes, then datapatch and recompilation. The one surprise came on node 2, where opatchauto failed halfway through the DB home with No space left on device. The space had been eaten by more than two years of rotated ASM network listener logs.
This post walks through the full runbook with the real outputs, the failure, the cleanup, and the opatchauto resume that finished it. Hostnames, database and instance names, Oracle home paths and session IDs have been replaced with generic placeholders.
Environment
| Cluster | Oracle Grid Infrastructure 19c, two nodes (node1, node2) |
| Database | Oracle Database 19c Standard Edition 2, non-CDB, managed by clusterware |
| From → To | 19.32.0.0.260721 → 19.33.0.0.261020 |
| Patch bundle | Combo 40121665 = GI RU 39922757 + OJVM RU 39803650 |
| OPatch | 12.2.0.1.52 |
Placeholders used throughout: $GRID_HOME = Grid Infrastructure home, $ORACLE_HOME = database home, $PATCH_TOP = directory where the combo patch was unzipped, <DB_NAME> / <INSTANCE> = database and instance names.
The GI RU 39922757 is a bundle of sub-patches. In this run, opatchauto applied 39915792 (Database RU), 39922783 (OCW RU) and 39922802 to the GI home, and only 39915792 and 39922783 to the DB home. Sub-patches 39107825 and 39107855 were skipped on both homes as “not required” or “not applicable to this target type”. That’s expected behaviour, not an error.
Runbook sequence
The same patch sequence ran on each node. Node 2 started while node 1 was still on its OJVM and datapatch steps, so the two nodes overlapped.
- Pre-patch baseline (database state, invalid objects, SQL patch registry, components)
- Client confirms the application is stopped, then shut down the database from node 1
- Stop clusterware on both nodes and take TAR backups of the GI and DB homes
- Node 1: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply) → start the database → datapatch → utlrp
- Node 2, in parallel: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply)
- Post-check and compare with the baseline
Step 1: Pre-patch baseline
SET LINES 230 PAGES 100
COL object_name FOR a40
COL object_type FOR a20
COL owner FOR a20
COL last_ddl FOR a22
SELECT object_name, object_type, owner,
TO_CHAR(last_ddl_time,'DD-MON-YYYY HH24:MI:SS') last_ddl, status
FROM dba_objects
WHERE status = 'INVALID'
ORDER BY owner;
SET LINES 300
COL action_time FOR a30
COL action FOR a15
COL status FOR a15
COL description FOR a60
SELECT patch_id, action, status, action_time, description
FROM dba_registry_sqlpatch
ORDER BY action_time;
COL comp_id FOR a12
COL comp_name FOR a44 HEA 'Component'
COL version FOR a17 HEA 'Version'
COL status FOR a17 HEA 'Status'
SELECT comp_id, comp_name, version, status FROM dba_registry;
Baseline findings:
- The database was
READ WRITE/PRIMARY. - 29 invalid objects, all in two application schemas, most with a LAST_DDL_TIME from 2015 to 2023. Nothing in SYS or SYSTEM. Recording these before you patch saves an argument later about whether patching caused them.
- The last registry entries were the July cycle: OJVM 39222882 and DB RU 39472050 (19.32), both SUCCESS.
- All components were VALID or OPTION OFF.
Step 2: Application stop and database shutdown
Nothing was touched until the client confirmed the application was stopped. Then, from node 1:
$ . ~/<DB_NAME>.env
$ sqlplus / as sysdba
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
Step 3: Stop clusterware and back up the homes
On each node, as root from $GRID_HOME/bin:
# ./crsctl check crs
# ./crsctl stop has
...
CRS-4133: Oracle High Availability Services has been stopped.
With the stack down, a TAR backup was taken of both the GI home and the DB home on each node. That’s your fastest way back if a home gets damaged mid-patch. An example of the command form (adjust the target location to wherever you have space):
# tar -czpf /backup/grid_home_$(hostname -s)_$(date +%Y%m%d).tar.gz -C $GRID_HOME .
# tar -czpf /backup/db_home_$(hostname -s)_$(date +%Y%m%d).tar.gz -C $ORACLE_HOME .
The stack was then started again with crsctl start has ahead of opatchauto. Right after the start, crsctl check crs showed the usual intermediate states for a few minutes:
# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
# ./crsctl check crs (a few minutes later)
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
CSS and EVM came up first and CRSD came up last. Give it time and keep checking before you assume something is broken.
Step 4: Node 1 GI home
# $GRID_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $GRID_HOME
Performing prepatch operations on CRS - bringing down CRS service on home $GRID_HOME
CRS service brought down successfully on home $GRID_HOME
Start applying binary patch on home $GRID_HOME
Binary patch applied successfully on home $GRID_HOME
Successfully executed rootadd_rdbms.sh on home $GRID_HOME
CRS service started successfully on home $GRID_HOME
OPatchAuto successful.
==Following patches were SUCCESSFULLY applied:
Patch: $PATCH_TOP/39922757/39915792
Patch: $PATCH_TOP/39922757/39922783
Patch: $PATCH_TOP/39922757/39922802
Time taken to complete the session 22 minutes, 51 seconds
Step 5: Node 1 DB home
# $ORACLE_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $ORACLE_HOME
Binary patch applied successfully on home $ORACLE_HOME
...
OPatchAuto successful.
==Following patches were SUCCESSFULLY applied:
Patch: $PATCH_TOP/39922757/39915792
Patch: $PATCH_TOP/39922757/39922783
[WARNING] The database instance '<INSTANCE>' from '$ORACLE_HOME', in host 'node1'
is not running. SQL changes, if any, will not be applied.
Time taken to complete the session 14 minutes, 55 seconds
The warning is expected here. The database was deliberately left down because OJVM still had to go on, so the SQL changes are applied later by a manual datapatch run.
Step 6: Node 1 OJVM, plus a small environment gotcha
The first attempt as oracle returned bash: opatch: command not found. The env file and export PATH lines had been pasted in together with sudo su - oracle, so they never ran in the shell that was actually used. ORACLE_HOME was empty and $ORACLE_HOME/OPatch expanded to /OPatch. The fix was to source the env file again in the current shell and check it:
$ . ~/<DB_NAME>.env
$ echo $ORACLE_HOME # must not be empty
$ export PATH=$ORACLE_HOME/OPatch:$PATH
$ which opatch
Then a conflict check, followed by the apply:
$ cd $PATCH_TOP/39803650
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$ opatch apply
OPatch continues with these patches: 39803650
All checks passed.
Applying interim patch '39803650' to OH '$ORACLE_HOME'
Patching component oracle.javavm.server, 19.0.0.0.0...
Patching component oracle.javavm.server.core, 19.0.0.0.0...
Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...
Patching component oracle.rdbms, 19.0.0.0.0...
Patching component oracle.javavm.client, 19.0.0.0.0...
Patch 39803650 successfully applied.
Sub-set patch [39222882] has become inactive due to the application of a super-set patch [39803650].
Please refer to Doc ID 2161861.1 for any possible further required actions.
OPatch succeeded.
$ opatch lsinventory | grep 39803650
Patch 39803650 : applied on Tue Nov 10 21:10:50 2026
Patch description: "OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)"
The “sub-set patch has become inactive” line is OPatch noting that the 19.32 OJVM is superseded by 19.33. datapatch handles the SQL-side rollback of 39222882 in the next step.
Step 7: Node 1 startup, datapatch and utlrp
$ sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production
Version 19.33.0.0.0
Connected to an idle instance.
SQL> startup;
Database mounted.
Database opened.
SQL> exit
$ cd $ORACLE_HOME/OPatch
$ ./datapatch -verbose
Current state of release update SQL patches:
Binary registry:
19.33.0.0.0 Release_Update 261009085306: Installed
SQL registry:
Applied 19.32.0.0.0 Release_Update 260705220710 successfully on 31-AUG-26
Installation queue:
The following interim patches will be rolled back:
39222882 (OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882))
Patch 39915792 (Database Release Update : 19.33.0.0.261020 (39915792)):
Apply from 19.32.0.0.0 Release_Update 260705220710 to 19.33.0.0.0 Release_Update 261009085306
The following interim patches will be applied:
39803650 (OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650))
Patch 39222882 rollback: SUCCESS (no errors)
Patch 39915792 apply: SUCCESS (no errors)
Patch 39803650 apply: SUCCESS (no errors)
SQL Patching tool complete
datapatch needs the database open, so it runs once, from one node, after startup. Here that was node 1, with the 19.33 binaries (DB RU and OJVM) already in place while node 2 was still being patched. It doesn’t need to be repeated on node 2. Invalid objects were then recompiled from the same node:
$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus / as sysdba
SQL> @utlrp.sql
Step 8: Node 2 GI home (in parallel)
Node 2’s GI patch started right after node 1’s DB home finished, and ran alongside node 1’s OJVM and datapatch work. The command was the same as on node 1. It completed successfully in 27 minutes 57 seconds with the same three sub-patches applied.
Step 9: Node 2 DB home fails with “No space left on device”
Start applying binary patch on home $ORACLE_HOME
Failed while applying binary patches on home $ORACLE_HOME
==Following patches FAILED in apply:
Patch: $PATCH_TOP/39922757/39915792
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException:
ApplySession failed in system modification phase... 'java.io.IOException: No space left on device'
Patch: $PATCH_TOP/39922757/39922783
Reason: ... 'java.io.IOException: No space left on device'
After fixing the cause of failure Run opatchauto resume
OPATCHAUTO-68061: The orchestration engine failed.
OPatchAuto failed.
opatchauto failed with error code 42
The filesystem holding both homes and ORACLE_BASE was at 97%:
# df -h /u01
Filesystem Size Used Avail Use% Mounted on
<oracle_fs> 130G 126G 4.5G 97% /u01
Node 1 had patched without any trouble, so free space on node 2 hadn’t been checked separately. That’s the first lesson.
Finding what was using the space
Drilling down with du -sh * led straight into the listener ADR:
$ORACLE_BASE 35G
diag 26G
tnslsnr 20G
<node2>/asmnet1lsnr_asm 15G
alert 9.4G
trace 5.3G
<node2>/listener_scan1 3.2G
Inside asmnet1lsnr_asm there were 32 rotated XML alert files of about 300 MB each (log_1.xml to log_32.xml) and 32 rotated text logs of about 175 MB each (asmnet1lsnr_asm_1.log to _32.log). The oldest went back to March 2024. Nothing had ever purged them.
Cleanup
Only the rotated, numbered files were removed. The current log.xml and asmnet1lsnr_asm.log that the listener was writing to were left in place.
# cd $ORACLE_BASE/diag/tnslsnr/<node2>/asmnet1lsnr_asm/alert
# rm -f log_*.xml
# df -h /u01
<oracle_fs> 130G 117G 14G 90% /u01
# cd ../trace
# rm -f asmnet1lsnr_asm_*.log
# df -h /u01
<oracle_fs> 130G 111G 20G 86% /u01
That took free space from 4.5 GB to 20 GB. A cleaner and more repeatable option is to purge through ADRCI rather than rm. This is a recommendation, not what was run in this window:
$ adrci
adrci> show homes
adrci> set home diag/tnslsnr/<node2>/asmnet1lsnr_asm
adrci> purge -age 43200 -type alert # older than 30 days (minutes)
adrci> purge -age 43200 -type trace
Rotated listener_scan1 logs (about 2.9 GB, back to 2024) are still there and are on the housekeeping list.
opatchauto resume
Running opatchauto resume from root’s shell returned command not found, even from inside the OPatch directory, because OPatch isn’t in root’s PATH. Use the full path or ./:
# cd $ORACLE_HOME/OPatch
# ./opatchauto resume
Resuming existing session with id <session_id>
Start applying binary patch on home $ORACLE_HOME
Binary patch applied successfully on home $ORACLE_HOME
Successfully executed rootadd_rdbms.sh on home $ORACLE_HOME
Postpatch operation completed successfully on home $ORACLE_HOME
OPatchAuto successful.
==Following patches were SUCCESSFULLY applied:
Patch: $PATCH_TOP/39922757/39915792
Patch: $PATCH_TOP/39922757/39922783
Time taken to complete the session 10 minutes, 28 seconds
opatchauto picked up the saved session and carried on from the failed binary step. There was no need to roll back or start over.
Step 10: Node 2 OJVM
The same prereq and opatch apply from the 39803650 directory as on node 1, with the same output: Patch 39803650 successfully applied and the same sub-set patch notice for 39222882. No datapatch was needed on node 2, because the SQL changes had already been applied to the database from node 1 in Step 7.
Step 11: Post-check against the baseline
$ $ORACLE_HOME/OPatch/opatch lspatches
39803650;OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
39915792;Database Release Update : 19.33.0.0.261020 (39915792)
OPatch succeeded.
SQL patch registry, last rows:
PATCH_ID ACTION STATUS ACTION_TIME DESCRIPTION
---------- --------------- --------------- ------------------------------ ----------------------------------------------------
39222882 APPLY SUCCESS 31-AUG-26 09.02.59.285596 PM OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
39472050 APPLY SUCCESS 31-AUG-26 09.03.55.262038 PM Database Release Update : 19.32.0.0.260721 (39472050)
39222882 ROLLBACK SUCCESS 10-NOV-26 09.18.28.051415 PM OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
39803650 APPLY SUCCESS 10-NOV-26 09.18.38.798776 PM OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
39915792 APPLY SUCCESS 10-NOV-26 09.19.41.446625 PM Database Release Update : 19.33.0.0.261020 (39915792)
- Invalid objects: 29 before patching,
no rows selectedafter utlrp. - All
dba_registrycomponents were VALID or OPTION OFF, the same as the baseline. - SQL*Plus now reports
Version 19.33.0.0.0. - Crontab entries were re-enabled and the application team was asked to validate.
Two rows that sometimes cause confusion: OCW 39922783 shows in lspatches but not in dba_registry_sqlpatch, because it has no SQL component. The 19.32 DB RU 39472050 has no ROLLBACK row, because a newer RU is applied on top of the older one rather than rolling it back.
Timeline
Clock times are from the opatchauto, opatch and datapatch outputs, and show where the two nodes overlapped.
| Start | Node 1 | Node 2 | Result |
|---|---|---|---|
| 20:21 | GI home (22m 51s) | — | Success |
| 20:44 | DB home (14m 55s) | — | Success (SQL deferred) |
| 21:01 | — | GI home (27m 57s) | Success |
| 21:05 | OJVM opatch apply | GI home running | Success |
| 21:15 | Startup + datapatch (done 21:19), then utlrp | GI home running | Success |
| 21:29 | — | DB home, first attempt (10m 56s) | Failed: no space |
| 21:40 | — | Log cleanup | 4.5 GB → 20 GB free |
| 21:47 | — | opatchauto resume (10m 28s) | Success |
| 22:08 | — | OJVM opatch apply | Success |
Lessons learned
- Check free space on every node, not just the first. Node 1 was fine and node 2 was at 97% on the same mount. A quick
df -hon the homes filesystem of each node belongs in the precheck, especially when you patch the nodes in parallel and won’t see node 2’s problem until it’s mid-apply. - Look at the listener ADR homes, not only the database ones. Most housekeeping scripts cover the RDBMS alert log and trace files. Here the ASM network listener alone had 15 GB of rotated files. Add the
tnslsnrhomes to a scheduled ADRCI purge. - Don’t panic when opatchauto fails mid-apply. Fix the cause and run
opatchauto resumefrom the same home with the full path. The session ID is kept and it continues from the failed step. The TAR backups from Step 3 were there if it hadn’t. - Source the environment in the shell you actually use. Pasting the env lines together with
sudo su - oracleleaves ORACLE_HOME empty. Runecho $ORACLE_HOMEbefore trustingwhich opatch. - Take a baseline first. Recording the 29 application-schema invalids before patching, then running utlrp after datapatch, made the post-check comparison simple.
ما شاء الله, one more cycle closed cleanly in spite of the space issue. If you run 19c on Grid Infrastructure, check your listener diag directories before the next CPU window.
Leave a Reply