Tag: Grid Infrastructure

  • Oracle 19c October 2026 RU (19.32 → 19.33) on Grid Infrastructure: Recovering from “No space left on device” Mid-opatchauto

    Alhamdulillah, the October 2026 CPU cycle for one of our 19c development databases is complete. Most of it was routine: GI RU, DB RU and OJVM on both nodes, then datapatch and recompilation. The one surprise came on node 2, where opatchauto failed halfway through the DB home with No space left on device. The space had been eaten by more than two years of rotated ASM network listener logs.

    This post walks through the full runbook with the real outputs, the failure, the cleanup, and the opatchauto resume that finished it. Hostnames, database and instance names, Oracle home paths and session IDs have been replaced with generic placeholders.

    Environment

    ClusterOracle Grid Infrastructure 19c, two nodes (node1, node2)
    DatabaseOracle Database 19c Standard Edition 2, non-CDB, managed by clusterware
    From → To19.32.0.0.260721 → 19.33.0.0.261020
    Patch bundleCombo 40121665 = GI RU 39922757 + OJVM RU 39803650
    OPatch12.2.0.1.52

    Placeholders used throughout: $GRID_HOME = Grid Infrastructure home, $ORACLE_HOME = database home, $PATCH_TOP = directory where the combo patch was unzipped, <DB_NAME> / <INSTANCE> = database and instance names.

    The GI RU 39922757 is a bundle of sub-patches. In this run, opatchauto applied 39915792 (Database RU), 39922783 (OCW RU) and 39922802 to the GI home, and only 39915792 and 39922783 to the DB home. Sub-patches 39107825 and 39107855 were skipped on both homes as “not required” or “not applicable to this target type”. That’s expected behaviour, not an error.

    Runbook sequence

    The same patch sequence ran on each node. Node 2 started while node 1 was still on its OJVM and datapatch steps, so the two nodes overlapped.

    1. Pre-patch baseline (database state, invalid objects, SQL patch registry, components)
    2. Client confirms the application is stopped, then shut down the database from node 1
    3. Stop clusterware on both nodes and take TAR backups of the GI and DB homes
    4. Node 1: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply) → start the database → datapatch → utlrp
    5. Node 2, in parallel: GI home (opatchauto) → DB home (opatchauto) → OJVM (prereq + opatch apply)
    6. Post-check and compare with the baseline

    Step 1: Pre-patch baseline

    SET LINES 230 PAGES 100
    COL object_name FOR a40
    COL object_type FOR a20
    COL owner       FOR a20
    COL last_ddl    FOR a22
    SELECT object_name, object_type, owner,
           TO_CHAR(last_ddl_time,'DD-MON-YYYY HH24:MI:SS') last_ddl, status
    FROM   dba_objects
    WHERE  status = 'INVALID'
    ORDER  BY owner;
    
    SET LINES 300
    COL action_time FOR a30
    COL action      FOR a15
    COL status      FOR a15
    COL description FOR a60
    SELECT patch_id, action, status, action_time, description
    FROM   dba_registry_sqlpatch
    ORDER  BY action_time;
    
    COL comp_id   FOR a12
    COL comp_name FOR a44 HEA 'Component'
    COL version   FOR a17 HEA 'Version'
    COL status    FOR a17 HEA 'Status'
    SELECT comp_id, comp_name, version, status FROM dba_registry;

    Baseline findings:

    • The database was READ WRITE / PRIMARY.
    • 29 invalid objects, all in two application schemas, most with a LAST_DDL_TIME from 2015 to 2023. Nothing in SYS or SYSTEM. Recording these before you patch saves an argument later about whether patching caused them.
    • The last registry entries were the July cycle: OJVM 39222882 and DB RU 39472050 (19.32), both SUCCESS.
    • All components were VALID or OPTION OFF.

    Step 2: Application stop and database shutdown

    Nothing was touched until the client confirmed the application was stopped. Then, from node 1:

    $ . ~/<DB_NAME>.env
    $ sqlplus / as sysdba
    SQL> shut immediate;
    Database closed.
    Database dismounted.
    ORACLE instance shut down.

    Step 3: Stop clusterware and back up the homes

    On each node, as root from $GRID_HOME/bin:

    # ./crsctl check crs
    # ./crsctl stop has
    ...
    CRS-4133: Oracle High Availability Services has been stopped.

    With the stack down, a TAR backup was taken of both the GI home and the DB home on each node. That’s your fastest way back if a home gets damaged mid-patch. An example of the command form (adjust the target location to wherever you have space):

    # tar -czpf /backup/grid_home_$(hostname -s)_$(date +%Y%m%d).tar.gz -C $GRID_HOME .
    # tar -czpf /backup/db_home_$(hostname -s)_$(date +%Y%m%d).tar.gz   -C $ORACLE_HOME .

    The stack was then started again with crsctl start has ahead of opatchauto. Right after the start, crsctl check crs showed the usual intermediate states for a few minutes:

    # ./crsctl check crs
    CRS-4638: Oracle High Availability Services is online
    CRS-4535: Cannot communicate with Cluster Ready Services
    CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
    CRS-4534: Cannot communicate with Event Manager
    
    # ./crsctl check crs        (a few minutes later)
    CRS-4638: Oracle High Availability Services is online
    CRS-4537: Cluster Ready Services is online
    CRS-4529: Cluster Synchronization Services is online
    CRS-4533: Event Manager is online

    CSS and EVM came up first and CRSD came up last. Give it time and keep checking before you assume something is broken.

    Step 4: Node 1 GI home

    # $GRID_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $GRID_HOME
    
    Performing prepatch operations on CRS - bringing down CRS service on home $GRID_HOME
    CRS service brought down successfully on home $GRID_HOME
    Start applying binary patch on home $GRID_HOME
    Binary patch applied successfully on home $GRID_HOME
    Successfully executed rootadd_rdbms.sh on home $GRID_HOME
    CRS service started successfully on home $GRID_HOME
    
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    Patch: $PATCH_TOP/39922757/39922802
    
    Time taken to complete the session 22 minutes, 51 seconds

    Step 5: Node 1 DB home

    # $ORACLE_HOME/OPatch/opatchauto apply $PATCH_TOP/39922757 -oh $ORACLE_HOME
    
    Binary patch applied successfully on home $ORACLE_HOME
    ...
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    
    [WARNING] The database instance '<INSTANCE>' from '$ORACLE_HOME', in host 'node1'
    is not running. SQL changes, if any, will not be applied.
    
    Time taken to complete the session 14 minutes, 55 seconds

    The warning is expected here. The database was deliberately left down because OJVM still had to go on, so the SQL changes are applied later by a manual datapatch run.

    Step 6: Node 1 OJVM, plus a small environment gotcha

    The first attempt as oracle returned bash: opatch: command not found. The env file and export PATH lines had been pasted in together with sudo su - oracle, so they never ran in the shell that was actually used. ORACLE_HOME was empty and $ORACLE_HOME/OPatch expanded to /OPatch. The fix was to source the env file again in the current shell and check it:

    $ . ~/<DB_NAME>.env
    $ echo $ORACLE_HOME          # must not be empty
    $ export PATH=$ORACLE_HOME/OPatch:$PATH
    $ which opatch

    Then a conflict check, followed by the apply:

    $ cd $PATCH_TOP/39803650
    $ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
    $ opatch apply
    
    OPatch continues with these patches:   39803650
    All checks passed.
    Applying interim patch '39803650' to OH '$ORACLE_HOME'
    Patching component oracle.javavm.server, 19.0.0.0.0...
    Patching component oracle.javavm.server.core, 19.0.0.0.0...
    Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...
    Patching component oracle.rdbms, 19.0.0.0.0...
    Patching component oracle.javavm.client, 19.0.0.0.0...
    Patch 39803650 successfully applied.
    Sub-set patch [39222882] has become inactive due to the application of a super-set patch [39803650].
    Please refer to Doc ID 2161861.1 for any possible further required actions.
    OPatch succeeded.
    
    $ opatch lsinventory | grep 39803650
    Patch  39803650     : applied on Tue Nov 10 21:10:50 2026
    Patch description:  "OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)"

    The “sub-set patch has become inactive” line is OPatch noting that the 19.32 OJVM is superseded by 19.33. datapatch handles the SQL-side rollback of 39222882 in the next step.

    Step 7: Node 1 startup, datapatch and utlrp

    $ sqlplus / as sysdba
    SQL*Plus: Release 19.0.0.0.0 - Production
    Version 19.33.0.0.0
    Connected to an idle instance.
    SQL> startup;
    Database mounted.
    Database opened.
    SQL> exit
    
    $ cd $ORACLE_HOME/OPatch
    $ ./datapatch -verbose
    
    Current state of release update SQL patches:
      Binary registry:
        19.33.0.0.0 Release_Update 261009085306: Installed
      SQL registry:
        Applied 19.32.0.0.0 Release_Update 260705220710 successfully on 31-AUG-26
    
    Installation queue:
      The following interim patches will be rolled back:
        39222882 (OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882))
      Patch 39915792 (Database Release Update : 19.33.0.0.261020 (39915792)):
        Apply from 19.32.0.0.0 Release_Update 260705220710 to 19.33.0.0.0 Release_Update 261009085306
      The following interim patches will be applied:
        39803650 (OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650))
    
    Patch 39222882 rollback: SUCCESS  (no errors)
    Patch 39915792 apply: SUCCESS     (no errors)
    Patch 39803650 apply: SUCCESS     (no errors)
    SQL Patching tool complete

    datapatch needs the database open, so it runs once, from one node, after startup. Here that was node 1, with the 19.33 binaries (DB RU and OJVM) already in place while node 2 was still being patched. It doesn’t need to be repeated on node 2. Invalid objects were then recompiled from the same node:

    $ cd $ORACLE_HOME/rdbms/admin
    $ sqlplus / as sysdba
    SQL> @utlrp.sql

    Step 8: Node 2 GI home (in parallel)

    Node 2’s GI patch started right after node 1’s DB home finished, and ran alongside node 1’s OJVM and datapatch work. The command was the same as on node 1. It completed successfully in 27 minutes 57 seconds with the same three sub-patches applied.

    Step 9: Node 2 DB home fails with “No space left on device”

    Start applying binary patch on home $ORACLE_HOME
    Failed while applying binary patches on home $ORACLE_HOME
    
    ==Following patches FAILED in apply:
    Patch: $PATCH_TOP/39922757/39915792
    Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException:
    ApplySession failed in system modification phase... 'java.io.IOException: No space left on device'
    
    Patch: $PATCH_TOP/39922757/39922783
    Reason: ... 'java.io.IOException: No space left on device'
    
    After fixing the cause of failure Run opatchauto resume
    
    OPATCHAUTO-68061: The orchestration engine failed.
    OPatchAuto failed.
     opatchauto failed with error code 42

    The filesystem holding both homes and ORACLE_BASE was at 97%:

    # df -h /u01
    Filesystem    Size  Used Avail Use% Mounted on
    <oracle_fs>   130G  126G  4.5G  97% /u01

    Node 1 had patched without any trouble, so free space on node 2 hadn’t been checked separately. That’s the first lesson.

    Finding what was using the space

    Drilling down with du -sh * led straight into the listener ADR:

    $ORACLE_BASE                                  35G
      diag                                        26G
        tnslsnr                                   20G
          <node2>/asmnet1lsnr_asm                 15G
            alert                                9.4G
            trace                                5.3G
          <node2>/listener_scan1                3.2G

    Inside asmnet1lsnr_asm there were 32 rotated XML alert files of about 300 MB each (log_1.xml to log_32.xml) and 32 rotated text logs of about 175 MB each (asmnet1lsnr_asm_1.log to _32.log). The oldest went back to March 2024. Nothing had ever purged them.

    Cleanup

    Only the rotated, numbered files were removed. The current log.xml and asmnet1lsnr_asm.log that the listener was writing to were left in place.

    # cd $ORACLE_BASE/diag/tnslsnr/<node2>/asmnet1lsnr_asm/alert
    # rm -f log_*.xml
    # df -h /u01
    <oracle_fs>   130G  117G   14G  90% /u01
    
    # cd ../trace
    # rm -f asmnet1lsnr_asm_*.log
    # df -h /u01
    <oracle_fs>   130G  111G   20G  86% /u01

    That took free space from 4.5 GB to 20 GB. A cleaner and more repeatable option is to purge through ADRCI rather than rm. This is a recommendation, not what was run in this window:

    $ adrci
    adrci> show homes
    adrci> set home diag/tnslsnr/<node2>/asmnet1lsnr_asm
    adrci> purge -age 43200 -type alert     # older than 30 days (minutes)
    adrci> purge -age 43200 -type trace

    Rotated listener_scan1 logs (about 2.9 GB, back to 2024) are still there and are on the housekeeping list.

    opatchauto resume

    Running opatchauto resume from root’s shell returned command not found, even from inside the OPatch directory, because OPatch isn’t in root’s PATH. Use the full path or ./:

    # cd $ORACLE_HOME/OPatch
    # ./opatchauto resume
    
    Resuming existing session with id <session_id>
    Start applying binary patch on home $ORACLE_HOME
    Binary patch applied successfully on home $ORACLE_HOME
    Successfully executed rootadd_rdbms.sh on home $ORACLE_HOME
    Postpatch operation completed successfully on home $ORACLE_HOME
    
    OPatchAuto successful.
    
    ==Following patches were SUCCESSFULLY applied:
    Patch: $PATCH_TOP/39922757/39915792
    Patch: $PATCH_TOP/39922757/39922783
    
    Time taken to complete the session 10 minutes, 28 seconds

    opatchauto picked up the saved session and carried on from the failed binary step. There was no need to roll back or start over.

    Step 10: Node 2 OJVM

    The same prereq and opatch apply from the 39803650 directory as on node 1, with the same output: Patch 39803650 successfully applied and the same sub-set patch notice for 39222882. No datapatch was needed on node 2, because the SQL changes had already been applied to the database from node 1 in Step 7.

    Step 11: Post-check against the baseline

    $ $ORACLE_HOME/OPatch/opatch lspatches
    39803650;OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
    39922783;OCW RELEASE UPDATE 19.33.0.0.0 (39922783)
    39915792;Database Release Update : 19.33.0.0.261020 (39915792)
    OPatch succeeded.

    SQL patch registry, last rows:

      PATCH_ID ACTION          STATUS          ACTION_TIME                    DESCRIPTION
    ---------- --------------- --------------- ------------------------------ ----------------------------------------------------
      39222882 APPLY           SUCCESS         31-AUG-26 09.02.59.285596 PM   OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
      39472050 APPLY           SUCCESS         31-AUG-26 09.03.55.262038 PM   Database Release Update : 19.32.0.0.260721 (39472050)
      39222882 ROLLBACK        SUCCESS         10-NOV-26 09.18.28.051415 PM   OJVM RELEASE UPDATE: 19.32.0.0.260721 (39222882)
      39803650 APPLY           SUCCESS         10-NOV-26 09.18.38.798776 PM   OJVM RELEASE UPDATE: 19.33.0.0.261020 (39803650)
      39915792 APPLY           SUCCESS         10-NOV-26 09.19.41.446625 PM   Database Release Update : 19.33.0.0.261020 (39915792)
    • Invalid objects: 29 before patching, no rows selected after utlrp.
    • All dba_registry components were VALID or OPTION OFF, the same as the baseline.
    • SQL*Plus now reports Version 19.33.0.0.0.
    • Crontab entries were re-enabled and the application team was asked to validate.

    Two rows that sometimes cause confusion: OCW 39922783 shows in lspatches but not in dba_registry_sqlpatch, because it has no SQL component. The 19.32 DB RU 39472050 has no ROLLBACK row, because a newer RU is applied on top of the older one rather than rolling it back.

    Timeline

    Clock times are from the opatchauto, opatch and datapatch outputs, and show where the two nodes overlapped.

    StartNode 1Node 2Result
    20:21GI home (22m 51s)—Success
    20:44DB home (14m 55s)—Success (SQL deferred)
    21:01—GI home (27m 57s)Success
    21:05OJVM opatch applyGI home runningSuccess
    21:15Startup + datapatch (done 21:19), then utlrpGI home runningSuccess
    21:29—DB home, first attempt (10m 56s)Failed: no space
    21:40—Log cleanup4.5 GB → 20 GB free
    21:47—opatchauto resume (10m 28s)Success
    22:08—OJVM opatch applySuccess

    Lessons learned

    • Check free space on every node, not just the first. Node 1 was fine and node 2 was at 97% on the same mount. A quick df -h on the homes filesystem of each node belongs in the precheck, especially when you patch the nodes in parallel and won’t see node 2’s problem until it’s mid-apply.
    • Look at the listener ADR homes, not only the database ones. Most housekeeping scripts cover the RDBMS alert log and trace files. Here the ASM network listener alone had 15 GB of rotated files. Add the tnslsnr homes to a scheduled ADRCI purge.
    • Don’t panic when opatchauto fails mid-apply. Fix the cause and run opatchauto resume from the same home with the full path. The session ID is kept and it continues from the failed step. The TAR backups from Step 3 were there if it hadn’t.
    • Source the environment in the shell you actually use. Pasting the env lines together with sudo su - oracle leaves ORACLE_HOME empty. Run echo $ORACLE_HOME before trusting which opatch.
    • Take a baseline first. Recording the 29 application-schema invalids before patching, then running utlrp after datapatch, made the post-check comparison simple.

    ما شاء الله, one more cycle closed cleanly in spite of the space issue. If you run 19c on Grid Infrastructure, check your listener diag directories before the next CPU window.