Syed Anwar Ahmed – Oracle DBA Blog

Category: Oracle Database Administration

  • Resolving 98% Filesystem Usage with RMAN DELETE OBSOLETE: A Production DBA Case Study

    On a Saturday afternoon, a production database host reported its backup filesystem at 98% used. This post walks through how I handled it with RMAN, what the output showed, and which parts were still open when I stopped capturing evidence.

    The symptom

    The backup filesystem (/backup_fs, about 9.9T) was nearly full. The first check showed 282G free, and a later check showed 221G:

    $ df -h /backup_fs
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdX1       9.9T  9.2T  221G  98% /backup_fs

    An incremental backup was also running to the same location, so the filesystem was still being written to. A full filesystem would have failed that job, which is why this was urgent.

    Initial checks

    I started with three things:

    1. df -h to confirm the numbers.
    2. date; hostname to confirm I was on the right server and to timestamp my notes.
    3. The status of the running backup job.
    SET LINESIZE 120
SET PAGESIZE 50

COLUMN STATUS               FORMAT A12
COLUMN OUTPUT_BYTES_DISPLAY FORMAT A15
COLUMN TIME_TAKEN_DISPLAY   FORMAT A15

-- Replace <session_key> with the relevant RMAN session key
SELECT SESSION_KEY,
       STATUS,
       OUTPUT_BYTES_DISPLAY,
       TIME_TAKEN_DISPLAY
FROM   V$RMAN_BACKUP_JOB_DETAILS
WHERE  SESSION_KEY = <session_key>;

    The job was RUNNING, so whatever I did next had to leave it alone.

    Obsolete vs. expired

    These two get mixed up often.

    • Obsolete is decided by the retention policy. Here it was RECOVERY WINDOW OF 5 DAYS. A backup is obsolete when RMAN determines it is no longer needed to satisfy that policy. That does not mean every backup older than five days is obsolete, because RMAN checks whether the recovery window can still be met without it.
    • Expired is a repository status set by CROSSCHECK when RMAN cannot find a piece on disk.

    I ran LIST EXPIRED BACKUP; first. It returned “specification does not match any backup in the repository.” At that point, RMAN had no backup pieces marked EXPIRED in its repository. That did not mean no obsolete backups existed, and REPORT OBSOLETE showed that they did.

    Reviewing what RMAN considered obsolete

    RMAN> REPORT OBSOLETE;

RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 5 days

    The report listed backup pieces completed on 03, 12, 13 and 17 October, all in the backup directory, including several control file autobackup pieces.

    REPORT OBSOLETE only reports. It applies the configured policy and changes nothing. I reviewed the list before going further.

    Approval and DELETE OBSOLETE

    This is a production backup area, so I got Tier 3 approval before deleting anything. Once that was confirmed:

    RMAN> DELETE OBSOLETE;

    RMAN allocated six disk channels, listed the objects, and asked for confirmation:

    Do you really want to delete the above objects (enter YES or NO)? YES

    After I confirmed, RMAN printed deleted backup piece messages along with Deleted 5 objects and Deleted 6 objects summaries.

    I did not remove any files from the shell with rm. Deleting RMAN files at the OS level leaves the repository believing they still exist. A CROSSCHECK can reconcile that later, but it adds cleanup work and risk you do not need on a production system.

    The RMAN-06207 / RMAN-06208 warnings

    Mixed in with the successful deletions, RMAN repeatedly printed:

    RMAN-06207: warning: 4 objects could not be deleted for DISK channel(s) due
RMAN-06208:          to mismatched status.  Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
...
RMAN-06214: Backup Piece    <backup_piece_handle>

    Each warning listed four backup pieces, and there were six such groups. Two observations from the log:

    • Every handle in the mismatch lists belonged to the 03-Oct group of obsolete backups.
    • RMAN’s message says what it expects: the repository status did not match what the channel found, and CROSSCHECK is the way to correct it.

    That is an observation from the captured log, not a root-cause conclusion. I am not guessing why those pieces mismatched, because the log does not say.

    CROSSCHECK BACKUP

    RMAN> CROSSCHECK BACKUP;

    CROSSCHECK BACKUP checks whether backup pieces recorded in the RMAN repository are accessible on their associated storage. Depending on the result, RMAN updates their status to AVAILABLE or EXPIRED. It does not delete backup pieces.

    The output I captured showed pieces reported like this:

    backup piece handle=<backup_piece_handle> RECID=<n> STAMP=<n>
crosschecked backup piece: found to be 'AVAILABLE'

    Be careful about what this does and does not prove. It confirmed that the pieces in my captured excerpt were AVAILABLE. It did not show that every handle from the RMAN-06207 lists had been crosschecked, and the excerpt displayed different handles from the warnings. So I cannot say the mismatched pieces were resolved.

    Before and after

    All values are rounded df -h output, so I am not claiming exact bytes reclaimed.

    CheckSizeUsedAvailUse%
    Initial9.9T9.2T282G98%
    Pre-cleanup9.9T9.2T221G98%
    After cleanup9.9T6.0T3.5T64%
    Later9.9T6.1T3.5T64%
    Latest9.9T6.1T3.4T65%

    Following the cleanup, reported free space increased from approximately 221G to 3.4–3.5T, while utilization dropped from 98% to 64–65%. These are observed filesystem measurements; I have not attributed every byte of the change to a specific operation.

    Free space then slipped from 3.5T to 3.4T. The incremental backup was still writing to the same filesystem, but I did not prove it was the cause. I treated it as something to keep watching.

    Monitoring the concurrent backup

    I kept checking the same session. Output size grew across checks:

    SESSION_KEY STATUS       OUTPUT_BYTES_DI TIME_TAKEN_DISP
----------- ------------ --------------- ---------------
   REDACTED RUNNING          2.52T       21:14:51

    Earlier readings for that session went 2.43T, 2.45T, 2.46T, 2.47T, 2.49T, 2.50T, then 2.52T. Growing output tells you the job is moving. It does not tell you how close it is to finishing, because output bytes are not a percent-complete measure.

    What was and was not confirmed

    Confirmed:

    • Obsolete backups were deleted under the 5-day recovery window, with approval.
    • Free space on the filesystem rose from about 221G to about 3.4–3.5T.
    • The incremental backup was still RUNNING and producing output at the last check.

    Not captured:

    • Final completion of the incremental backup.
    • Resolution of every handle listed in the RMAN-06207 warnings.

    I would rather say that plainly than write a tidy ending the evidence does not support.

    Lessons and checklist

    1. Check what is writing to the filesystem before you start deleting.
    2. Run REPORT OBSOLETE and read it before DELETE OBSOLETE.
    3. Get the right approval for production deletes.
    4. Use RMAN to remove RMAN backups. Do not use rm.
    5. Treat RMAN-06207 as a repository mismatch to investigate, not noise.
    6. Remember that CROSSCHECK updates status. It does not delete.
    7. Read crosscheck output against the specific handles you care about.
    8. Trend free space after cleanup, especially while a backup is active.
    9. Revisit retention and capacity if the filesystem keeps reaching 98%.

    Disclaimer: this post describes one incident in one environment. Production backup cleanup needs environment-specific validation and proper authorization before you run anything similar. Hostnames, database names, paths and backup handles have been redacted.