On a Saturday afternoon, a production database host reported its backup filesystem at 98% used. This post walks through how I handled it with RMAN, what the output showed, and which parts were still open when I stopped capturing evidence.
The symptom
The backup filesystem (/backup_fs, about 9.9T) was nearly full. The first check showed 282G free, and a later check showed 221G:
$ df -h /backup_fs
Filesystem Size Used Avail Use% Mounted on
/dev/sdX1 9.9T 9.2T 221G 98% /backup_fsAn incremental backup was also running to the same location, so the filesystem was still being written to. A full filesystem would have failed that job, which is why this was urgent.
Initial checks
I started with three things:
df -hto confirm the numbers.date; hostnameto confirm I was on the right server and to timestamp my notes.- The status of the running backup job.
SET LINESIZE 120
SET PAGESIZE 50
COLUMN STATUS FORMAT A12
COLUMN OUTPUT_BYTES_DISPLAY FORMAT A15
COLUMN TIME_TAKEN_DISPLAY FORMAT A15
-- Replace <session_key> with the relevant RMAN session key
SELECT SESSION_KEY,
STATUS,
OUTPUT_BYTES_DISPLAY,
TIME_TAKEN_DISPLAY
FROM V$RMAN_BACKUP_JOB_DETAILS
WHERE SESSION_KEY = <session_key>;The job was RUNNING, so whatever I did next had to leave it alone.
Obsolete vs. expired
These two get mixed up often.
- Obsolete is decided by the retention policy. Here it was
RECOVERY WINDOW OF 5 DAYS. A backup is obsolete when RMAN determines it is no longer needed to satisfy that policy. That does not mean every backup older than five days is obsolete, because RMAN checks whether the recovery window can still be met without it. - Expired is a repository status set by
CROSSCHECKwhen RMAN cannot find a piece on disk.
I ran LIST EXPIRED BACKUP; first. It returned “specification does not match any backup in the repository.” At that point, RMAN had no backup pieces marked EXPIRED in its repository. That did not mean no obsolete backups existed, and REPORT OBSOLETE showed that they did.
Reviewing what RMAN considered obsolete
RMAN> REPORT OBSOLETE;
RMAN retention policy will be applied to the command
RMAN retention policy is set to recovery window of 5 daysThe report listed backup pieces completed on 03, 12, 13 and 17 October, all in the backup directory, including several control file autobackup pieces.
REPORT OBSOLETE only reports. It applies the configured policy and changes nothing. I reviewed the list before going further.
Approval and DELETE OBSOLETE
This is a production backup area, so I got Tier 3 approval before deleting anything. Once that was confirmed:
RMAN> DELETE OBSOLETE;RMAN allocated six disk channels, listed the objects, and asked for confirmation:
Do you really want to delete the above objects (enter YES or NO)? YESAfter I confirmed, RMAN printed deleted backup piece messages along with Deleted 5 objects and Deleted 6 objects summaries.
I did not remove any files from the shell with rm. Deleting RMAN files at the OS level leaves the repository believing they still exist. A CROSSCHECK can reconcile that later, but it adds cleanup work and risk you do not need on a production system.
The RMAN-06207 / RMAN-06208 warnings
Mixed in with the successful deletions, RMAN repeatedly printed:
RMAN-06207: warning: 4 objects could not be deleted for DISK channel(s) due
RMAN-06208: to mismatched status. Use CROSSCHECK command to fix status
RMAN-06210: List of Mismatched objects
...
RMAN-06214: Backup Piece <backup_piece_handle>Each warning listed four backup pieces, and there were six such groups. Two observations from the log:
- Every handle in the mismatch lists belonged to the 03-Oct group of obsolete backups.
- RMAN’s message says what it expects: the repository status did not match what the channel found, and
CROSSCHECKis the way to correct it.
That is an observation from the captured log, not a root-cause conclusion. I am not guessing why those pieces mismatched, because the log does not say.
CROSSCHECK BACKUP
RMAN> CROSSCHECK BACKUP;CROSSCHECK BACKUP checks whether backup pieces recorded in the RMAN repository are accessible on their associated storage. Depending on the result, RMAN updates their status to AVAILABLE or EXPIRED. It does not delete backup pieces.
The output I captured showed pieces reported like this:
backup piece handle=<backup_piece_handle> RECID=<n> STAMP=<n>
crosschecked backup piece: found to be 'AVAILABLE'Be careful about what this does and does not prove. It confirmed that the pieces in my captured excerpt were AVAILABLE. It did not show that every handle from the RMAN-06207 lists had been crosschecked, and the excerpt displayed different handles from the warnings. So I cannot say the mismatched pieces were resolved.
Before and after
All values are rounded df -h output, so I am not claiming exact bytes reclaimed.
| Check | Size | Used | Avail | Use% |
|---|---|---|---|---|
| Initial | 9.9T | 9.2T | 282G | 98% |
| Pre-cleanup | 9.9T | 9.2T | 221G | 98% |
| After cleanup | 9.9T | 6.0T | 3.5T | 64% |
| Later | 9.9T | 6.1T | 3.5T | 64% |
| Latest | 9.9T | 6.1T | 3.4T | 65% |
Following the cleanup, reported free space increased from approximately 221G to 3.4–3.5T, while utilization dropped from 98% to 64–65%. These are observed filesystem measurements; I have not attributed every byte of the change to a specific operation.
Free space then slipped from 3.5T to 3.4T. The incremental backup was still writing to the same filesystem, but I did not prove it was the cause. I treated it as something to keep watching.
Monitoring the concurrent backup
I kept checking the same session. Output size grew across checks:
SESSION_KEY STATUS OUTPUT_BYTES_DI TIME_TAKEN_DISP
----------- ------------ --------------- ---------------
REDACTED RUNNING 2.52T 21:14:51Earlier readings for that session went 2.43T, 2.45T, 2.46T, 2.47T, 2.49T, 2.50T, then 2.52T. Growing output tells you the job is moving. It does not tell you how close it is to finishing, because output bytes are not a percent-complete measure.
What was and was not confirmed
Confirmed:
- Obsolete backups were deleted under the 5-day recovery window, with approval.
- Free space on the filesystem rose from about 221G to about 3.4–3.5T.
- The incremental backup was still
RUNNINGand producing output at the last check.
Not captured:
- Final completion of the incremental backup.
- Resolution of every handle listed in the
RMAN-06207warnings.
I would rather say that plainly than write a tidy ending the evidence does not support.
Lessons and checklist
- Check what is writing to the filesystem before you start deleting.
- Run
REPORT OBSOLETEand read it beforeDELETE OBSOLETE. - Get the right approval for production deletes.
- Use RMAN to remove RMAN backups. Do not use
rm. - Treat
RMAN-06207as a repository mismatch to investigate, not noise. - Remember that
CROSSCHECKupdates status. It does not delete. - Read crosscheck output against the specific handles you care about.
- Trend free space after cleanup, especially while a backup is active.
- Revisit retention and capacity if the filesystem keeps reaching 98%.
Disclaimer: this post describes one incident in one environment. Production backup cleanup needs environment-specific validation and proper authorization before you run anything similar. Hostnames, database names, paths and backup handles have been redacted.
