Syed Anwar Ahmed – Oracle DBA Blog

Tag: Oracle 19c

  • ORA-00257: Archiver Error in Oracle 19c CDB — Diagnosis and Resolution

    Background

    This post documents a real production incident where an Oracle E-Business Suite (EBS) environment became completely unresponsive due to a full Fast Recovery Area (FRA). Users were reporting that everything was either slow or hanging — a classic sign that something fundamental had broken at the database layer.

    Symptoms

    • EBS application users reporting sessions hanging or extremely slow response
    • sqlplus apps/<password> failing with:
    ORA-00257: Archiver error. Connect AS SYSDBA only until resolved.
    • Non-SYSDBA connections completely blocked
    • 30 FNDLIBR (Concurrent Manager) processes running on the application server — higher than expected for a QA environment

    Environment

    • Oracle Database 19c (CDB/PDB architecture)
    • PDB: Application database PDB
    • EBS 12.2 on AIX
    • FRA configured on ASM diskgroup (+RECO), size 4095 GB
    • No explicit log_archive_dest — archivelogs defaulting to FRA

    Step 1 — Identify the Root Cause

    Connected to the CDB as SYSDBA and confirmed the error:

    SHOW PARAMETER db_recovery_file_dest;
    NAME                        TYPE        VALUE
--------------------------- ----------- ------
db_recovery_file_dest       string      +RECO
db_recovery_file_dest_size  big integer 4095G

    Checked FRA usage:

    COLUMN name           FORMAT A10
COLUMN limit_gb       FORMAT 999,999.99 HEADING 'LIMIT GB'
COLUMN used_gb        FORMAT 999,999.99 HEADING 'USED GB'
COLUMN reclaimable_gb FORMAT 999,999.99 HEADING 'RECLAIMABLE GB'
COLUMN number_of_files FORMAT 99999     HEADING 'FILES'

SELECT name,
       ROUND(space_limit/1024/1024/1024,2)       limit_gb,
       ROUND(space_used/1024/1024/1024,2)        used_gb,
       ROUND(space_reclaimable/1024/1024/1024,2) reclaimable_gb,
       number_of_files
FROM   v$recovery_file_dest;

    Output:

    NAME         LIMIT GB     USED GB  RECLAIMABLE GB  FILES
---------- ----------- ----------- -------------- ------
+RECO         4,095.00    4,075.65            .00   2426

    FRA was at 99.5% — 4,075 GB used out of 4,095 GB, zero reclaimable, 2,426 archivelog files.

    This was the root cause. With the FRA full and nothing reclaimable, the archiver (ARCn) process could not write new archive logs, blocking all database activity.

    Step 2 — Assess Backup Status Before Taking Action

    Before deleting any archivelogs, it is critical to understand the backup posture. Blindly deleting archivelogs without knowing the backup state can leave the database unrecoverable.

    Started an RMAN session and ran a crosscheck first:

    rman target /
    RMAN> crosscheck archivelog all;

    All 2,416 objects returned “validation succeeded” — no expired logs. This meant the FRA was genuinely full of valid, undeleted archivelogs.

    Next, checked backup history:

    RMAN> list backup summary;

    Backups were listed daily going back several weeks — however, on closer inspection:

    SELECT session_key,
       TO_CHAR(start_time,'DD-MON-YY HH24:MI') start_time,
       TO_CHAR(end_time,'DD-MON-YY HH24:MI')   end_time,
       status,
       input_bytes,
       output_bytes
FROM   v$rman_backup_job_details
ORDER  BY start_time DESC
FETCH FIRST 10 ROWS ONLY;

    Output revealed a critical finding:

    SESSION_KEY  START_TIME         END_TIME           STATUS    INPUT_BYTES  OUTPUT_BYTES
-----------  -----------------  -----------------  --------  -----------  ------------
      19986  30-APR-26 01:00    30-APR-26 01:00    COMPLETED           0         98304
      19904  29-APR-26 01:00    29-APR-26 01:00    COMPLETED           0         98304
    • INPUT_BYTES = 0 — no datafiles were being backed up
    • OUTPUT_BYTES = 98304 (96 KB) — only a controlfile/SPFILE autobackup
    • Jobs completing in under a minute — impossible for a real full DB backup

    Additionally:

    RMAN> list backup of archivelog all;
-- specification does not match any backup in the repository

    No archivelog backups existed at all. The scheduled backup job was not performing proper datafile or archivelog backups — a separate issue to be addressed post-incident.

    Step 3 — Escalation and Approval

    Given that there were no archivelog backups and the datafile backup was questionable, the decision to delete archivelogs and  Approval was obtained with the instruction to start conservatively — delete older than 15 days first, then 7 days if needed.

    Step 4 — Resolution

    First attempt — delete older than 15 days:

    RMAN> delete noprompt archivelog until time 'SYSDATE-15';

    RMAN returned warnings:

    RMAN-08138: warning: archived log not deleted - must create more backups

    This is because the RMAN retention policy was blocking deletion of logs that had never been backed up. With Tier 3 approval, used the force option to override:

    RMAN> delete noprompt force archivelog until time 'SYSDATE-15';

    This ran successfully, deleting archivelogs from January through late April.

    FRA usage after deletion:

    USED_GB  RECLAIMABLE_GB  NUMBER_OF_FILES
-------  --------------  ---------------
 246.71               0              149

    Over 3,800 GB freed — FRA dropped from 99.5% to ~6%.

    Step 5 — Verification

    Forced a log switch and verified archiver status:

    ALTER SYSTEM ARCHIVE LOG CURRENT;

SELECT dest_id, status, error
FROM   v$archive_dest
WHERE  status != 'INACTIVE';

    Output:

    DEST_ID  STATUS    ERROR
-------  --------  -----
      1  VALID

    Archiver resumed successfully. Application connectivity was restored and users confirmed sessions were working normally.

    Redo Log Status Check

    As part of the verification, also confirmed redo log health:

    SELECT thread#,
       sequence#,
       TO_CHAR(first_time,'DD-MON-YY HH24:MI:SS') first_time,
       first_change#,
       archived,
       status
FROM   v$log
ORDER  BY thread#, sequence#;
    THREAD#  SEQUENCE#  FIRST_TIME                FIRST_CHANGE#  ARC  STATUS
-------  ---------  ------------------------  -------------  ---  -------
      1       2417  05-MAY-26 01:56:44         1.7867E+10    NO   INACTIVE
      1       2418  05-MAY-26 03:27:20         1.7867E+10    NO   INACTIVE
      1       2419  05-MAY-26 07:57:10         1.7868E+10    NO   CURRENT

    No stuck or unarchived redo logs — database in healthy state.

    Incident Summary

    Item Detail
    Error ORA-00257: Archiver error
    Root Cause FRA (+RECO ASM diskgroup) at 99.5% capacity
    FRA Before 4,075 GB used / 2,426 files
    FRA After 246 GB used / 149 files
    Action Taken delete noprompt force archivelog until time 'SYSDATE-15'
    Space Freed ~3,829 GB
    Resolution Time ~45 minutes from identification to restoration

    Key Lessons Learned

    1. Monitor FRA Proactively

    Set up OEM 13c threshold alerts on FRA usage at 70% and 85%. Do not wait for ORA-00257 to discover the problem.

    SELECT ROUND(space_used/space_limit*100,2) pct_used
FROM   v$recovery_file_dest;

    Alert when this exceeds 80%.

    2. Always Check Backup Status Before Deleting Archivelogs

    In this incident, the backup job appeared to be running daily but was only backing up the controlfile (INPUT_BYTES=0). This is a serious gap — verify actual backup content, not just job status.

    3. Investigate the Backup Job

    A proper RMAN backup script for a 19c CDB should include:

    BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;

    4. Consider an Archivelog Deletion Policy

    If archivelog backups are not being taken, configure an RMAN retention policy to prevent FRA accumulation:

    CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

    5. FRA Sizing Review

    Even at 4095 GB, the FRA filled up due to months of uncleaned archivelogs. Review archivelog generation rate and size FRA accordingly, or implement regular cleanup.

    Conclusion

    ORA-00257 is one of those errors that brings an entire EBS environment to its knees instantly. The fix itself is straightforward — free up FRA space — but the investigation matters. Rushing to delete archivelogs without understanding the backup posture can result in an unrecoverable database.

    In this case, careful investigation revealed a deeper issue with the backup job that would have gone unnoticed had we not looked. Always verify, always escalate, and always get approval before deleting recovery-critical files.

    Syed Anwar Ahmed is an Oracle Apps DBA with over 11 years of production experience across Oracle EBS, Database, RAC, GoldenGate, and OEM environments. He writes about real-world Oracle incidents at syedanwarahmedoracle.blog.