Syed Anwar Ahmed – Oracle DBA Blog

Author: SYED ANWAR AHMED

Oracle EBS OACORE Server in FAILED_NOT_RESTARTABLE State: Real-Time Issue, RCA and Fix
In Oracle E-Business Suite (EBS) environments, application tier stability is critical to ensure seamless user experience. However, there are scenarios where managed servers behave unexpectedly and require manual intervention. This post walks through a real-world production issue where an OACORE managed server entered a FAILED_NOT_RESTARTABLE state, its impact, root cause analysis, and how it was resolved.

Environment Details
- Oracle E-Business Suite: R12.2.x
- Application Tier: WebLogic Managed Servers
- Component Impacted: OACORE Server (oacore_server1)
- Environment Type: Production
Problem Statement

An alert was received indicating oacore_server1 was in FAILED_NOT_RESTARTABLE state. Upon verification, the server was running but Node Manager could not auto-restart it.

Key Observations

Despite the OACORE server being in a failed state, the application remained accessible and functional — traffic was being handled by other OACORE servers. This is due to the multi-OACORE architecture with load balancing via OHS/Web tier. However, this creates a hidden risk: load redistribution increases pressure on remaining servers and can lead to cascading failures if not addressed promptly.

Detailed Analysis
- Managed server restart attempts failed during initialization
- Bulk concurrent requests were actively running
- CPU utilization spiked on the application tier
- JVM resources were under pressure
Understanding FAILED_NOT_RESTARTABLE

In Oracle WebLogic Server, a managed server is marked as FAILED_NOT_RESTARTABLE after repeated unsuccessful restart attempts. This is a protective mechanism designed to prevent unstable restart loops when the server cannot recover successfully.

Root Cause Analysis

The OACORE managed server entered FAILED_NOT_RESTARTABLE state due to repeated startup failures following an unclean or resource-constrained shutdown. High CPU utilization and heavy concurrent workload placed JVM resources under pressure, preventing a clean restart cycle. Residual runtime artifacts (such as incomplete shutdown state or resource locks) prevented successful reinitialization, causing WebLogic to mark the server as FAILED_NOT_RESTARTABLE.

Resolution
```
cd $ADMIN_SCRIPTS_HOME
./admanagedsrvctl.sh stop oacore_server1
./admanagedsrvctl.sh start oacore_server1
```
After the controlled restart, the server returned to RUNNING state with all deployments active and the application stable.

Identify Inactive Forms Sessions

Inactive sessions holding resources can contribute to JVM pressure. Use this query to identify them safely — do not terminate without proper validation and approvals:
```
SELECT s.sid,
       s.serial#,
       s.username,
       s.status,
       s.program,
       s.machine,
       ROUND(s.last_call_et/3600,2) AS hours_inactive
FROM v$session s
WHERE s.status = 'INACTIVE'
AND s.username = 'APPS'
AND s.program LIKE 'frmweb%'
AND s.last_call_et > 28800   -- 8 hours
ORDER BY hours_inactive DESC;
```
Reference only — do NOT execute without validation:
```
ALTER SYSTEM KILL SESSION 'SID,SERIAL#' IMMEDIATE;
```
Automate Session Monitoring

Use this script to monitor inactive sessions every 8 hours via cron:
```
#!/bin/bash
export ORACLE_SID=your_sid
export ORACLE_HOME=/path/to/oracle_home
export PATH=$ORACLE_HOME/bin:$PATH

sqlplus -s / as sysdba <<EOF
SET LINES 200
SET PAGES 200
SELECT COUNT(*) AS inactive_sessions
FROM v\$session
WHERE status='INACTIVE'
AND username='APPS'
AND program LIKE 'frmweb%'
AND last_call_et > 28800;
EXIT;
EOF
```
```
# Crontab entry - every 8 hours
0 */8 * * * /path/to/inactive_sessions.sh >> /tmp/inactive_sessions.log
```
DBA Quick Commands
```
-- Check system load
top
uptime
ps -ef | grep oacore

-- Check running concurrent requests
SELECT request_id, phase_code, status_code
FROM fnd_concurrent_requests
WHERE phase_code = 'R';
```
Key Takeaways
- Application may appear healthy even when an OACORE server fails due to load balancing
- FAILED_NOT_RESTARTABLE is a protective mechanism, not the root cause itself
- Resource pressure and restart failures must be analyzed together
- Controlled and governed actions are critical in production environments
- Proactive session monitoring via automation helps prevent recurrence
Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.
24/03/2026
Oracle DB Health Check Scripts: Real-World DBA Monitoring Guide
In day-to-day Oracle Database and E-Business Suite (EBS) administration, proactive monitoring is critical to ensure system stability, performance, and availability. As part of my real-world DBA experience, I created a set of health check scripts to quickly assess the status of key database components.

Objective

The goal of these scripts is to provide a quick and efficient health check covering database status, instance status, tablespace usage, invalid objects, and RMAN backup status — all from a single SQL*Plus session.

Solution Overview

I developed a SQL-based script (hc.sql) that gathers essential health metrics from Oracle dynamic views and DBA tables. The scripts are designed to be lightweight, fast, and provide immediate actionable insights.

GitHub Repository

The complete script is available on GitHub:
https://github.com/Syedanwarahmed/scripts_for_healthcheck

Key Features
- Database and instance status check
- Tablespace usage monitoring
- Active session tracking
- Invalid object detection
- RMAN backup job verification
- Easy execution using SQL*Plus
How to Use
```
sqlplus / as sysdba
@hc.sql
```
Sample Checks Included

Database Status
```
SELECT NAME, OPEN_MODE, DATABASE_ROLE FROM V$DATABASE;
```
Tablespace Usage
```
SELECT tablespace_name,
       ROUND(used_space * 8192 / 1024 / 1024, 2) used_mb,
       ROUND(tablespace_size * 8192 / 1024 / 1024, 2) total_mb,
       ROUND(used_percent, 2) pct_used
FROM dba_tablespace_usage_metrics
ORDER BY used_percent DESC;
```
Active Sessions
```
SELECT COUNT(*) active_sessions FROM V$SESSION WHERE STATUS='ACTIVE';

-- Session breakdown by status
SELECT status, COUNT(*) cnt
FROM v$session
GROUP BY status
ORDER BY cnt DESC;
```
Invalid Objects
```
SELECT owner, object_type, COUNT(*) cnt
FROM dba_objects
WHERE status = 'INVALID'
GROUP BY owner, object_type
ORDER BY cnt DESC;
```
RMAN Backup Status
```
SELECT status, start_time, end_time,
       ROUND((output_bytes/1024/1024/1024),2) output_gb
FROM V$RMAN_BACKUP_JOB_DETAILS
ORDER BY start_time DESC
FETCH FIRST 5 ROWS ONLY;
```
Real-World Value

These scripts are derived from real-time production support scenarios where quick diagnosis is required during high CPU issues, backup failures, tablespace alerts, and performance degradation. Having a single script to validate system health saves valuable time during critical situations.

Future Enhancements

I plan to enhance this repository further by adding shell automation scripts, alerting mechanisms, integration with monitoring tools, and additional EBS-specific checks.

Conclusion

A well-designed health check script is an essential tool for every DBA. It not only helps in proactive monitoring but also ensures faster troubleshooting and improved system reliability. Feel free to explore the repository, use the scripts, and share your feedback.

Written by Syed Anwar Ahmed — Oracle Apps DBA with 11 years of production experience.
Connect: sdanwarahmed@gmail.com | LinkedIn | GitHub
19/03/2026
Real-World Oracle Database and EBS Troubleshooting Guides
A collection of real-world Oracle Database and EBS troubleshooting scenarios from production environments. These guides cover common issues encountered during day-to-day DBA operations.

Resolving RMAN ORA-19502 Backup Failure Due to Disk Full

RMAN backup job failed with the following errors:
```
ORA-19502: write error on file
ORA-27072: File I/O error
Linux-x86_64 Error: 28: No space left on device
```
Environment

Oracle Database 19c, RMAN Disk Backup, Linux Environment

Diagnosis

Checked RMAN logs and identified the backup destination filesystem was full:
```
-- Check filesystem usage
df -h

-- Check FRA usage from database
SELECT name, space_limit/1024/1024/1024 limit_gb,
       space_used/1024/1024/1024 used_gb,
       ROUND(space_used/space_limit*100,2) pct_used
FROM v$recovery_file_dest;
```
The backup mount point reached 100% utilization.

Resolution

Cleaned older backup files after validating they were no longer needed, then reran the backup job:
```
-- Use RMAN to safely clean up obsolete backups (preferred over OS delete)
RMAN> CROSSCHECK BACKUP;
RMAN> DELETE EXPIRED BACKUP;
RMAN> DELETE OBSOLETE;
```
After freeing space, the RMAN backup completed successfully.

Key Lesson

Always monitor backup destinations proactively. Schedule DELETE OBSOLETE as part of your regular RMAN maintenance to prevent FRA saturation.

Written by Syed Anwar Ahmed — Oracle Apps DBA with 11 years of production experience.
Connect: sdanwarahmed@gmail.com | LinkedIn
16/03/2026