Category: Uncategorized

الحمد لله — This post documents a real Oracle Database 19c patching activity on a Linux x86-64 environment, progressing from 19.31 to 19.32 using the July 2026 Combo Patch. The activity included precheck validation, binary patching, datapatch execution, and post-patch verification with actual command outputs.

Introduction

This article documents the steps followed to patch an Oracle Database 19c environment from 19.31 to 19.32 using the July 2026 Database and OJVM Release Update Combo Patch on Linux x86-64.

The patches applied were:

Patch 39618649
Combo of OJVM Component Release Update 19.32.0.0.260721
+ Database Jul 2026 Release Update 19.32.0.0.260721

Component patches:
39472050 - Database Release Update 19.32.0.0.260721
39222882 - Oracle JavaVM Component Release Update 19.32.0.0.260721

Pre-Patching Validation

1. Verify Current Database Patch Level

First, verify the current patches installed in the Oracle Home:

$ $ORACLE_HOME/OPatch/opatch lspatches

The environment was running the following baseline before July patching:

38906621;OJVM RELEASE UPDATE: 19.31.0.0.260421
39034528;Database Release Update : 19.31.0.0.260421
29585399;OCW RELEASE UPDATE 19.3.0.0.0

Patching objective:

• Database RU: 19.31 → 19.32 (39034528 → 39472050)
• OJVM RU: 19.31 → 19.32 (38906621 → 39222882)

6. Run DB RU Conflict Check

Before applying the Database RU, validate for conflicting one-off patches:

$ cd 39618649/39472050
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

✓ No conflicting one-off patch was identified for the DB RU 39472050.

7. Run OJVM RU Conflict Check

Perform the same validation for the OJVM RU:

$ cd 39618649/39222882
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

✓ No conflicting one-off patch was identified for the OJVM RU 39222882.

Post-Patching Results

OPatch Inventory Verification

$ $ORACLE_HOME/OPatch/opatch lspatches

39222882;OJVM RELEASE UPDATE: 19.32.0.0.260721
39472050;Database Release Update : 19.32.0.0.260721
29585399;OCW RELEASE UPDATE 19.3.0.0.0

OPatch succeeded.

Patch progression:

• OJVM RU: 38906621 (19.31) → 39222882 (19.32) ✓
• Database RU: 39034528 (19.31) → 39472050 (19.32) ✓
• OCW: 29585399 (unchanged) ✓

Component Registry Status

All major database components are VALID after patching:

CATALOG         VALID
CATEGORY        VALID
CATJAVA         VALID
CATOJAVA        VALID
JAVAVM          VALID
XML             VALID
XDB             VALID

Database and PDB Status

SQL> select name, open_mode, database_role from v$database;

NAME            OPEN_MODE       DATABASE_ROLE
----------      -----------     ----------------
PRODDB          READ WRITE      PRIMARY

SQL> show pdbs;

    CON_ID CON_NAME                 OPEN_MODE  RESTRICTED
---------- ---------------------- ---------- ----------
         2 PDB$SEED                 READ ONLY  NO
         3 APPPDB                   READ WRITE NO
         4 ARCHPDB                  READ WRITE NO

✓ Database and all PDBs are open and operational.

SQL Patch Registry

Both 19.32 patches applied successfully to CDB and APPPDB PDB:

PATCH_ID      ACTION     STATUS          DESCRIPTION
-----------   ---------- --------------- ----------------------------------
39222882      APPLY      SUCCESS         OJVM RU 19.32.0.0.260721
39472050      APPLY      SUCCESS         Database RU 19.32.0.0.260721

Note: During the 19.32 OJVM update, the previous 19.31 OJVM patch (38906621) may be recorded as ROLLBACK SUCCESS, followed by successful application of OJVM RU 39222882. This is normal and expected during the RU transition — it is not an indication of a patching failure.

Invalid Objects Validation

Post-patch invalid objects count: 83 (unchanged from pre-patch baseline)

✓ Invalid object count matches pre-patch baseline. No new invalids introduced by 19.32 RU.

oradism Permissions Restored

$ ls -l $ORACLE_HOME/bin/oradism
-rwsr-x---. 1 root oinstall 147848 May  5 12:25 /u01/app/oracle/product/19c/db_1/bin/oradism

✓ Permissions correctly restored to original state (root:oinstall 4750).

Rollback Plan

Important: Before executing any rollback, always:

• Verify the exact version from the patch component README
• Follow the documented rollback sequence from the README
• Test in a non-production environment first
• Obtain explicit approval from your database administration team or management

If the 19.32 RU must be rolled back, the procedure generally follows this sequence (consult the specific README for patches 39472050 and 39222882 for the authoritative step-by-step guidance):

1. Shut down the database and listener
2. Revert the OJVM RU first, then the Database RU (reverse order of application)
3. Start the database in restricted mode to verify compatibility
4. Run datapatch to revert SQL patches: ./datapatch -verbose
5. Verify patch registry and OPatch inventory reflect the reverted state
6. Open the database normally and run post-revert health checks

Always retain a clean, tested backup of the Oracle Home taken immediately before patching. This allows for rapid restoration if rollback becomes necessary.

Key Takeaways

A reliable Oracle Database RU patching activity is more than running opatch apply twice. A structured approach includes:

• Capture existing state: Patch inventory, SQL patch registry, invalid objects, database role, PDB status, and oradism permissions before making any changes.
• Validate prerequisites: Run conflict checks on both component patches and ensure OPatch is at the recommended version.
• Follow the documented sequence: Binary patches in the correct order, then startup, then datapatch.
• Verify at each step: Check OPatch inventory after binary patching, verify component registry after startup, and confirm datapatch success via the SQL registry.
• Compare against baseline: Invalid objects, registry status, and listener configuration should match pre-patch evidence.
• Plan for rollback: Maintain a clean backup and understand the documented rollback procedure from the official patch README before starting.

Oracle Database 19c was successfully upgraded from the 19.31 RU level to the July 2026 19.32 RU level, including OJVM, with successful binary patching, SQL patching, and post-patch health validation.

You receive a call: WebLogic requests are being rejected.

The error appears in the logs:

[Socket:000445]Connection rejected, filter blocked Socket,
weblogic.security.net.FilterException:
[Security:090220]rule 2

At first, you might start checking WebLogic authentication, AdminServer credentials, or database connectivity.

Don’t start there.

The exception is already giving you a very strong clue about where to investigate.

The Error Tells the Story

The important part is:

weblogic.security.net.FilterException
[Security:090220]rule 2

Oracle documents BEA-090220 as:

A connection filter rule denied access.

In other words, WebLogic is rejecting the connection because of a configured connection-filter rule.

This is different from an authentication failure.

The request is reaching the WebLogic server, and the connection filter is evaluating the incoming connection.

What Is a WebLogic Connection Filter?

WebLogic connection filters provide a network-level access-control layer.

They can be used to control which clients are permitted to connect to a WebLogic Server instance. Oracle describes connection filters as an additional security layer that can restrict access at the network level.

A simplified flow looks like this:

Client
   |
   v
WebLogic Listener
   |
   v
Connection Filter
   |
   +---- ALLOW ----> Continue processing
   |
   +---- DENY -----> Connection rejected

This is why an error such as:

Connection rejected, filter blocked Socket

is so useful during troubleshooting.

Understanding Security:090220 Rule 2

This is where we need to be precise.

WebLogic evaluates connection-filter rules in the order in which they are configured.

The first matching rule determines whether the connection is allowed or denied. If no rule matches, the connection is permitted by the default filter implementation.

For example:

Rule 1: 192.0.2.11 * * allow
Rule 2: 192.0.2.12 * * allow
Rule 3: 0.0.0.0/0 * * deny

If a connection matches Rule 3, WebLogic reports:

[Security:090220]rule 3

Therefore:

rule 2 identifies the connection-filter rule associated with the rejection reported by WebLogic. Always inspect the actual configured rule sequence rather than assuming it is the catch-all deny rule.

Where Is the Configuration?

For Oracle E-Business Suite Release 12.2, the WebLogic connection-filter configuration can be found in:

$DOMAIN_HOME/config/config.xml

Oracle’s EBS documentation shows the connection-filter configuration in this file.

For example:

<connection-filter>weblogic.security.net.ConnectionFilterImpl</connection-filter>

<connection-filter-rule>192.0.2.11 * * allow</connection-filter-rule>

<connection-filter-rule>192.0.2.12 * * allow</connection-filter-rule>

<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

In an EBS environment you may also see an EBS-specific connection-filter implementation, such as:

<connection-filter>oracle.apps.ad.tools.configuration.wls.filter.EBSConnectionFilterImpl</connection-filter>

If that is what your environment is using, don’t replace the filter class just because you are troubleshooting a rule. Focus on the configured rules.

Check the Rules Before Changing Anything

First inspect the actual configuration:

cd $DOMAIN_HOME/config

grep -n -A10 -B5 "connection-filter" config.xml

You want to establish:

• Which connection-filter implementation is configured?
• What are the connection-filter rules?
• Which source IP is making the connection?
• Which rule is actually denying the connection?
• Is the source IP supposed to be allowed?

This is much better than immediately changing the filter to allow.

Always Back Up config.xml

Before making any manual configuration change:

cd $DOMAIN_HOME/config

cp -ip config.xml config.xml_$(date +%Y%m%d_%H%M%S).bak

If this is an EBS environment, also consider the run/patch filesystem implications before making a change that needs to persist through online patching. Oracle’s EBS documentation notes that configuration changes on the run filesystem are propagated to the patch filesystem during fs_clone.

Don’t Confuse This With Basic Authentication

While reviewing config.xml, you may encounter:

<enforce-valid-basic-auth-credentials>
    false
</enforce-valid-basic-auth-credentials>

It is tempting to associate this with every WebLogic security-related error.

But BEA-090220 specifically indicates that a connection-filter rule denied access.

Therefore, for this particular error, the first investigation should be:

Security:090220
       ↓
Connection Filter
       ↓
Connection Filter Rules
       ↓
Source IP / Port / Protocol

rather than Basic Authentication.

A Typical Configuration

Oracle’s EBS documentation provides examples such as:

<connection-filter-rule>192.0.2.11 * * allow</connection-filter-rule>

<connection-filter-rule>192.0.2.12 * * allow</connection-filter-rule>

<connection-filter-rule>192.0.2.100 * 17001 allow https</connection-filter-rule>

<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

The final rule:

0.0.0.0/0 * * deny

acts as a catch-all deny.

Oracle specifically documents this pattern: allow the required sources first, then use 0.0.0.0/0 * * deny as the final rule so that all other sources are rejected.

The Incident Scenario

In our case, the error was:

[Socket:000445]Connection rejected, filter blocked Socket,
weblogic.security.net.FilterException:
[Security:090220]rule 2

During the investigation, we reviewed the WebLogic configuration and traced the rejection back to the connection-filter rules.

The important distinction was:

WebLogic is running
        |
        v
Request reaches WebLogic
        |
        v
Connection filter evaluates request
        |
        v
Configured rule denies connection
        |
        v
Security:090220

That changed the direction of the investigation immediately.

Resolution

Once the offending rule has been identified, there are two approaches.

Option 1 — Correct the Allow Rule

This is the preferred approach.

For example, if the trusted WebLogic client is:

192.0.2.50

the configuration could contain:

<connection-filter-rule>192.0.2.50 * * allow</connection-filter-rule>

<connection-filter-rule>0.0.0.0/0 * * deny</connection-filter-rule>

The important point is that the allow rule must appear before the catch-all deny rule.

Option 2 — Temporary Troubleshooting Workaround

In a controlled troubleshooting situation, changing:

0.0.0.0/0 * * deny

to:

0.0.0.0/0 * * allow

can be used to confirm that the connection filter is the cause of the problem.

However, this effectively removes the intended network restriction.

So: Use this as a diagnostic/workaround step, not as the preferred permanent production configuration.

Oracle’s documented security model is to allow the required sources and retain a final deny rule.

If Manually Editing config.xml

If you are manually modifying the domain configuration rather than using the WebLogic Administration Console, perform the change under your environment’s standard configuration-management procedure and ensure the Administration Server is not concurrently modifying the file.

For a controlled manual change, the practical sequence is:

Stop AdminServer
      |
      v
Backup config.xml
      |
      v
Review current connection-filter rules
      |
      v
Make the required change
      |
      v
Validate config.xml
      |
      v
Start AdminServer
      |
      v
Monitor logs
      |
      v
Retest the connection

Oracle also supports configuring connection filters through the WebLogic Administration Console and then activating the configuration and restarting as required.

Troubleshooting Decision Tree

When you encounter:

[Socket:000445]Connection rejected, filter blocked Socket
weblogic.security.net.FilterException
[Security:090220]rule N

use this approach:

                 WebLogic Request
                       |
                       v
              WebLogic reachable?
                       |
                      YES
                       |
                       v
             Check exact exception
                       |
                       v
          weblogic.security.net
             FilterException?
                       |
                      YES
                       |
                       v
              Security:090220?
                       |
                      YES
                       |
                       v
       Review connection-filter rules
                       |
                       v
          Identify rule N
                       |
                       v
        Identify source IP/port/protocol
                       |
                       v
          Does the rule allow it?
                 /          \
               YES          NO
                |            |
                v            v
        Investigate next   Correct the
             layer          filter rule

Key Takeaways

1. Read the exception literally — FilterException is a major clue.
2. Security:090220 means a connection-filter rule denied access — Oracle explicitly documents BEA-090220 this way.
3. Rule ordering matters — The first matching rule determines the result.
4. rule 2 is not automatically the catch-all deny — Check the actual rule list before making assumptions.
5. 0.0.0.0/0 * * deny is normally a catch-all restriction — It should generally come after the required allow rules.
6. Don’t use a global allow as the permanent fix — If the issue is caused by a missing source IP rule, add the required source rather than permanently opening the filter.

Final Thought

One of the most useful troubleshooting skills in middleware administration is learning to read the error as a diagnostic path.

Consider the original message:

[Socket:000445]
Connection rejected,
filter blocked Socket,
weblogic.security.net.FilterException:
[Security:090220]rule 2

Break it down:

Connection rejected
        ↓
Filter blocked Socket
        ↓
WebLogic Connection Filter
        ↓
BEA/Security 090220
        ↓
Connection-filter rule denied access
        ↓
Check the actual rule identified by the error

Instead of treating it as another generic WebLogic failure, the error points us directly toward the connection-filter configuration.

Read the error carefully. Sometimes the error message has already told you where the problem is.

Alhamdulillah.

Introduction

While working on an Oracle E-Business Suite R12.1 environment running on Oracle Database 19c, I encountered a failure while configuring the database-side UTL_FILE_DIR replacement using txkCfgUtlfileDir.pl.

The script repeatedly reported:

ERRORMSG: Invalid APPS database user credentials.
ERRORCODE = 1 ERRORCODE_END

At first glance, this looked like a straightforward APPS password problem.

It wasn’t.

The investigation eventually uncovered two separate contributing factors:

• Issue 1: The APPS account was being locked by repeated authentication attempts from an external application.
• Issue 2: The TXK utility expected a JRE component that was missing from the AppsUtil directory structure.

These were independent issues discovered during different phases of the troubleshooting investigation. After resolving the APPS lockout and verifying the JRE availability, the UTL_FILE_DIR synchronization proceeded to completion: directory objects → setUtlFileDir → syncUtlFileDir → FND_CONC_CLONE.SETUP_CLEAN → Database AutoConfig.

This post documents the investigation and the commands used.

1. Environment

The database environment was:

Oracle E-Business Suite R12.1
Oracle Database 19c
Database CDB : SRCDB
PDB          : TESTDB
Listener     : 1522
Service      : ebs_service

The ORACLE_HOME was:

ORACLE_HOME=$ORACLE_BASE/TESTDB/db/tech_st/19.3.0.0

The EBS database context file was:

CONTEXT_FILE=$ORACLE_HOME/appsutil/TESTDB_dbctx.xml

The environment was loaded with:

cd $ORACLE_HOME
. ./TESTDB_dbctx.env

Validation:

echo "ORACLE_SID=$ORACLE_SID"
echo "ORACLE_HOME=$ORACLE_HOME"
echo "TNS_ADMIN=$TNS_ADMIN"
echo "CONTEXT_FILE=$CONTEXT_FILE"

Expected values:

ORACLE_SID=TESTDB
ORACLE_HOME=$ORACLE_BASE/TESTDB/db/tech_st/19.3.0.0
TNS_ADMIN=$ORACLE_HOME/network/admin/TESTDB_dbctx
CONTEXT_FILE=$ORACLE_HOME/appsutil/TESTDB_dbctx.xml

2. Initial txkCfgUtlfileDir.pl Failure

The initial command was:

perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl \
-contextfile=$CONTEXT_FILE \
-oraclehome=$ORACLE_HOME \
-outdir=$ORACLE_HOME/appsutil/log \
-mode=getUtlFileDir

The script failed with:

FUNCTION: main::validateAppsSchemaCredentials
ERRORMSG: Invalid APPS database user credentials.
ERRORCODE = 1 ERRORCODE_END

The context file itself was found successfully:

Context file:
$ORACLE_HOME/appsutil/TESTDB_dbctx.xml exists.

So the failure was occurring during APPS credential validation.

3. Validate the PDB

Because this was a multitenant database, I first verified the PDB state.

SHOW PDBS;

Output:

CON_ID CON_NAME    OPEN MODE   RESTRICTED
------ ----------- ----------- ----------
2      PDB$SEED    READ ONLY   NO
3      TESTDB      READ WRITE  NO

Then:

ALTER SESSION SET CONTAINER=TESTDB;

The PDB was open read/write, so the next step was to validate the APPS account.

4. APPS Account Investigation

Initially, the APPS account showed as open after an unlock:

ALTER USER APPS ACCOUNT UNLOCK;

Verification:

SELECT username,
       account_status,
       lock_date
FROM dba_users
WHERE username = 'APPS';

At one point:

APPS    OPEN

However, the TXK utility continued to report invalid APPS credentials.

The most useful clue came from the TXK-generated diagnostic log:

cat $ORACLE_HOME/appsutil/log/TXK_UTIL_DIR_*/validate_apps_password.log

The log showed:

Connected to an idle instance.
ERROR:
ORA-28000: The account is locked.

This was much more specific than the generic TXK error. The failure was occurring at the database authentication layer, not at the password validation level.

5. Understanding the TXK Workflow

Rather than treating the generic TXK error at face value, I inspected txkCfgUtlfileDir.pl to understand how the utility validates the APPS credentials.

The script constructs the database connection using the APPS credentials from the EBS configuration and performs the validation through SQL*Plus. The detailed validate_apps_password.log showed ORA-28000, confirming that the failure was occurring at the database authentication layer.

6. Verify Context Configuration

The context file contained:

grep -n -iE 's_apps_user|s_dbSid|s_tools_twotask|s_dbService' \
$CONTEXT_FILE

Relevant values:

s_dbSid       = TESTDB
s_dbService   = ebs_service
s_apps_user   = apps

The database SID, service and listener port were also checked in the context configuration:

grep -n -iE 's_dbSid|s_dbService|s_dbport' \
$CONTEXT_FILE

Result:

s_dbSid       = TESTDB
s_dbService   = ebs_service
s_dbport      = 1522

These values were consistent with the environment.

7. Verify TNS Connectivity

The EBS TNS entry was properly configured for the PDB service.

export TNS_ADMIN=$ORACLE_HOME/network/admin/TESTDB_dbctx

Then:

tnsping TESTDB

Result:

Used TNSNAMES adapter to resolve the alias
Attempting to contact ...
OK (0 msec)

Therefore: TNS resolution and listener connectivity were not the problem.

8. Important Discovery: APPS Account Was Being Locked

At another point during testing, the account showed:

USERNAME        ACCOUNT_STATUS
--------------- -------------------------
APPS            LOCKED(TIMED)

The account was unlocked:

ALTER USER APPS ACCOUNT UNLOCK;

Then a direct connection succeeded:

sqlplus apps@TESTDB

SHOW USER;
USER is "APPS"

This confirmed that the APPS credentials being used for the direct SQL*Plus connection were valid at that point. However, subsequent attempts to run the TXK utility showed the account locked again:

ORA-28000: The account is locked.

This pattern indicated that something was repeatedly attempting authentication with invalid credentials, causing the account to lock after a number of failed attempts. The generic TXK error message “Invalid APPS database user credentials” did not hint at this root cause.

9. SplashBI Was Contributing to the Lockout

During the investigation, SplashBI sessions were identified as contributing to repeated APPS authentication attempts, and stale credentials were suspected.

Those sessions were stopped/killed before continuing with the TXK procedure.

This was a critical troubleshooting lesson:

Do not repeatedly unlock APPS and rerun the TXK utility while another application is continuously attempting to authenticate with stale APPS credentials.

Doing so simply creates another lockout cycle. After the interfering sessions were stopped, APPS authentication could be validated successfully, and the TXK procedure proceeded.

10. A Second Prerequisite Issue: Missing AppsUtil JRE

This was a separate issue discovered during the later stages of the TXK workflow. An earlier execution of syncUtlFileDir had shown:

FUNCTION: main::getJavaCmd

ERRORMSG:
Invalid JDK TOP:
$ORACLE_HOME/appsutil/jre/bin/java

I checked:

ls -ld $ORACLE_HOME/appsutil/jre

and found:

No such file or directory

However, the AppsUtil clone area contained a valid JRE:

ls -ld $ORACLE_HOME/appsutil/clone/jre

and:

ls -l $ORACLE_HOME/appsutil/clone/jre/bin/java

The Java executable was present. Validation:

$ORACLE_HOME/appsutil/clone/jre/bin/java -version

java version "1.8.0_451"
Java(TM) SE Runtime Environment

11. Backup Before Correcting AppsUtil

Before making the AppsUtil change, a backup was taken:

cd $ORACLE_HOME/appsutil

tar -czf $ORACLE_BASE/TESTDB/appsutil_pre_jre_fix_$(date +%Y%m%d_%H%M%S).tar.gz \
    TESTDB_dbctx.xml \
    txkSetCfgCDB.env \
    clone/jre

The backup was verified:

ls -lh $ORACLE_BASE/TESTDB/appsutil_pre_jre_fix_*.tar.gz

-rw-r--r-- 1 oracle dba 46M Aug 18 11:32 appsutil_pre_jre_fix_20260818_113218.tar.gz

This provided a rollback point before modifying the directory structure.

12. Restore the Expected AppsUtil JRE

The TXK source explicitly showed that getJavaCmd expected:

$ORACLE_HOME/appsutil/jre/bin/java

The existing valid JRE under clone/jre was copied to the expected location:

cp -pR $ORACLE_HOME/appsutil/clone/jre \
       $ORACLE_HOME/appsutil/jre

Validation:

ls -ld $ORACLE_HOME/appsutil/jre
ls -l $ORACLE_HOME/appsutil/jre/bin/java

$ORACLE_HOME/appsutil/jre/bin/java -version
java version "1.8.0_451"

This restored the Java path expected by the TXK utility.

13. Existing UTL_FILE_DIR Configuration

An earlier execution had already generated:

$ORACLE_HOME/dbs/TESTDB_utlfiledir.txt

The original getUtlFileDir execution had failed during APPS credential validation when the account was locked. After resolving the APPS lockout and stopping the interfering authentication attempts, the TXK workflow was able to proceed. The existing configuration file from the previous successful run was available as a reference for the required filesystem paths.

The paths included:

$ORACLE_BASE/TESTDB/temp
$APP_MOUNT/TESTDB/inst/apps/TESTDB_dbctx/appltmp
$APP_MOUNT/TESTDB/apps/apps_st/comn/temp
$ORACLE_BASE/TESTDB/db/tech_st/19.3.0.0/admin/po_terms
$ORACLE_BASE/TESTDB/db/tech_st/temp/TESTDB

14. Create the Directory Objects

The TXK utility was then run in createDirObject mode for each required OS path.

perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl \
-contextfile=$CONTEXT_FILE \
-oraclehome=$ORACLE_HOME \
-outdir=$ORACLE_HOME/appsutil/log \
-mode=createDirObject

The utility prompted for:

Enter the APPS Password:
Enter the SYSTEM Password:
Enter the OS path for which directory object needs to be created:

Each of the required paths completed with:

Successfully Completed the script
ERRORCODE = 0 ERRORCODE_END

15. Run setUtlFileDir

The next step was:

perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl \
-contextfile=$CONTEXT_FILE \
-oraclehome=$ORACLE_HOME \
-outdir=$ORACLE_HOME/appsutil/log \
-mode=setUtlFileDir

The script returned:

WARNING: Incorrect value s_applptmp detected on Apps Tier nodes.

However:

Successfully Completed the script
ERRORCODE = 0 ERRORCODE_END

Therefore the operation itself succeeded, while the s_applptmp issue was recorded as a warning for further review.

16. Synchronize UTL_FILE_DIR

The next documented step was:

perl $ORACLE_HOME/appsutil/bin/txkCfgUtlfileDir.pl \
-contextfile=$CONTEXT_FILE \
-oraclehome=$ORACLE_HOME \
-outdir=$ORACLE_HOME/appsutil/log \
-mode=syncUtlFileDir \
-skipautoconfig=yes

This completed successfully:

Successfully Completed the script
ERRORCODE = 0 ERRORCODE_END

At this point the TXK UTL_FILE_DIR synchronization portion was complete.

17. Clean EBS Node Configuration

The next database-side EBS step was:

EXEC FND_CONC_CLONE.SETUP_CLEAN;

COMMIT;

Output:

PL/SQL procedure successfully completed.

Commit complete.

Immediately afterward, querying FND_NODES returned:

no rows selected

This was expected at this point because SETUP_CLEAN removes the existing node registrations so that AutoConfig can rebuild them.

18. Run Database AutoConfig

Database AutoConfig was then executed from the database context directory:

cd $ORACLE_HOME/appsutil
sh adautocfg.sh

The script prompted:

Enter the APPS user password:

The AutoConfig log was located at:

$ORACLE_HOME/appsutil/log/TESTDB_dbctx/[timestamp]/adconfig.log

The important output was:

Context Value Management will now update the Context file

Updating Context file...COMPLETED

Attempting upload of Context file and templates to database...COMPLETED

Updating rdbms version in Context file to db19
Updating rdbms type in Context file to 64 bits

Configuring templates from ORACLE_HOME ...

AutoConfig completed successfully.

This confirmed that the database AutoConfig completed successfully.

19. Validate EBS Node Registration

After AutoConfig completed successfully, the final validation was to confirm that the EBS node registrations had been recreated.

SET LINES 200
SET PAGES 50

COLUMN node_name FORMAT A15
COLUMN host_name FORMAT A30
COLUMN status FORMAT A10
COLUMN cp FORMAT A5
COLUMN forms FORMAT A5
COLUMN web FORMAT A5
COLUMN admin FORMAT A5
COLUMN db FORMAT A5
COLUMN webhost FORMAT A30
COLUMN last_update FORMAT A20

ALTER SESSION SET NLS_DATE_FORMAT='DD-MON-YYYY HH24:MI:SS';

SELECT node_name,
       host || '.' || domain host_name,
       DECODE(status,'Y','active','inactive') status,
       DECODE(support_cp,'Y','yes','no') cp,
       DECODE(support_forms,'Y','yes','no') forms,
       DECODE(support_web,'Y','yes','no') web,
       DECODE(support_admin,'Y','yes','no') admin,
       DECODE(support_db,'Y','yes','no') db,
       webhost,
       last_update_date
FROM apps.fnd_nodes
WHERE node_name != 'AUTHENTICATION'
ORDER BY 7,1;

Output:

NODE_NAME       HOST_NAME                      STATUS     CP    FORMS WEB   ADMIN DB    WEBHOST                   LAST_UPDATE_DATE
--------------- ------------------------------ ---------- ----- ----- ----- ----- ----- ------------------------- --------------------
EBSNODE         [masked-hostname]              inactive   no    no    no    no    yes                           18-AUG-2026 12:50:45

The node registration was successfully rebuilt with the database tier configuration active (DB = yes) and other tiers set to inactive pending app-tier AutoConfig execution.

Root Cause: Two Independent Issues

The investigation identified two distinct, independent issues that required separate solutions. This is important to understand because confusing the two would have masked the actual root cause in each case.

Issue 1 — APPS Account Lockout (Database Authentication Layer)

The TXK utility reported:

Invalid APPS database user credentials.

The detailed validation log showed:

ORA-28000: The account is locked.

The APPS account was repeatedly becoming LOCKED(TIMED) due to repeated authentication attempts. SplashBI sessions were identified as contributing to those attempts. This was resolved by stopping the interfering sessions and unlocking the APPS account.

Issue 2 — Missing AppsUtil JRE (Environment/Configuration Layer)

During the later TXK workflow, syncUtlFileDir encountered:

Invalid JDK TOP:
$ORACLE_HOME/appsutil/jre/bin/java

The expected JRE path did not exist, while a valid JRE was available under:

$ORACLE_HOME/appsutil/clone/jre

This was resolved by copying the JRE to the expected AppsUtil location after taking a backup. This issue was completely independent of the APPS account lockout and would have persisted even if APPS had been unlocked from the start.

Key lesson: Do not conflate unrelated failures during troubleshooting. Each issue required its own diagnosis and its own fix. Resolving one would not have solved the other.

Troubleshooting Sequence

Phase 1: APPS/TXK Troubleshooting

1. Validate CDB/PDB state
2. Validate EBS context configuration
3. Validate TNS connectivity
4. Check APPS account status
5. Inspect validate_apps_password.log
6. Identify repeated APPS authentication attempts
7. Stop interfering SplashBI sessions
8. Unlock APPS
9. Validate direct APPS login

Phase 2: TXK/UTL_FILE_DIR Configuration

1. Verify AppsUtil JRE
2. Take AppsUtil backup
3. Restore expected appsutil/jre if required
4. Create directory objects
5. setUtlFileDir
6. syncUtlFileDir -skipautoconfig=yes
7. FND_CONC_CLONE.SETUP_CLEAN
8. COMMIT
9. Run Database AutoConfig
10. Validate FND_NODES
11. Perform final verification

Troubleshooting Lessons

1. Don’t trust the top-level TXK error blindly

This:

Invalid APPS database user credentials

does not necessarily mean the APPS password is wrong.

Always inspect:

validate_apps_password.log

In this case it exposed:

ORA-28000: The account is locked.

2. Check for external applications causing APPS lockouts

If APPS repeatedly becomes:

LOCKED(TIMED)

don’t continuously unlock it and retry.

Look for applications such as:

• SplashBI
• integrations
• monitoring tools
• application servers
• scheduled jobs

that may still have stale APPS credentials.

3. In a CDB/PDB environment, verify the complete connection path

Validate:

• CDB
• PDB
• SERVICE_NAME
• TNS alias
• PORT
• APPS account

For example:

tnsping TESTDB
sqlplus apps@TESTDB

4. Inspect the TXK source when the error is unclear

The getJavaCmd function made the missing JRE requirement explicit in the source code. This was much more useful than simply treating the Invalid JDK TOP error as a generic Java problem.

5. Take a backup before modifying AppsUtil

The backup provided a simple rollback point:

tar -czf $ORACLE_BASE/TESTDB/appsutil_pre_jre_fix_$(date +%Y%m%d_%H%M%S).tar.gz \
    TESTDB_dbctx.xml \
    txkSetCfgCDB.env \
    clone/jre

Final Validation

With AutoConfig completed successfully, the final validations confirmed that the database-side configuration was healthy and that the EBS node registration had been rebuilt:

• FND_NODES query returned the EBSNODE with DB support active
• APPS login successful
• AutoConfig completed with no errors
• txkCfgUtlfileDir.pl returned ERRORCODE = 0 for all modes
• UTL_FILE_DIR directory objects successfully created

The database-tier configuration was now ready for app-tier AutoConfig to follow, which would complete the clone or refresh procedure.

Closing

This issue initially looked like a simple APPS authentication problem. The actual troubleshooting required following the failure through several layers: APPS account state → PDB/service connectivity → authentication log inspection → identification of external authentication interference → TXK environment validation → prerequisite verification → directory object creation → EBS node cleanup → AutoConfig.

The key was to validate each layer independently and avoid assuming causality between separate findings. The APPS lockout issue and the missing JRE issue were distinct problems requiring distinct solutions.

ما شاء الله

How I Tracked Down a Crash Sequence Across EBS Application Logs, OPMN, and AIX OS Events

Alhamdulillah.

Around 3 AM, the production EBS application tier was in an unstable state. The application tier was experiencing repeated process crashes and restart failures, and the OPMN logs showed multiple processes entering a recovery cycle.

I start scrolling through logs and find the kind of chaos that makes troubleshooting interesting: OACORE processes that won’t stay alive, Forms instances crashing in sequence, OPMN repeatedly detecting failed JVM processes and attempting to restart them.

This is the story of how a single incident revealed the importance of timeline building, log correlation, and knowing the difference between recovery and root cause.

The Environment

• Oracle E-Business Suite R12.1
• AIX 7.1
• OC4J 10.1.3 with OPMN management
• IBM Java 7
• Multiple OACORE and Forms processes

The First Red Flag

I start with the OACORE process startup logs. The timeline shows a clean start:

20:57 — OACORE Startup
26/08/12 20:57:06 Start process

But three seconds later, something unexpected appears. The JVM is configured with 2560 MB heap and Gencon garbage collection. Then:

26/08/12 20:57:09 WARNING: GC: Unable to access pending References.
26/08/12 20:57:09 WARNING: GC: Unable to access pending Reference lock.

That’s interesting, but not necessarily fatal. GC warnings can be transient. I keep digging.

In the same timeframe, OACORE logs show an RMI failure:

Error: Could not connect to the remote server
caused by: oracle.oc4j.security.KeyExchange.getSecretKey
caused by: Unsupported secret key algorithm: DES

This is worth noting. An unsupported DES algorithm error in the OC4J RMI layer. But here’s the thing: I have two observations now, and I don’t yet know if they’re related, or if either one caused what happens next.

The Cascade

Fast forward to 01:26 AM on August 13. The OPMN logs suddenly light up:

01:26 — Multiple Forms Processes Crash
26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~2
26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~5
26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~7
26/08/13 01:26:07 Process Crashed: default_group~forms~default_group~8
26/08/13 01:26:07 Process Crashed: default_group~forms-c4ws~default_group~1

Not just one. Not even two. Five processes are crashing in the same second. The simultaneous failure of multiple Forms-related processes suggested that the issue was broader than a single application process failure.

OPMN immediately begins detecting these failures. It does what it’s designed to do: attempt to restart failed processes. But the restart attempts are not succeeding cleanly:

Process Start Error: default_group~forms~default_group~4
Failed to construct stop command
Forcefully Terminating Process
Could not connect to socket while pinging an OC4J process
Operation: internal (Connection refused)

OPMN detects the failed processes and repeatedly attempts to restart them, but several restart attempts encounter process communication, socket, and port-information errors. This is the kind of situation where the logs quickly become a wall of repeated error messages as OPMN continues attempting to recover the affected processes.

The Port Information Mystery

As I dig deeper into the OPMN logs, I find another clue that doesn’t quite fit the picture yet:

Host and Port information for port type jms not sent by the OC4J process
WARNING: OC4J did not send the protocol value for port id: jms

OPMN relies on port information to manage processes. OPMN was not receiving the expected JMS port/protocol information from the OC4J process, which complicated process management and restart attempts.

At this point I have:

• GC warnings on OACORE startup
• An RMI/DES algorithm error
• Multiple OC4J processes crashing simultaneously
• OPMN unable to restart them cleanly
• Missing JMS port/protocol information

But which one caused the others? That’s when I make an important decision: instead of trying to decide, I document what I observe and build a timeline.

Build a Timeline, Not a Theory

When several application processes fail around the same period, it is tempting to take the first recognizable error and declare it the root cause. In this incident, the OACORE GC warnings and OC4J DES error were early clues, but the subsequent OPMN logs showed that the failure was broader.

At this point, I have a comprehensive picture of application failures. But I’ve been told the server was rebooted. So I check the AIX logs.

Incident Timeline

Time	Observation
Aug 12, 20:57	OACORE processes started
20:57:09	IBM JVM GC warnings observed (Unable to access pending References)
20:57	OC4J RMI connection failure — Unsupported secret key algorithm: DES
Aug 13, 01:26:07	Multiple Forms / Forms-C4WS processes crashed simultaneously
01:28 onward	OPMN restart attempts encountered process start, socket, and port-information errors
02:36	AIX system shutdown recorded; system subsequently rebooted
After reboot	Application services recovered and remained stable

The OS Logs Tell a Different Story

Now I verify the OS timeline. First, boot information:

who -b

This tells me the boot time was Aug 13 at 02:36. Next I check the AIX error log with a time-window query around the known failure period:

errpt -s 0813010026 -e 0813030026

This returns:

2BFA76F6   0813023626 T S SYSPROC
SYSTEM SHUTDOWN BY USER

Confirmed: SYSTEM SHUTDOWN BY USER at 02:36.

02:36 — AIX Reboot
System boot time: Aug 13 02:36

After the reboot, the application services came back up and remained stable.

What the Logs Proved — and What They Didn’t

Proved by the evidence:

• Multiple EBS application processes crashed
• OPMN attempted recovery and restarts
• Several restart attempts encountered communication and port-related errors
• AIX recorded a system shutdown by user at 02:36
• who -b confirmed the system boot time as 02:36
• Application services recovered after the reboot

Not proved by the evidence:

• That the GC warnings caused the incident
• That the DES error caused the process crashes
• That the JMS port issue was the initiating failure
• That the reboot fixed the underlying root cause
• The exact initiating cause of the overall process-crash cascade

The Critical Distinction

Here’s where I have to be careful about my conclusions.

The reboot restored service. That’s a fact. But did the reboot fix the root cause, or did it just clear transient state? These are two different things.

A reboot can:

• Clear unstable JVM runtime state
• Terminate stuck processes
• Clear transient socket/process communication state
• Reinitialize application-tier processes and their associated listeners/ports

None of those tell me what originally triggered the failure.

What I can confirm:

• OACORE: Startup with GC warnings
• OC4J RMI: DES algorithm error
• Forms and Forms-C4WS: Multiple process crashes during the recovery/restart sequence
• OPMN: Multiple restart attempts encountered process communication failures, connection refusals, and missing JMS port/protocol information
• Recovery: Reboot restored stability

What I cannot conclusively determine from logs: the exact initiating cause of the cascade.

When to Raise an Oracle Support SR

In production, your first priority is recovery. But recovery should not be your last action.

If you encounter OC4J/OPMN instability involving multiple process crashes and restart failures, here’s the right approach:

Before you reboot:
• If service impact is acceptable, preserve application logs, OPMN logs, and OS error logs (errpt)
• Consider opening an SR with Oracle Support before recovery actions
• Provide an SR reference for all follow-up communication

If service impact demands immediate recovery:
• Proceed with the reboot
• Collect and preserve log artifacts afterward
• Raise the SR as soon as practical with the preserved log artifacts
• Include full OPMN, OACORE, Forms logs and AIX errpt output

An SR with Oracle Support positions your incident for definitive analysis. It moves you from “we rebooted and it worked” to “here’s what happened, and here’s what Oracle Support says about it.”

The Troubleshooting Checklist

When you see OC4J instability, use these commands to build your timeline:

Grep for key patterns across logs:

grep -n -E "GC: Unable|Unsupported secret|Process Start Error|Process Crashed|Connection refused|Exception|ERROR|FATAL" <oacore_log>

Check OPMN for the restart cycle:

grep -n -E "Process Crashed|Process Start Error|Connection refused|Port information" <opmn.log>

Verify boot timeline and OS events:

who -b && uptime && last reboot | head -5

Extract the error events in a time window (the most useful approach):

errpt -s 0813010026 -e 0813030026   # Adjust dates for your incident

This time-window query is much more powerful than a simple grep because it shows you events in context, within the period your incident occurred.

What I Learned

Alhamdulillah for this incident, because it reinforced several critical principles:

The first error is not the root cause. I started with GC warnings and an RMI error. But the real story emerged from the sequence of crashes, not from the opening symptoms.

Build a timeline, not a theory. A timeline shows sequence and correlation; theories can introduce assumptions. Timestamps tied together the OACORE observations, the Forms crashes, OPMN’s restart failures, and ultimately the reboot recovery.

Recovery and RCA are different questions. The reboot answered “what restored service?” but not “what caused the failure?” It’s important to say the difference out loud.

Correlate application and OS logs. OPMN logs showed application behavior; AIX errpt showed OS timeline. Together they tell the full story.

Open an SR when multiple OC4J components fail. When you have a cascade of crashes involving Forms, OACORE, and OPMN restart failures, that’s beyond the scope of “try restarting the services.” Engage Oracle Support early with an SR. It positions you for definitive analysis and creates a record for future reference.

Closing Thoughts

This incident reinforced something I’ve learned through years of production troubleshooting in EBS and Oracle Database: patience with logs pays off.

The reboot restored application stability, but the available evidence did not establish whether the reboot cleared a transient runtime condition or whether the underlying initiating problem had already resolved.

Instead of manufacturing an RCA that the logs don’t support, I documented: application symptoms → OPMN behavior → OS timeline → reboot → recovery → RCA limitation.

For any DBA handling production incidents, this is the framework:

1. Document what the evidence proves
2. Clearly identify what it does not prove
3. Continue the investigation with Oracle Support if RCA is incomplete

ما شاء الله