Oracle EBS R12.2 Multi-Node Patching in Downtime Mode: Recovering a Failed Slave Node with restart=yes

Alhamdulillah, sharing another real patching session — this one from an Oracle E-Business Suite R12.2 multi-node environment. While applying a patch in downtime mode, I encountered an interesting situation: the patch completed successfully on the master node but failed on the slave node, while the final ADOP output still showed:

adop exiting with status = 0 (Success)

This post walks through the complete sequence: pre-checks, the slave-node failure, investigation, recovery using restart=yes, and the SQL checks used to verify the final patch state.

Environment

• Oracle E-Business Suite R12.2.13
• AD/TXK Delta 15 (ADOP C.Delta.15)
• Two application nodes: appsnode1 (master/admin node) and webnode1 (slave/web node)
• Oracle Database 19c on dbnode1
• Patch mode: downtime (single file system apply; no online patching cutover involved)

Pre-Checks Before the Apply

Before starting the patch, I checked three things: the current ADOP session status, application processes on all application nodes, and the current file-system edition.

From the master node:

[applmgr@appsnode1]$ adop -status
Node Name       Node Type  Phase           Status
--------------- ---------- --------------- ---------------
appsnode1       master     APPLY           ACTIVE
                           CLEANUP         NOT STARTED
webnode1        slave      APPLY           ACTIVE
                           CLEANUP         NOT STARTED

I also confirmed I was operating from the run edition:

[applmgr@appsnode1]$ echo $FILE_EDITION
run

Then I checked for application-tier processes:

[applmgr@appsnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|FNDCRM|FNDOPP|oacore|forms|WebLogic|NodeManager' | grep -v grep

There was no output, confirming the expected services were down on the master node.

Important: In a multi-node environment, repeat these process checks on every application node. Checking only the master node is not sufficient — a condition on a slave node can otherwise go unnoticed until the apply reaches it.

Starting the Downtime Apply

The downtime apply was initiated from the master node:

[applmgr@appsnode1]$ adop phase=apply apply_mode=downtime patches=<patch_number>

ADOP manages execution across the registered application-tier nodes, so there was no need to manually start ADOP on the slave node.

During initialization, ETCC reported missing database fixes:

[WARNING]   ETCC: The following required database fixes have not been applied to node dbnode1:
            ...
            Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

ETCC warnings should always be reviewed before proceeding. In this case, the apply was allowed to continue based on the maintenance plan, while the outstanding database fixes were documented and tracked separately for a subsequent maintenance activity.

Master Succeeded, Slave Failed

The patch application succeeded on the master node but failed on the slave:

Applying <patch> patch(es) on admin node: [appsnode1].
        txkADOPEvalSrvStatus.pl returned SUCCESS
Applying <patch> patch(es) on node(s): [webnode1].
Running in Serial
    [ERROR]     adop phase=apply failed on Node: "webnode1"
Summary report for current adop session:
    Node webnode1: Failed
       - Apply status:      Failed
    Node appsnode1: Completed successfully
       - Apply status:      Completed successfully
adop exiting with status = 0 (Success)

This is an important operational lesson. Despite the slave-node failure being clearly reported in the summary, the ADOP invocation ended with adop exiting with status = 0 (Success).

Therefore, relying only on the shell return code or the final ADOP exit line is not sufficient for determining whether every application node completed successfully. The per-node summary must also be reviewed.

I confirmed the state with:

adop -status -detail

which showed:

Node Name       Node Type  Phase           Status
--------------- ---------- --------------- ---------------
appsnode1       master     APPLY           ACTIVE
webnode1        slave      APPLY           FAILED

For automated patching this matters even more: scripts should validate the ADOP session and per-node phase status rather than treating $? = 0 alone as proof of a successful multi-node patch.

Investigating the Slave Node

I then logged into webnode1. Application services were expected to be completely down, so I checked for NodeManager:

[applmgr@webnode1]$ ps -ef | grep -i nodemanager | grep -v grep
applmgr  4148261  1  0 Jun18 ... weblogic.NodeManager -v

A NodeManager process was still running, and had been since a previous month.

I checked whether other WebLogic or EBS application processes were present:

[applmgr@webnode1]$ ps -ef | grep -i weblogic | grep -v grep

Only the NodeManager process remained. Concurrent Manager processes were also absent:

[applmgr@webnode1]$ ps -ef | grep FNDLIBR | grep -v grep

No output was returned.

Since the environment was in a planned patching outage and no managed servers were running, the stale NodeManager process was stopped:

[applmgr@webnode1]$ kill 4148261

I then verified again:

[applmgr@webnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|oacore|forms|WebLogic|NodeManager' | grep -v grep

No relevant processes remained.

A Note About Root Cause

The presence of the stale NodeManager was an abnormal condition that needed to be corrected before retrying the patch. However, unless the ADOP logs explicitly identify that process as the reason for the failure, it is safer not to state that NodeManager was definitively the root cause.

What can be established from this incident is:

1. The slave-node apply failed.
2. A stale NodeManager process was discovered on that node.
3. The process was stopped after confirming application services were intended to be down.
4. The failed ADOP apply subsequently completed successfully with restart=yes.

For a formal RCA, the relevant ADOP and slave-node logs should be used to establish the exact failure mechanism.

Also, before terminating any process, verify that:

• It belongs to the expected EBS environment/file system.
• No managed servers depend on it.
• Application services are intended to be down.
• The action complies with your production change procedure.

Recovering the Failed Apply with restart=yes

After cleaning up the slave node, I returned to the master node. The existing failed apply was resumed using:

[applmgr@appsnode1]$ adop phase=apply patches=<patch_number> apply_mode=downtime restart=yes

ADOP detected the existing session:

Checking for existing adop sessions.
Application tier services are down.
Continuing with the existing session [Session ID: <n>].

The apply was then processed successfully:

Applying <patch> patch(es) on admin node: [appsnode1].
        txkADOPEvalSrvStatus.pl returned SUCCESS
Applying <patch> patch(es) on node(s): [webnode1].
Running in Serial
        txkADOPEvalSrvStatus.pl returned SUCCESS

The final summary showed both nodes successful:

Summary report for current adop session:
    Node webnode1: Completed successfully
       - Apply status:      Completed successfully
    Node appsnode1: Completed successfully
       - Apply status:      Completed successfully

The important point here is that I resumed the existing ADOP session rather than initiating an unrelated fresh patching attempt. The recovery command was executed from the master/admin node, allowing ADOP to coordinate processing on the registered slave node.

For the exact semantics of restart=yes in downtime mode and multi-node processing, refer to the Oracle E-Business Suite Patching Guide and the relevant My Oracle Support documentation. What I can report from this incident is the observed behaviour above: ADOP continued the existing session and both nodes subsequently reported success.

Post-Restart Validation

After the successful rerun, I checked the detailed ADOP status:

[applmgr@appsnode1]$ adop -status -detail

The nodes no longer showed a failed apply state.

I also scanned the latest ADOP logs:

[applmgr@appsnode1]$ adopscanlog -latest=yes
Scanning .../log/adop/<session>/ directory ...
No Errors.

adopscanlog is a useful quick check after an ADOP phase because it searches the session logs for reported errors that may otherwise be easy to miss.

SQL Verification: Confirming Patch Registration

I don’t rely only on the ADOP console output. After patching, I also verify the database-side patch records.

1. Check AD_BUGS

SET LINES 200
COL BUG_NUMBER FORMAT A15
COL CREATION_DATE FORMAT A22
SELECT bug_number,
       creation_date
FROM   ad_bugs
WHERE  bug_number = '<patch_number>';

This confirms whether the specified bug/patch number is recorded in AD_BUGS. Because patch contents and registration behaviour can vary, this check should be used together with ADOP status, logs, and the relevant applied-patch records rather than as standalone proof of complete patch application. Running the same query before patching is also useful, because it establishes a clear before-and-after state.

2. Check Applied Patch Records

SELECT applied_patch_id,
       patch_name,
       patch_type,
       source_code,
       creation_date,
       last_update_date,
       data_model_done_flag
FROM   ad_applied_patches
WHERE  patch_name = '<patch_number>';

Depending on the patch structure, this helps confirm the corresponding patch-driver records.

3. Verify AD/TXK Code Levels

COL ABBREVIATION FORMAT A15
COL CODELEVEL FORMAT A25
SELECT abbreviation,
       codelevel
FROM   ad_trackable_entities
WHERE  abbreviation IN ('ad','txk')
ORDER  BY abbreviation;

For this environment the query returned:

ABBREVIATION    CODELEVEL
--------------- -------------------------
ad              C.15
txk             C.15

This check is especially important when applying AD/TXK-related patches or preparing for an EBS Release Update Pack.

4. Verify the EBS Release

SELECT release_name
FROM   fnd_product_groups;

Expected for this environment: 12.2.13.

5. Verify Node Registration

COL NODE_NAME FORMAT A25
SELECT node_name,
       support_cp,
       support_forms,
       support_web,
       support_admin
FROM   fnd_nodes
ORDER  BY node_name;

This provides a quick sanity check of application-tier node registration and responsibilities.

6. Verify ADOP Valid Nodes

SELECT *
FROM   adop_valid_nodes
ORDER  BY node_name;

Both application nodes should appear as expected.

In this case, the final checks confirmed:

• The bug/patch number was recorded in AD_BUGS.
• The corresponding applied-patch records were present.
• AD and TXK were at the expected C.15 level.
• EBS remained at Release 12.2.13.
• Both application nodes were properly registered in FND_NODES.
• Both nodes appeared correctly in ADOP_VALID_NODES.
• adopscanlog reported no errors after the successful rerun.

Together, these checks provide much stronger verification than relying on the final ADOP console message alone.

Complete the Session with Cleanup

After confirming the apply completed successfully, the session still showed:

CLEANUP         NOT STARTED

The cleanup phase was then run:

adop phase=cleanup

After cleanup, perform another status and log review to make sure the session is in the expected final state.

Key Takeaways

Don’t rely only on ADOP’s exit status in a multi-node environment. In this incident, ADOP printed status = 0 (Success) even though the per-node summary clearly showed the slave node had failed.

Check every application node before a downtime apply. A clean process list on the master does not guarantee the slave nodes are equally clean.

Use the per-node ADOP status as part of your success criteria — adop -status -detail.

Investigate the failed node before retrying. In this case a stale NodeManager process was discovered and removed before the successful retry. Unless supported by the logs, however, avoid treating correlation as a confirmed root cause.

Resume the existing failed apply appropriately — after correcting the condition on the slave node, the existing apply was resumed from the master node with restart=yes.

Use adopscanlog -latest=yes after the rerun.

Verify patch registration in the database — AD_BUGS, AD_APPLIED_PATCHES, AD_TRACKABLE_ENTITIES, FND_NODES, ADOP_VALID_NODES, FND_PRODUCT_GROUPS.

Run ETCC and review its findings. If required database fixes are deferred, document and track them for the appropriate maintenance window rather than allowing them to become forgotten technical debt.

Complete the ADOP cleanup phase — adop phase=cleanup.

Final Thought

Multi-node EBS patching adds a layer of validation that is easy to overlook: success on the master node does not automatically mean success across the application tier.

The most useful lesson from this incident was not the retry command itself, but the validation process around it:

Check every node → read the ADOP per-node summary → investigate the failed node → resume the existing session → scan the logs → verify the database records → complete cleanup.

That sequence gives you a far more defensible and auditable patching result than relying on a single Success message at the end of the command.

I hope this helps anyone troubleshooting a failed application-tier node during Oracle E-Business Suite R12.2 downtime patching. Feel free to leave a comment if you have questions.

Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.