Syed Anwar Ahmed – Oracle DBA Blog

Category: EBS Patching

Oracle E-Business Suite patching guides, ADOP troubleshooting and best practices

  • Oracle EBS R12.2 Multi-Node Patching in Downtime Mode: Recovering a Failed Slave Node with restart=yes

    Alhamdulillah, sharing another real patching session — this one from an Oracle E-Business Suite R12.2 multi-node environment. While applying a patch in downtime mode, I encountered an interesting situation: the patch completed successfully on the master node but failed on the slave node, while the final ADOP output still showed:

    adop exiting with status = 0 (Success)

    This post walks through the complete sequence: pre-checks, the slave-node failure, investigation, recovery using restart=yes, and the SQL checks used to verify the final patch state.

    Environment

    • Oracle E-Business Suite R12.2.13
    • AD/TXK Delta 15 (ADOP C.Delta.15)
    • Two application nodes: appsnode1 (master/admin node) and webnode1 (slave/web node)
    • Oracle Database 19c on dbnode1
    • Patch mode: downtime (single file system apply; no online patching cutover involved)

    Pre-Checks Before the Apply

    Before starting the patch, I checked three things: the current ADOP session status, application processes on all application nodes, and the current file-system edition.

    From the master node:

    [applmgr@appsnode1]$ adop -status
    Node Name       Node Type  Phase           Status
    --------------- ---------- --------------- ---------------
    appsnode1       master     APPLY           ACTIVE
                               CLEANUP         NOT STARTED
    webnode1        slave      APPLY           ACTIVE
                               CLEANUP         NOT STARTED

    I also confirmed I was operating from the run edition:

    [applmgr@appsnode1]$ echo $FILE_EDITION
    run

    Then I checked for application-tier processes:

    [applmgr@appsnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|FNDCRM|FNDOPP|oacore|forms|WebLogic|NodeManager' | grep -v grep

    There was no output, confirming the expected services were down on the master node.

    Important: In a multi-node environment, repeat these process checks on every application node. Checking only the master node is not sufficient — a condition on a slave node can otherwise go unnoticed until the apply reaches it.

    Starting the Downtime Apply

    The downtime apply was initiated from the master node:

    [applmgr@appsnode1]$ adop phase=apply apply_mode=downtime patches=<patch_number>

    ADOP manages execution across the registered application-tier nodes, so there was no need to manually start ADOP on the slave node.

    During initialization, ETCC reported missing database fixes:

    [WARNING]   ETCC: The following required database fixes have not been applied to node dbnode1:
                ...
                Refer to My Oracle Support Knowledge Document 1594274.1 for instructions.

    ETCC warnings should always be reviewed before proceeding. In this case, the apply was allowed to continue based on the maintenance plan, while the outstanding database fixes were documented and tracked separately for a subsequent maintenance activity.

    Master Succeeded, Slave Failed

    The patch application succeeded on the master node but failed on the slave:

    Applying <patch> patch(es) on admin node: [appsnode1].
            txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
        [ERROR]     adop phase=apply failed on Node: "webnode1"
    Summary report for current adop session:
        Node webnode1: Failed
           - Apply status:      Failed
        Node appsnode1: Completed successfully
           - Apply status:      Completed successfully
    adop exiting with status = 0 (Success)

    This is an important operational lesson. Despite the slave-node failure being clearly reported in the summary, the ADOP invocation ended with adop exiting with status = 0 (Success).

    Therefore, relying only on the shell return code or the final ADOP exit line is not sufficient for determining whether every application node completed successfully. The per-node summary must also be reviewed.

    I confirmed the state with:

    adop -status -detail

    which showed:

    Node Name       Node Type  Phase           Status
    --------------- ---------- --------------- ---------------
    appsnode1       master     APPLY           ACTIVE
    webnode1        slave      APPLY           FAILED

    For automated patching this matters even more: scripts should validate the ADOP session and per-node phase status rather than treating $? = 0 alone as proof of a successful multi-node patch.

    Investigating the Slave Node

    I then logged into webnode1. Application services were expected to be completely down, so I checked for NodeManager:

    [applmgr@webnode1]$ ps -ef | grep -i nodemanager | grep -v grep
    applmgr  4148261  1  0 Jun18 ... weblogic.NodeManager -v

    A NodeManager process was still running, and had been since a previous month.

    I checked whether other WebLogic or EBS application processes were present:

    [applmgr@webnode1]$ ps -ef | grep -i weblogic | grep -v grep

    Only the NodeManager process remained. Concurrent Manager processes were also absent:

    [applmgr@webnode1]$ ps -ef | grep FNDLIBR | grep -v grep

    No output was returned.

    Since the environment was in a planned patching outage and no managed servers were running, the stale NodeManager process was stopped:

    [applmgr@webnode1]$ kill 4148261

    I then verified again:

    [applmgr@webnode1]$ ps -ef | egrep 'FNDLIBR|FNDSM|oacore|forms|WebLogic|NodeManager' | grep -v grep

    No relevant processes remained.

    A Note About Root Cause

    The presence of the stale NodeManager was an abnormal condition that needed to be corrected before retrying the patch. However, unless the ADOP logs explicitly identify that process as the reason for the failure, it is safer not to state that NodeManager was definitively the root cause.

    What can be established from this incident is:

    1. The slave-node apply failed.
    2. A stale NodeManager process was discovered on that node.
    3. The process was stopped after confirming application services were intended to be down.
    4. The failed ADOP apply subsequently completed successfully with restart=yes.

    For a formal RCA, the relevant ADOP and slave-node logs should be used to establish the exact failure mechanism.

    Also, before terminating any process, verify that:

    • It belongs to the expected EBS environment/file system.
    • No managed servers depend on it.
    • Application services are intended to be down.
    • The action complies with your production change procedure.

    Recovering the Failed Apply with restart=yes

    After cleaning up the slave node, I returned to the master node. The existing failed apply was resumed using:

    [applmgr@appsnode1]$ adop phase=apply patches=<patch_number> apply_mode=downtime restart=yes

    ADOP detected the existing session:

    Checking for existing adop sessions.
    Application tier services are down.
    Continuing with the existing session [Session ID: <n>].

    The apply was then processed successfully:

    Applying <patch> patch(es) on admin node: [appsnode1].
            txkADOPEvalSrvStatus.pl returned SUCCESS
    Applying <patch> patch(es) on node(s): [webnode1].
    Running in Serial
            txkADOPEvalSrvStatus.pl returned SUCCESS

    The final summary showed both nodes successful:

    Summary report for current adop session:
        Node webnode1: Completed successfully
           - Apply status:      Completed successfully
        Node appsnode1: Completed successfully
           - Apply status:      Completed successfully

    The important point here is that I resumed the existing ADOP session rather than initiating an unrelated fresh patching attempt. The recovery command was executed from the master/admin node, allowing ADOP to coordinate processing on the registered slave node.

    For the exact semantics of restart=yes in downtime mode and multi-node processing, refer to the Oracle E-Business Suite Patching Guide and the relevant My Oracle Support documentation. What I can report from this incident is the observed behaviour above: ADOP continued the existing session and both nodes subsequently reported success.

    Post-Restart Validation

    After the successful rerun, I checked the detailed ADOP status:

    [applmgr@appsnode1]$ adop -status -detail

    The nodes no longer showed a failed apply state.

    I also scanned the latest ADOP logs:

    [applmgr@appsnode1]$ adopscanlog -latest=yes
    Scanning .../log/adop/<session>/ directory ...
    No Errors.

    adopscanlog is a useful quick check after an ADOP phase because it searches the session logs for reported errors that may otherwise be easy to miss.

    SQL Verification: Confirming Patch Registration

    I don’t rely only on the ADOP console output. After patching, I also verify the database-side patch records.

    1. Check AD_BUGS

    SET LINES 200
    COL BUG_NUMBER FORMAT A15
    COL CREATION_DATE FORMAT A22
    SELECT bug_number,
           creation_date
    FROM   ad_bugs
    WHERE  bug_number = '<patch_number>';

    This confirms whether the specified bug/patch number is recorded in AD_BUGS. Because patch contents and registration behaviour can vary, this check should be used together with ADOP status, logs, and the relevant applied-patch records rather than as standalone proof of complete patch application. Running the same query before patching is also useful, because it establishes a clear before-and-after state.

    2. Check Applied Patch Records

    SELECT applied_patch_id,
           patch_name,
           patch_type,
           source_code,
           creation_date,
           last_update_date,
           data_model_done_flag
    FROM   ad_applied_patches
    WHERE  patch_name = '<patch_number>';

    Depending on the patch structure, this helps confirm the corresponding patch-driver records.

    3. Verify AD/TXK Code Levels

    COL ABBREVIATION FORMAT A15
    COL CODELEVEL FORMAT A25
    SELECT abbreviation,
           codelevel
    FROM   ad_trackable_entities
    WHERE  abbreviation IN ('ad','txk')
    ORDER  BY abbreviation;

    For this environment the query returned:

    ABBREVIATION    CODELEVEL
    --------------- -------------------------
    ad              C.15
    txk             C.15

    This check is especially important when applying AD/TXK-related patches or preparing for an EBS Release Update Pack.

    4. Verify the EBS Release

    SELECT release_name
    FROM   fnd_product_groups;

    Expected for this environment: 12.2.13.

    5. Verify Node Registration

    COL NODE_NAME FORMAT A25
    SELECT node_name,
           support_cp,
           support_forms,
           support_web,
           support_admin
    FROM   fnd_nodes
    ORDER  BY node_name;

    This provides a quick sanity check of application-tier node registration and responsibilities.

    6. Verify ADOP Valid Nodes

    SELECT *
    FROM   adop_valid_nodes
    ORDER  BY node_name;

    Both application nodes should appear as expected.

    In this case, the final checks confirmed:

    • The bug/patch number was recorded in AD_BUGS.
    • The corresponding applied-patch records were present.
    • AD and TXK were at the expected C.15 level.
    • EBS remained at Release 12.2.13.
    • Both application nodes were properly registered in FND_NODES.
    • Both nodes appeared correctly in ADOP_VALID_NODES.
    • adopscanlog reported no errors after the successful rerun.

    Together, these checks provide much stronger verification than relying on the final ADOP console message alone.

    Complete the Session with Cleanup

    After confirming the apply completed successfully, the session still showed:

    CLEANUP         NOT STARTED

    The cleanup phase was then run:

    adop phase=cleanup

    After cleanup, perform another status and log review to make sure the session is in the expected final state.

    Key Takeaways

    Don’t rely only on ADOP’s exit status in a multi-node environment. In this incident, ADOP printed status = 0 (Success) even though the per-node summary clearly showed the slave node had failed.

    Check every application node before a downtime apply. A clean process list on the master does not guarantee the slave nodes are equally clean.

    Use the per-node ADOP status as part of your success criteria — adop -status -detail.

    Investigate the failed node before retrying. In this case a stale NodeManager process was discovered and removed before the successful retry. Unless supported by the logs, however, avoid treating correlation as a confirmed root cause.

    Resume the existing failed apply appropriately — after correcting the condition on the slave node, the existing apply was resumed from the master node with restart=yes.

    Use adopscanlog -latest=yes after the rerun.

    Verify patch registration in the database — AD_BUGS, AD_APPLIED_PATCHES, AD_TRACKABLE_ENTITIES, FND_NODES, ADOP_VALID_NODES, FND_PRODUCT_GROUPS.

    Run ETCC and review its findings. If required database fixes are deferred, document and track them for the appropriate maintenance window rather than allowing them to become forgotten technical debt.

    Complete the ADOP cleanup phase — adop phase=cleanup.

    Final Thought

    Multi-node EBS patching adds a layer of validation that is easy to overlook: success on the master node does not automatically mean success across the application tier.

    The most useful lesson from this incident was not the retry command itself, but the validation process around it:

    Check every node → read the ADOP per-node summary → investigate the failed node → resume the existing session → scan the logs → verify the database records → complete cleanup.

    That sequence gives you a far more defensible and auditable patching result than relying on a single Success message at the end of the command.

    I hope this helps anyone troubleshooting a failed application-tier node during Oracle E-Business Suite R12.2 downtime patching. Feel free to leave a comment if you have questions.

    Disclaimer: The views expressed on this blog are my own and do not reflect the views of my employer or any client. All environment names, hostnames, and identifiers used in this post are anonymized. Always test in a non-production environment before applying any change to production.

  • Oracle EBS R12.2 Dual File System Demystified: Run Edition, Patch Edition, and ADOP Explained

    Many Oracle EBS DBAs know how to run ADOP. Fewer understand why the dual file system exists or what is actually happening behind each phase. This post fills that gap — the architecture first, then the lifecycle, then the real-world traps.

    Why two file systems exist

    The biggest shift moving from R12.1 to R12.2 is that the application tier is no longer one file system. R12.2 keeps two full, identical copies of the application code — fs1 and fs2 — plus a third, non-editioned area, fs_ne.

    At any moment one of fs1/fs2 is the run file system (what users are connected to) and the other is the patch file system (where ADOP applies patches in the background). That is the whole point: you patch the idle copy while users keep working on the live one, then switch over in a short cutover window.

    The layout on disk

    /u01/oracle/<SID>/
│
├── fs1      (RUN or PATCH)
├── fs2      (PATCH or RUN)
└── fs_ne    (Non-editioned — never swaps)

    The two never have fixed roles. After every cycle the labels swap, so fs1 might be run today and patch after the next cutover:

    Before Cutover                 After Cutover

   Users                          Users
     |                              |
     v                              v
   RUN  = fs1                     RUN  = fs2
   PATCH = fs2                    PATCH = fs1

    If you remember only one thing from this article, remember this: fs1 and fs2 are just labels. RUN and PATCH are the roles that matter.

    Never hardcode a path assuming fs1 is always live. Read the environment, don’t memorize it.

    Determining the current run and patch file systems

    This is one of the most common questions in R12.2, and the answer is in the environment, not in your head:

    source EBSapps.env run

echo "RUN_BASE     = $RUN_BASE"
echo "PATCH_BASE   = $PATCH_BASE"
echo "FILE_EDITION = $FILE_EDITION"

    Example output:

    RUN_BASE     = /u01/oracle/<SID>/fs1
PATCH_BASE   = /u01/oracle/<SID>/fs2
FILE_EDITION = run

    FILE_EDITION tells you which copy your current shell is pointed at. A surprising number of “my patch went to the wrong place” problems are just someone working in the wrong edition’s shell.

    What fs_ne is and why it exists

    Most write-ups mention fs_ne and move on. It matters because it holds everything that must survive a cutover unchanged:

    fs_ne (non-editioned) contains data that must NOT switch at cutover:

  - Concurrent Manager logs and output
  - Application / debug logs
  - Inbound interface files
  - Outbound interface files
  - Custom file drops
  - Temp files

    If these lived inside fs1 and fs2, they would belong to whichever copy was live — and the moment you cut over to the other copy, those files would appear to “vanish.” Pulling them into a single non-editioned area means logs and interface files stay put no matter which file system is running.

    File system editions and database editions

    R12.2 pairs the dual file system with Edition-Based Redefinition (EBR) on the database. Each patching cycle stacks a new database edition on top of the last:

    COL edition_name        FORMAT A28
COL parent_edition_name FORMAT A28
COL usable              FORMAT A8

SELECT edition_name,
       NVL(parent_edition_name,'-') AS parent_edition_name,
       usable
FROM   dba_editions
ORDER  BY edition_name;
    EDITION_NAME                 PARENT_EDITION_NAME          USABLE
---------------------------- ---------------------------- --------
ORA$BASE                     -                            YES
V_20260115_0830              ORA$BASE                     YES
V_20260301_2145              V_20260115_0830              YES

    The run edition is the currently active database edition. During prepare, ADOP creates a new child patch edition where database changes are applied. During cutover, that patch edition becomes the new run edition. The file system and the database edition move together — patch edition on the patch file system, run edition on the run file system.

    The ADOP cycle, phase by phase

    ADOP (AD Online Patching) drives everything. A full cycle is five phases:

    1. prepare — validates the environment, synchronizes the patch file system with the run file system when necessary, and creates the new database patch edition. Run and patch are now identical and ready to diverge.
    2. apply — applies your patch(es) to the patch file system and patch edition only. Users are untouched. You can apply many patches across multiple apply calls in one cycle.
    3. finalize — does the heavy compile/precompute work it can do before cutover, to keep the downtime window short.
    4. cutover — the only disruptive step. Services bounce, the patch file system becomes the new run, and the patch edition becomes the new run edition.
    5. cleanup — drops obsolete editions and old objects so the system is ready for the next cycle.

    “Online patching” — so why do users still get disconnected?

    This is the question every customer asks. The honest answer: only cutover is disruptive.

    Prepare    ->  No downtime
Apply      ->  No downtime
Finalize   ->  No downtime
Cutover    ->  DOWNTIME  (services stop and restart on the patched FS)
Cleanup    ->  No downtime

    All the slow work — applying patches, compiling, copying files — happens on the idle patch file system while users keep working. The only outage is the cutover window.

    What actually happens during cutover

    During cutover:

1. Application services stop.
2. Database patch edition becomes the new run edition.
3. Patch file system becomes the new run file system.
4. Context files are updated.
5. Application services restart.
6. Users reconnect to the newly patched environment.

    That sequence is the bridge between the architecture and what you actually watch happen in the logs during your maintenance window.

    Monitoring an ADOP session

    Check where any cycle stands with the status command and the AD tables:

    adop -status
    COL node_name   FORMAT A14
COL prepare     FORMAT A8
COL apply       FORMAT A8
COL cutover     FORMAT A8
COL cleanup     FORMAT A8
COL status      FORMAT A12

SELECT adop_session_id,
       node_name,
       NVL(prepare_status,'-')  AS prepare,
       NVL(apply_status,'-')    AS apply,
       NVL(cutover_status,'-')  AS cutover,
       NVL(cleanup_status,'-')  AS cleanup,
       NVL(status,'-')          AS status
FROM   ad_adop_sessions
ORDER  BY adop_session_id DESC;

    A clean idle system shows the last session fully completed. A row stuck mid-phase is an abandoned cycle that must be sorted before the next patch — you cannot start a fresh prepare cleanly on top of a half-finished one.

    fs_clone: the part that bites people

    fs_clone resyncs the patch file system from run so the two start identical:

    adop phase=fs_clone

    You run it explicitly when the file systems have drifted — after a failed cutover, a manual change to one side, or a long gap since the last cycle. prepare also triggers it internally when it detects drift.

    It rarely fails because of the clone logic. It fails on file ownership and permissions. fs_clone deletes and rebuilds the patch file system’s middleware home (FMW_Home), and if something there is not owned by the apps OS user, the delete stalls. The classic case is OHS files that ended up root-owned after a privileged process touched them. The clone tries to remove FMW_Home, hits files it cannot delete, and aborts.

    Safe recovery is to move the blocking tree aside rather than force-delete it:

    # As the apps owner, after confirming what is in there:
mv $PATCH_BASE/FMW_Home $PATCH_BASE/FMW_Home_stale_$(date +%Y%m%d)
# then re-run
adop phase=fs_clone

    mv over rm -rf is deliberate. A recursive force-delete on a partially root-owned tree on a production node is how a recoverable patch problem becomes a restore-from-backup problem. Rename it aside, let fs_clone rebuild cleanly, and clear the stale copy later once the system is confirmed healthy. If root-owned files genuinely must be removed, get the ownership corrected through the proper channel — not a blind sudo rm on a live box.

    Plan for space

    Remember that R12.2 maintains two complete application file systems plus fs_ne. Any major patching activity, an fs_clone, or a technology stack upgrade can temporarily require significantly more space than a comparable R12.1 environment. fs_clone failing with cryptic errors is often nothing more than a full mount — check headroom before you start, not after it fails halfway. This is a lesson many administrators learn the hard way.

    ADOP command quick reference

    adop -status                          # where does the current/last cycle stand

adop phase=prepare                    # sync patch FS, create patch edition
adop phase=apply patches=12345678     # apply patch to patch FS / patch edition
adop phase=finalize                   # pre-cutover compile work
adop phase=cutover                    # the downtime step — switch run/patch
adop phase=cleanup                    # drop obsolete editions and objects

adop phase=fs_clone                   # resync patch FS from run after drift
adop phase=abort                      # cleanly abandon an in-progress cycle

    Common real-world mistakes

    1. Assuming fs1 is always RUN.
2. Applying custom changes directly on the RUN file system.
3. Forgetting fs_clone after manual changes to one file system.
4. Running out of disk space during fs_clone or apply.
5. Leaving an ADOP session unfinished (blocks the next prepare).
6. Ignoring cleanup for months (editions pile up, performance degrades).
7. Modifying or letting processes touch FMW_Home as root.
8. Making custom changes on the RUN file system and forgetting to sync them
   to PATCH before the next ADOP cycle (lost custom JSPs, forms, and reports
   after cutover; unexplained differences between fs1 and fs2).

    Most production ADOP pain traces back to one of these eight, not to the tool itself.

    In summary

    Oracle EBS R12.2 achieves near-zero downtime patching by combining two application file systems (fs1/fs2) with Edition-Based Redefinition in the database. While users continue working on the run edition, patches are applied to an isolated patch edition. During cutover, Oracle switches application services from the old run file system and database edition to the newly patched run file system and edition.

    A few principles to carry away:

    • Run/patch roles are not fixed — read the environment, never assume fs1 is live.
    • fs_ne is non-editioned so logs and interface files survive cutover instead of vanishing.
    • Only cutover causes an outage. Everything else runs during business hours.
    • Most fs_clone failures are ownership problems on FMW_Home. Move the blocking tree aside and let it rebuild; reach for rm -rf only as an absolute last resort and never against a partially root-owned tree.
    • Treat disk space, edition awareness, and cleanup as routine discipline. The architecture is forgiving when you respect those and unforgiving when you don’t.

    Understanding ADOP becomes much easier when you stop thinking about patches and start thinking about roles: one file system serves users, the other is prepared for the next cutover.

    Disclaimer: All paths, identifiers, and examples in this post are anonymized and generic. They do not represent any specific client, employer, or environment. Views are my own.

  • When ADOP Remembers Too Much: Fixing Patch Failures Caused by Stale Metadata in Oracle EBS

    During an Oracle E-Business Suite ADOP patching cycle in a multi-node environment, the apply phase failed on one node while completing successfully on others. Despite retries — including downtime mode — the issue persisted, pointing to a deeper inconsistency within the patching framework.

    Symptoms Observed

    • ADOP session status: FAILED
    • Patch applied successfully on some nodes, failed on admin node
    • Repeated failures even with restart=no, abandon=yes, and downtime mode
    • No immediate actionable error from standard logs

    Timeline of Events

    T0 -- Patch execution initiated (ADOP apply phase)
T1 -- Failure observed on admin node
T2 -- Retry using downtime mode -- Failure persists
T3 -- ADOP session review shows inconsistent state
T4 -- Internal metadata tables analyzed
T5 -- Cleanup performed (tables + restart directory)
T6 -- Patch re-executed -- Success across all nodes

    Investigation

    Step 1: Check ADOP Session State

    Query the ADOP session status to understand the current state across all nodes:

    -- Check current ADOP session status
SELECT session_id, node_name, phase, status,
       start_date, end_date
FROM applsys.ad_adop_sessions
ORDER BY start_date DESC;

-- Check apply phase status per node
SELECT s.session_id, n.node_name, p.phase_code,
       p.status, p.start_date, p.end_date
FROM applsys.ad_adop_sessions s,
     applsys.ad_adop_session_phases p,
     applsys.fnd_nodes n
WHERE s.session_id = p.session_id
AND p.node_id = n.node_id
ORDER BY p.start_date DESC;

    The existing session showed status FAILED with the apply phase partially completed — a clear indicator of inconsistent execution state across nodes.

    Step 2: Check adalldefaults.txt

    Reviewed the defaults file for any relevant configuration:

    cat $APPL_TOP/admin/$TWO_TASK/adalldefaults.txt | grep -i missing
-- Key parameter found:
-- MISSING_TRANSLATED_VERSION = No

    Modifying and retrying with this parameter had no impact, confirming the issue was not translation-related.

    Step 3: Check Install Processes Table

    -- Check for stale entries in FND_INSTALL_PROCESSES
SELECT COUNT(*) FROM applsys.fnd_install_processes;

-- View stale entries in detail
SELECT process_status, process_name, last_update_date
FROM applsys.fnd_install_processes
ORDER BY last_update_date DESC;

-- Check AD_DEFERRED_JOBS
SELECT COUNT(*) FROM applsys.ad_deferred_jobs;
SELECT * FROM applsys.ad_deferred_jobs;

    Observation: FND_INSTALL_PROCESSES contained stale entries from the failed session. AD_DEFERRED_JOBS was empty.

    Root Cause

    The failure was caused by stale and inconsistent ADOP metadata tables — specifically APPLSYS.FND_INSTALL_PROCESSES and APPLSYS.AD_DEFERRED_JOBS. ADOP internally relies on these tables to track patch progress checkpoints, deferred job execution, and restart state management. When these tables retain entries from failed or incomplete sessions, ADOP assumes an incorrect execution state, leading to patch reconciliation failure, apply phase breakdown, and node-level inconsistencies.

    Resolution Steps

    Step 1: Backup Critical Tables

    -- Always backup before any cleanup
CREATE TABLE applsys.fnd_install_processes_bak AS
SELECT * FROM applsys.fnd_install_processes;

CREATE TABLE applsys.ad_deferred_jobs_bak AS
SELECT * FROM applsys.ad_deferred_jobs;

-- Verify backups
SELECT COUNT(*) FROM applsys.fnd_install_processes_bak;
SELECT COUNT(*) FROM applsys.ad_deferred_jobs_bak;

    Step 2: Drop Stale Metadata Tables

    Dropping these tables forces ADOP to rebuild clean metadata during the next run:

    DROP TABLE applsys.fnd_install_processes;
DROP TABLE applsys.ad_deferred_jobs;

    Step 3: Reset the Restart Directory

    The restart directory can silently preserve failure states. Back it up and create a fresh one:

    cd $APPL_TOP/admin/$TWO_TASK

-- Backup existing restart directory
mv restart restart_bkp_$(date +%Y%m%d)

-- Create fresh restart directory
mkdir restart

-- Verify
ls -la | grep restart

    Step 4: Re-run the Patch

    adop phase=apply \
     patches=<patch_id> \
     restart=no \
     abandon=yes \
     apply_mode=downtime

    The patch completed successfully across all nodes after the metadata cleanup.

    Before vs After

    ComponentBefore FixAfter Fix
    ADOP SessionFailedSuccessful
    Node ConsistencyPartialFull
    Restart BehaviorStuckClean
    Patch ExecutionIncompleteCompleted

    Key Takeaways

    • ADOP is state-driven — even when logs appear clean, internal metadata drives execution decisions
    • Partial success is a clue — if some nodes succeed and one fails, focus on local metadata, not the patch itself
    • The restart directory matters — it can silently preserve failure states and must be validated before retrying
    • Downtime mode is not a fix-all — even in downtime, ADOP still reads metadata tables; corruption persists unless cleaned
    • Always backup before cleanup — never drop tables without creating a backup first

    When NOT to Use This Approach

    Avoid applying this fix if the issue is caused by missing database patches (ETCC warnings), file system or permission issues, incorrect patch sequencing, or environment misconfiguration. Always validate the root cause before performing any metadata cleanup.

    This scenario highlights a subtle but critical behavior in ADOP — sometimes patch failures are not caused by the patch itself, but by what the system remembers about past attempts. By resetting stale metadata, we allow ADOP to re-evaluate the environment cleanly, leading to successful execution.

    Have questions or faced a similar issue? Reach out at sdanwarahmed@gmail.com.