Oracle EBS R12.2 ADOP: MWA Template Conflict (AC-00058), a Hung ICM at Cutover, and a Node That Wasn’t There

Alhamdulillah. This post covers one online patching cycle on an Oracle EBS 12.2 DEV environment (AD/TXK C.Delta.17, two application nodes on a shared file system). Applying a single patch turned into three separate problems:

1. The apply phase failed because AutoConfig aborted on MWA custom templates.
2. Someone asked whether a third application node had been missed by the patch.
3. During cutover, the Internal Concurrent Manager (ICM) did not respond to shutdown.

Each is short on its own. Together they show why it pays to check evidence before you act on a quick instruction. Hostnames, paths and custom package names below are anonymized.

Problem 1: apply fails on AutoConfig

The apply phase completed on the admin node but failed on the second node (the concurrent manager node) at this step:

EVENT        Running AutoConfig on patch edition file system.
UNEXPECTED   Error occurred running "sh .../ad/12.0.0/bin/adconfig.sh contextfile=.../fs2/inst/apps/<CONTEXT>/appl/admin/<CONTEXT>.xml -parallel ..."
UNEXPECTED   Error occurred while executing AutoConfig on current node.
UNEXPECTED   Apply phase has failed.

The admin node then reported its own error from txkADOPEvalSrvStatus.pl. That error is only a consequence: adop could not continue because one node had failed. The real error was in the AutoConfig log of the failed node:

[ VERSION CONFLICTS INFORMATION ]
Template shipped by Oracle : .../fs2/EBSapps/appl/mwa/12.0.0/admin/template/mwa.cfg(version: 120.2.12020000.12)
Custom template            : .../fs2/EBSapps/appl/mwa/12.0.0/admin/template/custom/mwa.cfg(version: 120.2.12020000.8)
[ ERROR_REPORT ]
AC-00058: Error: Unable to determine the version information for file - .../mwa/12.0.0/admin/template/custom/mwactl.sh; hence nothing can be said about its version conflicts.
Version Conflicts among development maintained and customized templates encountered; aborting AutoConfig run.

Two different problems in the same directory:

• mwa.cfg: a normal version conflict. The patch delivered version .12 of the template, and the custom copy was still at .8.
• mwactl.sh: AC-00058, which is not a version conflict. AutoConfig could not read a version from the custom file at all.

Why AC-00058 happened

The $Header line in the custom mwactl.sh named a different file:

# $Header: mwaexpctl.sh 120.10.12020000.2 2017/03/14 06:40:25 sowu ship $

At some point a copy of a related control script had been placed in custom/ under the name mwactl.sh. The header still carried the old file name, so AutoConfig could not match it to the shipped mwactl.sh. Correcting the file name in the header resolved the error. The shipped mwactl.sh was still at 120.10.12020000.2, so after the correction the versions matched.

Check what the custom file actually does before replacing it

The fastest fix would have been to replace both custom files with the shipped templates. A diff against the shipped mwactl.sh showed why that would have been a mistake for this file:

cd $MWA_TOP/admin/template
diff mwactl.sh custom/mwactl.sh

< JAVA="%s_afjvaprg%  -Doracle.apps.mwa=${MWA_TOP} ..."
---
> JAVA="${ADMIN_SCRIPTS_HOME}/java.sh  -Doracle.apps.mwa=${MWA_TOP} ... -javaagent:${AF_JLIB}/aspectjweaver.jar"

< CLASSPATH="${AF_CLASSPATH}"
---
> CLASSPATH="${AF_JLIB}/mwaexp.jar:${AF_CLASSPATH}"

<           ${JAVA} oracle.apps.mwa.presentation.telnet.Listener "$@" &
---
>           ${JAVA} XX.oracle.apps.mwa.presentation.telnet.Listener2 "$@" &

The custom file starts a custom MWA telnet listener, with its own jar and a Java agent. If it had been replaced, AutoConfig would have generated the standard mwactl.sh and the custom listener would no longer have started. Nothing in the patch output would have flagged this. So mwactl.sh was kept, and only its header was corrected.

mwa.cfg was different. Its only differences from the shipped file were two settings:

mwa.OutOfOrderScanEnabled=Yes
mwa.GlobalBarcodeDelimiter=64

The usual approach is to rebase the custom copy on the new shipped version and re-apply the real customizations. In this case the team lead decided to replace the custom file with the shipped one and not carry the two settings forward. That is a functional decision, so it was recorded with the backup file name. If scanner users report problems with out-of-order scans or barcode delimiters, those two lines are the first place to look.

The fix

The file system was shared between both application nodes, so the change was made once. Before trusting that, confirm it: identical df figures on two nodes suggest a shared disk, but a test file created on one node and listed on the other proves it.

cd $MWA_TOP/admin/template          # patch edition (fs2) path
TS=$(date +%Y%m%d_%H%M%S)
cp -p custom/mwa.cfg custom/mwa.cfg.pre_replace_$TS
cksum custom/mwa.cfg custom/mwa.cfg.pre_replace_$TS

cp -p mwa.cfg custom/mwa.cfg

grep -n '\$Header' mwa.cfg custom/mwa.cfg custom/mwactl.sh
diff mwa.cfg custom/mwa.cfg && echo "IDENTICAL"

mwa.cfg:23:# $Header: mwa.cfg 120.2.12020000.12 2026/09/09 12:32:24 ukoneti ship $
custom/mwa.cfg:23:# $Header: mwa.cfg 120.2.12020000.12 2026/09/09 12:32:24 ukoneti ship $
custom/mwactl.sh:14:# $Header: mwactl.sh 120.10.12020000.2 2017/03/14 06:40:25 sowu ship $
IDENTICAL

Before restarting, I checked the whole patch file system for other custom files without a header:

find $PATCH_BASE/EBSapps/appl -path "*admin/template/custom/*" -type f \
  ! -name "*.bkp_*" ! -name "*.pre_replace_*" \
  -exec sh -c 'grep -q "\$Header" "$1" || echo "NO HEADER: $1"' _ {} \;

It found four files in the MWA custom/ directory: a text file, a log, an .ini file and a file named devnull. None of them has a matching template in the parent directory, and the original AutoConfig error had not listed any of them. AutoConfig only compares custom files that have a shipped counterpart, so these were left in place. The .ini file looked like part of the same custom extension.

Restart the apply

adop phase=apply patches=<patch_number> restart=yes

The restart picked up the existing session, skipped the node that had already finished, and ran AutoConfig on the patch file system of the failed node. It completed on both nodes in about six and a half minutes. The status lines Session status: FAILED and ATLEAST ONE PATCH IS ALREADY APPLIED at the start of a restart come from the previous run and are expected.

After the apply, I checked that the generated script in the patch edition still started the custom listener:

grep -E "XX|mwaexp.jar" $PATCH_BASE/inst/apps/<CONTEXT>/admin/scripts/mwactl.sh

CLASSPATH="${AF_JLIB}/mwaexp.jar:${AF_CLASSPATH}"
                if ${JAVA} XX.oracle.apps.mwa.presentation.telnet.GenerateLauncher start
            ${JAVA} XX.oracle.apps.mwa.presentation.telnet.Listener2 "$@" &

One note on adop -status: after a successful apply, the APPLY phase shows ACTIVE, not COMPLETED. The apply phase stays open so more patches can be added to the cycle. It changes to COMPLETED at cutover.

Problem 2: the node that wasn’t there

Before finalize, a question came in: had the patch been applied on a third application node, an external-facing iSupplier server? The suggested fix was to copy the patch there and run:

adop phase=apply patches=<patch_list> allnodes=no action=nodb

That command is the right recovery for a registered node that failed or was unavailable during a cycle. But the adop output had only ever mentioned two nodes, so I checked what the database knew first:

SET LINES 200 PAGES 100
COL node_name FORMAT A25
SELECT NVL(node_name,'-') node_name FROM adop_valid_nodes ORDER BY 1;

COL support_web   FORMAT A4
COL support_cp    FORMAT A4
COL support_forms FORMAT A5
COL status        FORMAT A6
SELECT NVL(node_name,'-') node_name, NVL(support_web,'-') support_web,
       NVL(support_cp,'-') support_cp, NVL(support_forms,'-') support_forms,
       NVL(status,'-') status
FROM   fnd_nodes ORDER BY 1;

COL path FORMAT A90
SELECT NVL(node_name,'-') node_name, NVL(status,'-') status, NVL(path,'-') path
FROM   fnd_oam_context_files
WHERE  LOWER(node_name) LIKE '%<node_pattern>%';

adop_valid_nodes listed only the two nodes in the session. fnd_nodes listed those two, the database node and the standard non-server entries. fnd_oam_context_files returned no rows for the third server. Its patch directory also had nothing newer than a patch staged months earlier, and no adop processes were running on it.

So the server was not registered in this instance at all. adop had not skipped it. Running an apply there would have failed node validation, and the patch did not need to go there as part of this cycle. The cycle went ahead with the two registered nodes.

One small thing from this step: in this AD/TXK level, adop_valid_nodes has no node_type column. The columns are NODE_NAME, CONTEXT_NAME, ZD_EDITION_NAME and ZD_SYNC. Run DESC before writing queries against adop tables.

Problem 3: the ICM would not go down at cutover

Finalize completed in about eight minutes. The only running concurrent request before cutover was the Online Patching In Progress request (ADZDPATCH), which adop submits for every cycle and cancels itself during cutover. Do not cancel it manually.

Cutover was run with services left down afterwards:

adop phase=cutover mtrestart=no

It then stopped here:

EVENT        Submitting request for Internal Concurrent Manager shutdown.
EVENT        Cancelling ADZDPATCH concurrent request
EVENT        Waiting for Internal Concurrent Manager to go down.

An ICM shutdown normally waits for running requests to finish, so the first check was for running requests. There were none. The managers were still holding processes, though:

CONCURRENT_QUEUE_NAME          RUNNING_PROCESSES MAX_PROCESSES
------------------------------ ----------------- -------------
FNDCPOPP                                       5             5
FNDICM                                         1             0
FNDSM_<NODE1>                                  1             0
FNDSM_<NODE2>                                  1             0
IEXCONMGR                                      1             1
PASMGR                                         1             1
STANDARD                                       1            60

The ICM log on the concurrent manager node showed the process monitor running every 30 seconds. Each pass listed a request in state Q that never progressed:

Request  : <request_id>
Priority : 0
Program  : 0/5
State    : Q

                   Process monitor session started : 04-NOV-2026 05:01:40
                     Process monitor session ended : 04-NOV-2026 05:01:40

A manual adcmctl.sh stop only submitted a second shutdown request, which queued behind the first. adcmctl.sh status still reported the ICM as active more than an hour after cutover started. The ICM database session was not blocked:

COL program  FORMAT A20
COL event    FORMAT A35
COL blocker  FORMAT A10
SELECT s.sid, RPAD(NVL(s.program,'-'),20) program, RPAD(NVL(s.event,'-'),35) event,
       NVL(TO_CHAR(s.blocking_session),'-') blocker, s.seconds_in_wait
FROM   v$session s, v$process p
WHERE  s.paddr = p.addr
AND    s.process = '<icm_os_pid>';

  SID PROGRAM              EVENT                               BLOCKER    SECONDS_IN_WAIT
----- -------------------- ----------------------------------- ---------- ---------------
 3511 ICM@<NODE2>          SQL*Net message from client         -                       16

So the ICM was alive and idle, not waiting on a lock, and not acting on its shutdown requests. I could not establish why from the available logs. With no running requests and the cutover blocked, the team lead approved killing the remaining concurrent manager processes on the concurrent manager node: the ICM, its startmgr parent shell, the Service Manager, the Output Post Processor JVMs and the remaining manager processes. Once they were gone, adop saw the ICM as down and the cutover carried on.

Two things to handle if you end up here:

• Leave adop alone. Kill the manager processes, never the adop or adopmon session.
• Clear the queued shutdown requests before the managers start again. If they are still pending, the new ICM can pick them up and shut itself down straight after startup.

SELECT request_id, phase_code, status_code
FROM   fnd_concurrent_requests
WHERE  request_id IN (<shutdown_request_1>, <shutdown_request_2>);

Option worth knowing: cm_wait. Cutover accepts a cm_wait parameter: the maximum number of minutes adop waits for the concurrent managers to shut down before it stops them forcefully. Without it, cutover waits as long as the managers take. For a cycle like this one, the command would be:

adop phase=cutover cm_wait=3 mtrestart=no

A forced shutdown also ends any request that is still running when the timer expires, so agree the value with the application team before the cutover window. I did not use cm_wait in this cycle, so I can’t say whether it would have cleared this particular hang.

What normal cutover output looks like after that

A few lines in the rest of the cutover look alarming but are expected:

• DB Cutover did not complete on Admin node ... Will wait for another minute and check again. on the second node. It polls every minute until the admin node finishes the database cutover.
• Entered into KILL SESSIONS LOOP inside RENAME_AW. adop clears sessions on the OLAP analytic workspace before renaming it, and the next lines show it finishing.
• ORA-00028: your session has been killed right after Switching to Patch Edition. adop kills old-edition sessions, including its own, and reconnects. The log itself says this is normal.

Cutover completed on both nodes in about 1 hour 22 minutes. Most of that was the wait on the ICM.

After cutover

With mtrestart=no, services stay down until you start them. The order I followed:

1. Re-source the run environment and confirm $RUN_BASE now points to the other file system.
2. Confirm no manager processes from the old run edition are left on either node.
3. Clear any pending shutdown requests.
4. Start services on the admin node, then the second node.
5. Check the ICM, the custom MWA listener process, front-end login and a test concurrent request.
6. Run adop phase=cleanup, then adop phase=fs_clone so the corrected custom templates reach the other file system. Until then, an AutoConfig run on the old file system would hit the same version conflict.

In this cycle the post-cutover steps were completed and the cleanup phase finished successfully.

Lessons

• Read the AutoConfig log, not the adop summary. The admin node error pointed at a status script. The real cause was two lines in the failed node’s adconfig.log.
• A version conflict and AC-00058 are different problems. One means the custom template is older. The other means AutoConfig cannot read a version at all, often because the $Header line is missing or names a different file.
• Diff before you replace. “Back up and replace with the shipped file” was correct for mwa.cfg and would have quietly broken the custom MWA listener if applied to mwactl.sh.
• Check node registration before patching a node. adop_valid_nodes, fnd_nodes and fnd_oam_context_files answer “was this node missed?” in under a minute.
• A hung ICM at cutover needs evidence first, then approval. Running requests, queue counts, the ICM log and the ICM session’s wait event together showed there was nothing to wait for. I could not find the cause, so I have not claimed one.

ما شاء الله, a single patch, three different lessons. If you have seen an ICM ignore its shutdown requests during cutover and found the cause, I would like to hear about it in the comments.