Tag: DualFS

Alhamdulillah, this post documents the issues hit during an EBS R12.2 QA application-tier refresh and how each was isolated. Server names, paths and identifiers are generic examples.

Environment (generic)

Component	Example
App server	qa-app-server01
DB server	qa-db-server01
EBS environment	QA_EBS
Application base	/u01/apps
Pairs file	/u01/cloning/pairs/qa_apps_pairs.txt
Central inventory	/opt/oraInventory

Background

A DualFS clone recreates both fs1 and fs2. Per the cloning procedure used here, the old fs2, inst and FMW_Home were moved aside and removed first:

cd /u01/apps/
mv fs2 fs2_old
rm -rf fs2_old &

cd fs1
mv inst inst_old
mv FMW_Home FMW_Home_old
rm -rf inst_old FMW_Home_old &

Cleanup steps should follow your own environment’s cloning document.

Symptom

adcfgclone.pl failed at the FMW prerequisite stage:

ERROR: Failed to execute FMW pre-requisite check

Re-running would not have helped. The prerequisite check reads OS state, so that is where the investigation started.

cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'

Issue 1: RPM database corruption

On one app server, rpm --verifydb returned Berkeley DB errors:

BDB0682 Page ... bad prev_pgno
BDB0683 Page ... overflow item incomplete
Packages: BDB0090 DB_VERIFY_BAD

Package queries also showed rpmdbNextIterator: skipping h# ... and region trailer: BAD.

RPMDB corruption was identified and the package state was subsequently validated. If the RPM database is unhealthy, package-based prerequisite results can be incomplete or misleading. Running rpm --verifydb early is a useful way to identify this condition before spending time troubleshooting the Oracle prerequisite scripts.

Issue 2: Missing compatibility packages

Two packages were missing: compat-libcap1 and compat-libstdc++-33. After confirming repository availability, both were installed with dnf install, then the RPM database was verified again.

A similar QA server passed the check without this problem, so package state can differ between servers that look identical.

Issue 3: refhost.xml OL7 block on an OL8 target

The target was Oracle Linux 8, but the first OS block in refhost.xml was defined for Oracle Linux 7:

/u01/apps/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/prereq/linux64/refhost.xml

<VERSION VALUE="7"/>

Only that block was changed to VALUE="8". A global sed replacement is risky because the file contains multiple OS definitions. Back up the file, identify the applicable block, change only that, and validate the XML.

Issue 4: Stale Oracle Central Inventory

Removing FMW_Home from disk does not remove it from the central inventory. Entries such as these were still registered after the directories were gone:

/u01/apps/fs1/FMW_Home/oracle_common
/u01/apps/fs1/FMW_Home/webtier
/u01/apps/fs1/FMW_Home/Oracle_EBS-app1

Documented cloning step. The cloning document’s inventory cleanup applies to the target database server (check /etc/oraInst.loc, remove the DB home entry from ContentsXML/inventory.xml). It does not cover resetting the application-server inventory, so the app-server reset described here is not part of the standard cloning steps.

Before any inventory change: confirm the active inventory via /etc/oraInst.loc, preserve a copy, then apply the approved procedure. The existing inventory was kept as a backup. Note that the first mv below is conditional: run it only if /opt/oraInventory_old already exists.

# Only if /opt/oraInventory_old already exists:
mv /opt/oraInventory_old /opt/oraInventory_old_backup

mv /opt/oraInventory /opt/oraInventory_old
mkdir /opt/oraInventory
chown oracle:oinstall /opt/oraInventory

Issue 5: EXECUTE_FMW_SYS_PREREQS

The cloning procedure sets this variable before the DualFS run:

export EXECUTE_FMW_SYS_PREREQS=false
echo $EXECUTE_FMW_SYS_PREREQS

This skips the FMW OS prerequisite execution. It was set here only after the failures above had been investigated, and only because the procedure specifies it. It is not a general substitute for fixing prerequisite failures.

Issue 6: SSH disconnect during a multi-hour clone

RapidClone can run for hours, and the procedure calls for a screen session (or VDI). During this exercise the SSH session dropped mid-run.

screen -S rapidclone
cd /u01/apps/fs1/EBSapps/comn/clone/bin
export EXECUTE_FMW_SYS_PREREQS=false
perl adcfgclone.pl component=appsTier \
pairsfile=/u01/cloning/pairs/qa_apps_pairs.txt \
dualfs=yes

After the disconnect, check before restarting anything:

screen -ls
screen -r <session>

If the original process is still running, a second adcfgclone.pl would only add complications.

Result

After the identified OS, package, prerequisite and inventory issues were addressed, the DualFS clone completed and created both fs1 and fs2.

ما شاء الله

The same approach worked on the second QA environment. This establishes that the DualFS RapidClone completed, not that the entire QA refresh is finished: post-clone configuration is separate, and the procedure says not to start services at the end of the clone until the required post-clone steps are completed.

Summary

Finding	Layer	Action
FMW prerequisite failure	OS / packages	Investigated underlying checks
RPMDB corruption	OS	Identified the corruption and validated the package state after remediation
Missing compat RPMs	OS	Installed required packages
OL7 block on OL8 target	Prerequisite config	Corrected the applicable block only
Stale FMW homes in inventory	Oracle Inventory	Backed up the existing inventory, then applied the approved remediation
Long-running clone	Session	Ran inside screen

Suggested pre-clone checklist

cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'
cat /etc/oraInst.loc
ls -ld /u01/apps/fs1 /u01/apps/fs2
ls -l /u01/apps/fs1/EBSapps/comn/clone/bin/adcfgclone.pl

Post-clone

RapidClone success is not the end of the refresh. The cloning procedure continues with custom environment configuration, AutoConfig, WebLogic configuration, Concurrent Manager/MWA node configuration, shared filesystem setup and further AutoConfig runs, followed by application validation.

Takeaway

RapidClone is often where infrastructure problems become visible, not where they start. When it fails, check the layers underneath before rerunning:

OS → RPM database → packages → prerequisite config → Oracle Inventory → filesystem → RapidClone → post-clone configuration

Examples use generic names. No real server names, IP addresses, domains, company names or screenshots are included.