Alhamdulillah, this post documents the issues hit during an EBS R12.2 QA application-tier refresh and how each was isolated. Server names, paths and identifiers are generic examples.
Environment (generic)
| Component | Example |
|---|---|
| App server | qa-app-server01 |
| DB server | qa-db-server01 |
| EBS environment | QA_EBS |
| Application base | /u01/apps |
| Pairs file | /u01/cloning/pairs/qa_apps_pairs.txt |
| Central inventory | /opt/oraInventory |
Background
A DualFS clone recreates both fs1 and fs2. Per the cloning procedure used here, the old fs2, inst and FMW_Home were moved aside and removed first:
cd /u01/apps/
mv fs2 fs2_old
rm -rf fs2_old &
cd fs1
mv inst inst_old
mv FMW_Home FMW_Home_old
rm -rf inst_old FMW_Home_old &Cleanup steps should follow your own environment’s cloning document.
Symptom
adcfgclone.pl failed at the FMW prerequisite stage:
ERROR: Failed to execute FMW pre-requisite checkRe-running would not have helped. The prerequisite check reads OS state, so that is where the investigation started.
cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'Issue 1: RPM database corruption
On one app server, rpm --verifydb returned Berkeley DB errors:
BDB0682 Page ... bad prev_pgno
BDB0683 Page ... overflow item incomplete
Packages: BDB0090 DB_VERIFY_BADPackage queries also showed rpmdbNextIterator: skipping h# ... and region trailer: BAD.
RPMDB corruption was identified and the package state was subsequently validated. If the RPM database is unhealthy, package-based prerequisite results can be incomplete or misleading. Running rpm --verifydb early is a useful way to identify this condition before spending time troubleshooting the Oracle prerequisite scripts.
Issue 2: Missing compatibility packages
Two packages were missing: compat-libcap1 and compat-libstdc++-33. After confirming repository availability, both were installed with dnf install, then the RPM database was verified again.
A similar QA server passed the check without this problem, so package state can differ between servers that look identical.
Issue 3: refhost.xml OL7 block on an OL8 target
The target was Oracle Linux 8, but the first OS block in refhost.xml was defined for Oracle Linux 7:
/u01/apps/fs1/EBSapps/comn/clone/prereq/webtier/Scripts/prereq/linux64/refhost.xml<VERSION VALUE="7"/>Only that block was changed to VALUE="8". A global sed replacement is risky because the file contains multiple OS definitions. Back up the file, identify the applicable block, change only that, and validate the XML.
Issue 4: Stale Oracle Central Inventory
Removing FMW_Home from disk does not remove it from the central inventory. Entries such as these were still registered after the directories were gone:
/u01/apps/fs1/FMW_Home/oracle_common
/u01/apps/fs1/FMW_Home/webtier
/u01/apps/fs1/FMW_Home/Oracle_EBS-app1Documented cloning step. The cloning document’s inventory cleanup applies to the target database server (check /etc/oraInst.loc, remove the DB home entry from ContentsXML/inventory.xml). It does not cover resetting the application-server inventory, so the app-server reset described here is not part of the standard cloning steps.
Before any inventory change: confirm the active inventory via /etc/oraInst.loc, preserve a copy, then apply the approved procedure. The existing inventory was kept as a backup. Note that the first mv below is conditional: run it only if /opt/oraInventory_old already exists.
# Only if /opt/oraInventory_old already exists:
mv /opt/oraInventory_old /opt/oraInventory_old_backup
mv /opt/oraInventory /opt/oraInventory_old
mkdir /opt/oraInventory
chown oracle:oinstall /opt/oraInventoryIssue 5: EXECUTE_FMW_SYS_PREREQS
The cloning procedure sets this variable before the DualFS run:
export EXECUTE_FMW_SYS_PREREQS=false
echo $EXECUTE_FMW_SYS_PREREQSThis skips the FMW OS prerequisite execution. It was set here only after the failures above had been investigated, and only because the procedure specifies it. It is not a general substitute for fixing prerequisite failures.
Issue 6: SSH disconnect during a multi-hour clone
RapidClone can run for hours, and the procedure calls for a screen session (or VDI). During this exercise the SSH session dropped mid-run.
screen -S rapidclone
cd /u01/apps/fs1/EBSapps/comn/clone/bin
export EXECUTE_FMW_SYS_PREREQS=false
perl adcfgclone.pl component=appsTier \
pairsfile=/u01/cloning/pairs/qa_apps_pairs.txt \
dualfs=yesAfter the disconnect, check before restarting anything:
screen -ls
screen -r <session>If the original process is still running, a second adcfgclone.pl would only add complications.
Result
After the identified OS, package, prerequisite and inventory issues were addressed, the DualFS clone completed and created both fs1 and fs2.
ما شاء الله
The same approach worked on the second QA environment. This establishes that the DualFS RapidClone completed, not that the entire QA refresh is finished: post-clone configuration is separate, and the procedure says not to start services at the end of the clone until the required post-clone steps are completed.
Summary
| Finding | Layer | Action |
|---|---|---|
| FMW prerequisite failure | OS / packages | Investigated underlying checks |
| RPMDB corruption | OS | Identified the corruption and validated the package state after remediation |
| Missing compat RPMs | OS | Installed required packages |
| OL7 block on OL8 target | Prerequisite config | Corrected the applicable block only |
| Stale FMW homes in inventory | Oracle Inventory | Backed up the existing inventory, then applied the approved remediation |
| Long-running clone | Session | Ran inside screen |
Suggested pre-clone checklist
cat /etc/os-release
rpm --verifydb
rpm -qa | grep -E 'compat-libcap1|compat-libstdc'
cat /etc/oraInst.loc
ls -ld /u01/apps/fs1 /u01/apps/fs2
ls -l /u01/apps/fs1/EBSapps/comn/clone/bin/adcfgclone.plPost-clone
RapidClone success is not the end of the refresh. The cloning procedure continues with custom environment configuration, AutoConfig, WebLogic configuration, Concurrent Manager/MWA node configuration, shared filesystem setup and further AutoConfig runs, followed by application validation.
Takeaway
RapidClone is often where infrastructure problems become visible, not where they start. When it fails, check the layers underneath before rerunning:
OS → RPM database → packages → prerequisite config → Oracle Inventory → filesystem → RapidClone → post-clone configuration
Examples use generic names. No real server names, IP addresses, domains, company names or screenshots are included.
