Alhamdulillah.
Introduction
During a routine Oracle E-Business Suite R12.2 maintenance cycle, adop phase=fs_clone repeatedly failed while rebuilding the PATCH file system (fs2). The initial error message pointed to an edition issue, but the actual failure was buried deeper in the WebLogic configuration logs: the WebLogic Node Manager password could not be decrypted during fs2 configuration.
After correcting the WebLogic credential configuration and re-running AutoConfig and Preclone on the RUN file system, fs_clone force=yes completed successfully.
Context: Oracle EBS R12.2 uses WebLogic Server in the application tier. File system cloning involves WebLogic configuration and credentials when configuring the PATCH file system from the RUN file system.
Environment
Oracle E-Business Suite: R12.2
Database: Oracle 19c
Application tier: Multiple nodes
RUN file system: fs1
PATCH file system: fs2
Clone method: Full file system synchronization
RUN File System: /u01/oracle//fs1
PATCH File System: /u01/oracle//fs2
The Failure
The adop phase=fs_clone operation reported:
ERROR: Unable to set edition. Invalid edition type : PATCH
Followed by:
adclone.pl did not go through successfully.
EXIT STATUS: 1
These messages indicated failure but not the cause. The detailed WebLogic configuration logs revealed the actual exception.
Actual Failure: WebLogic Password Decryption
While WLST configured Node Manager credentials on fs2, the logs showed:
File “…/txkSetAppsLibsTarget.py”, line 705, in setNodeManagerCredentials
java.lang.AssertionError:
java.lang.reflect.InvocationTargetException
The stack trace continued through:
SecurityConfigurationMBeanImpl.getNodeManagerPassword
EncryptionServiceException:
com.rsa.jsafe.JSAFE_PaddingException: Invalid padding
The failure chain:
txkSetAppsLibsTarget.py
-> setNodeManagerCredentials()
-> getNodeManagerPassword()
-> WebLogic decrypt()
-> JSAFE_PaddingException: Invalid padding
WebLogic was unable to decrypt the Node Manager password during fs2 configuration. This pointed to a WebLogic credential/encryption configuration issue rather than a filesystem capacity or OS resource issue.
Investigation
Before pinpointing the WebLogic credential issue, these areas were examined:
Filesystem Space
The application filesystem had adequate free space. The FMW Home cloning process had progressed substantially before failing, making a simple disk-space issue unlikely.
Port Allocation
The clone successfully acquired all required temporary ports. No port conflicts were identified.
Operating System
OS logs showed no evidence of: OOM kills, Java process termination, segmentation faults, core dumps, or hs_err_pid files.
FMW Home Cloning
The pasteBinary operation progressed to approximately 90% before stopping. However, the actionable error was the WebLogic decryption failure:
getNodeManagerPassword
JSAFE_PaddingException: Invalid padding
The 90% progress point indicated where the clone stopped, not the root cause.
Resolution
After reviewing the issue with a senior DBA, the WebLogic credential configuration was corrected and the RUN file system configuration was refreshed before retrying the clone.
The following steps resolved the issue:
1. Update boot.properties
Update the WebLogic boot.properties file with the appropriate credentials as part of correcting the WebLogic credential configuration.
2. Restart WebLogic
Restart WebLogic to load the updated credential configuration.
3. Run AutoConfig on the RUN File System
cd $COMMON_TOP/admin/scripts
./adautocfg.sh
This regenerates EBS configuration on the RUN file system.
4. Run Preclone on the RUN File System
cd $ADMIN_SCRIPTS_HOME
perl adpreclone.pl appsTier
Preclone refreshes the RUN file system configuration used for the subsequent clone operation.
5. Rerun fs_clone in Force Mode
adop phase=fs_clone force=yes
The operation completed successfully.
Result
The PATCH file system was successfully rebuilt after the WebLogic credential configuration was corrected.
adop phase=fs_clone force=yes -> SUCCESS
Troubleshooting Takeaways
Don’t Stop at Generic Errors
Messages such as “adclone.pl did not go through successfully” or “Unable to set edition. Invalid edition type : PATCH” only indicate that the clone failed. Always examine the detailed FSCloneApplyAppsTier and WLST logs to identify the actual component failure.
WebLogic Stack Traces Matter
If you encounter this pattern:
setNodeManagerCredentials
getNodeManagerPassword
JSAFE_PaddingException: Invalid padding
Investigate the WebLogic credential and encryption configuration. Do not assume filesystem space, port allocation, or OS resource exhaustion is responsible.
Look Beyond the Final FAILED Message
The most useful error is often several layers below the final adop failure. In this case:
SecurityConfigurationMBeanImpl.getNodeManagerPassword
com.rsa.jsafe.JSAFE_PaddingException: Invalid padding
This identified the WebLogic password decryption step as the failure point.
The PATCH Edition Message Was Not the Root Cause
The message “Unable to set edition. Invalid edition type : PATCH” appeared during the investigation but was not the cause of the fs_clone failure. The actionable failure was the WebLogic password decryption error.
Summary
The adop phase=fs_clone failure occurred while configuring WebLogic Node Manager credentials during PATCH file system construction. The failure occurred because WebLogic was unable to decrypt the Node Manager password during the Node Manager credential configuration step:
java.lang.AssertionError:
java.lang.reflect.InvocationTargetException
SecurityConfigurationMBeanImpl.getNodeManagerPassword
com.rsa.jsafe.JSAFE_PaddingException: Invalid padding
The resolution involved correcting the WebLogic boot.properties configuration, restarting WebLogic, and re-running AutoConfig and Preclone on the RUN file system. A subsequent fs_clone force=yes completed successfully.
Note: This resolution is based on remediation in a specific environment. JSAFE_PaddingException errors during EBS cloning can have different underlying causes. Always review the detailed WebLogic and WLST logs before applying this remediation to another environment.
Disclaimer: All environment identifiers, hostnames, IP addresses, and server paths in this article have been masked with generic placeholders to protect production system information and organizational confidentiality. The technical troubleshooting methodology and error signatures remain directly applicable to similar incidents in your environment.
Leave a comment