Background:
We have 2 nodes RAC running on Oracle Enterprise Linux
racdb1 and racdb2, with fresh installation of 19.3 GI and database
After replace the latest OPatch on GI Home and DB Home on both node,
We extracted the bundle patch p30463609 at /dbbackup/patch
And apply the GI patch at follow
Apply patch on node 1:
[root@racdb1 bin]# export PATH=$PATH:/u01/app/19.0.0.0/grid/OPatch
[root@racdb1 bin]# opatchauto apply /dbbackup/patch/30463609/30501910
OPatchauto session is initiated at Thu Feb 6 15:54:30 2020
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-02-06_03-54-32PM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-02-06_03-55-21PM.log
The id for this session is HTHU
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome1
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0.0/grid
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Patch applicability verified successfully on home /u01/app/19.0.0.0/grid
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0.0/dbhome1
Successfully prepared home /u01/app/oracle/product/19.0.0.0/dbhome1 to bring down database service
Bringing down CRS service on home /u01/app/19.0.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/racdb1/crsconfig/crspatch_racdb1_2019-12-12_03-08-10PM.log
CRS service brought down successfully on home /u01/app/19.0.0.0/grid
Performing prepatch operation on home /u01/app/oracle/product/19.0.0.0/dbhome1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Start applying binary patch on home /u01/app/oracle/product/19.0.0.0/dbhome1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Performing postpatch operation on home /u01/app/oracle/product/19.0.0.0/dbhome1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Start applying binary patch on home /u01/app/19.0.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0.0/grid
Starting CRS service on home /u01/app/19.0.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/racdb1/crsconfig/crspatch_racdb1_2019-12-12_03-08-10PM.log
CRS service started successfully on home /u01/app/19.0.0.0/grid
Preparing home /u01/app/oracle/product/19.0.0.0/dbhome1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0.0/dbhome1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:racdb1
RAC Home:/u01/app/oracle/product/19.0.0.0/dbhome1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /dbbackup/patch/30463609/30501910/30489632
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /dbbackup/patch/30463609/30501910/30655595
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /dbbackup/patch/30463609/30501910/30489227
Log: /u01/app/oracle/product/19.0.0.0/dbhome1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-00-30PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30557433
Log: /u01/app/oracle/product/19.0.0.0/dbhome1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-00-30PM_1.log
Host:racdb1
CRS Home:/u01/app/19.0.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SUCCESSFULLY applied:
Patch: /dbbackup/patch/30463609/30501910/30489227
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-03-05PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30489632
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-03-05PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30557433
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-03-05PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30655595
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_16-03-05PM_1.log
OPatchauto session completed at Thu Feb 6 16:13:42 2020
Time taken to complete the session 19 minutes, 12 seconds
[root@racdb1 bin]#Apply patch on Node 2 but fail
[root@racdb2 ~]# export PATH=$PATH:/u01/app/19.0.0.0/grid/OPatch
[root@racdb2 ~]# opatchauto apply /dbbackup/patch/30463609/30501910
OPatchauto session is initiated at Thu Feb 6 19:04:03 2020
System initialization log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2020-02-06_07-04-06PM.log.
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-02-06_07-04-33PM.log
The id for this session is Y2QQ
Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome1
Executing OPatch prereq operations to verify patch applicability on home /u01/app/19.0.0.0/grid
Patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Patch applicability verified successfully on home /u01/app/19.0.0.0/grid
Verifying SQL patch applicability on home /u01/app/oracle/product/19.0.0.0/dbhome1
SQL patch applicability verified successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Preparing to bring down database service on home /u01/app/oracle/product/19.0.0.0/dbhome1
Successfully prepared home /u01/app/oracle/product/19.0.0.0/dbhome1 to bring down database service
Bringing down CRS service on home /u01/app/19.0.0.0/grid
Prepatch operation log file location: /u01/app/grid/crsdata/racdb2/crsconfig/crspatch_racdb2_2019-12-12_03-57-47PM.log
CRS service brought down successfully on home /u01/app/19.0.0.0/grid
Performing prepatch operation on home /u01/app/oracle/product/19.0.0.0/dbhome1
Perpatch operation completed successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Start applying binary patch on home /u01/app/oracle/product/19.0.0.0/dbhome1
Binary patch applied successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Performing postpatch operation on home /u01/app/oracle/product/19.0.0.0/dbhome1
Postpatch operation completed successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
Start applying binary patch on home /u01/app/19.0.0.0/grid
Failed while applying binary patches on home /u01/app/19.0.0.0/grid
Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : racdb2->/u01/app/19.0.0.0/grid Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.0.0.0/grid, host: racdb2.
Command failed: /u01/app/19.0.0.0/grid/OPatch/opatchauto apply /dbbackup/patch/30463609/30501910 -oh /u01/app/19.0.0.0/grid -target_type cluster -binary -invPtrLoc /u01/app/19.0.0.0/grid/oraInst.loc -jre /u01/app/19.0.0.0/grid/OPatch/jre -persistresult /u01/app/19.0.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_racdb2_crs.ser -analyzedresult /u01/app/19.0.0.0/grid/opatchautocfg/db/sessioninfo/sessionresult_analyze_racdb2_crs.ser
Command failure output:
==Following patches FAILED in apply:
Patch: /dbbackup/patch/30463609/30501910/30489227
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-09-24PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: ApplySession failed in system modification phase... 'ApplySession::apply failed: java.io.IOException: oracle.sysman.oui.patch.PatchException: java.io.FileNotFoundException: /u01/app/oraInventory/ContentsXML/oui-patch.xml (Permission denied)'
After fixing the cause of failure Run opatchauto resume
]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.
OPatchauto session completed at Thu Feb 6 19:10:21 2020
Time taken to complete the session 6 minutes, 18 seconds
opatchauto failed with error code 42
[root@racdb2 ~]#
The problem is we applying GI patch but grid cannot update oui-patch.xml with group oinstall since the file is 640
Troubleshoot Method:
- Keep the fail opatch session and don’t close it !!
- change the oui-patch.xml permission to 660
[root@racdb2 ~]# ls -l /u01/app/oraInventory/ContentsXML/oui-patch.xml
-rw-r-----. 1 oracle oinstall 174 Feb 6 19:08 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@racdb2 ~]# chmod 660 /u01/app/oraInventory/ContentsXML/oui-patch.xml
[root@racdb2 ~]#
3. Copy necessary files from Node1 to Node2
[grid@racdb1 oneoffs]$ pwd
/u01/app/19.0.0.0/grid/inventory/oneoffs
[grid@racdb1 oneoffs]$ ls -l
total 0
drwxr-x---. 4 grid oinstall 29 Apr 18 2019 29517242
drwxr-x---. 4 grid oinstall 29 Apr 18 2019 29517247
drwxr-x---. 4 grid oinstall 29 Apr 18 2019 29585399
drwxr-x---. 4 grid oinstall 29 Feb 6 16:03 30489227
drwxr-x---. 4 grid oinstall 29 Feb 6 16:04 30489632
drwxr-x---. 4 grid oinstall 29 Feb 6 16:05 30557433
drwxr-x---. 4 grid oinstall 29 Feb 6 16:06 30655595
[grid@racdb1 oneoffs]$
[grid@racdb1 oneoffs]$ scp -r * racdb2:/u01/app/19.0.0.0/grid/inventory/oneoffs/
actions.xml 100% 96KB 29.3MB/s 00:00
inventory.xml 100% 64KB 29.3MB/s 00:00
actions.xml 100% 351 512.4KB/s 00:00
inventory.xml 100% 2857 2.9MB/s 00:00
actions.xml 100% 347 602.8KB/s 00:00
inventory.xml 100% 18KB 16.3MB/s 00:00
inventory.xml 100% 45KB 1.1MB/s 00:00
actions.xml 100% 227KB 4.3MB/s 00:00
inventory.xml 100% 12KB 4.5MB/s 00:00
actions.xml 100% 54KB 937.0KB/s 00:00
inventory.xml 100% 163KB 3.2MB/s 00:00
actions.xml 100% 1652KB 73.6MB/s 00:00
inventory.xml 100% 1683 1.2MB/s 00:00
actions.xml 100% 3657 2.4MB/s 00:00
[grid@racdb1 oneoffs]$
4. resume the opatch which we hold at step 1
[root@racdb2 ~]# opatchauto resume
OPatchauto session is initiated at Thu Feb 6 19:16:10 2020
Session log file is /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/opatchauto2020-02-06_07-16-10PM.log
Resuming existing session with id Y2QQ
Start applying binary patch on home /u01/app/19.0.0.0/grid
Binary patch applied successfully on home /u01/app/19.0.0.0/grid
Starting CRS service on home /u01/app/19.0.0.0/grid
Postpatch operation log file location: /u01/app/grid/crsdata/racdb2/crsconfig/crspatch_racdb2_2019-12-12_03-57-47PM.log
CRS service started successfully on home /u01/app/19.0.0.0/grid
Preparing home /u01/app/oracle/product/19.0.0.0/dbhome1 after database service restarted
No step execution required.........
Trying to apply SQL patch on home /u01/app/oracle/product/19.0.0.0/dbhome1
SQL patch applied successfully on home /u01/app/oracle/product/19.0.0.0/dbhome1
OPatchAuto successful.
--------------------------------Summary--------------------------------
Patching is completed successfully. Please find the summary as follows:
Host:racdb2
RAC Home:/u01/app/oracle/product/19.0.0.0/dbhome1
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /dbbackup/patch/30463609/30501910/30489632
Reason: This patch is not applicable to this specified target type - "rac_database"
Patch: /dbbackup/patch/30463609/30501910/30655595
Reason: This patch is not applicable to this specified target type - "rac_database"
==Following patches were SUCCESSFULLY applied:
Patch: /dbbackup/patch/30463609/30501910/30489227
Log: /u01/app/oracle/product/19.0.0.0/dbhome1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-06-51PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30557433
Log: /u01/app/oracle/product/19.0.0.0/dbhome1/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-06-51PM_1.log
Host:racdb2
CRS Home:/u01/app/19.0.0.0/grid
Version:19.0.0.0.0
Summary:
==Following patches were SKIPPED:
Patch: /dbbackup/patch/30463609/30501910/30489227
Reason: This patch is already been applied, so not going to apply again.
==Following patches were SUCCESSFULLY applied:
Patch: /dbbackup/patch/30463609/30501910/30489632
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-16-25PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30557433
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-16-25PM_1.log
Patch: /dbbackup/patch/30463609/30501910/30655595
Log: /u01/app/19.0.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-02-06_19-16-25PM_1.log
OPatchauto session completed at Thu Feb 6 19:36:20 2020
Time taken to complete the session 20 minutes, 10 seconds
[root@racdb2 ~]#
Patch success !!
but….. not finish yet…….