EBS R12.1.1 upgrade – Seen errors when applying upgrade patch driver u6678700.drv

Error message:
Unable to allocate memory in procedure aiumab().

Solution:
# Export next environment variable and start adpatch again.
export LDR_CNTRL=MAXDATA=0x40000000

Reference:
Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.1.1) for IBM AIX on Power Systems (64-bit) (Doc ID 761569.1)See under “KNOWN ISSUES => Applying large Applications Patches”.

Error message:
FAILED: file amviccn.sql on worker  1.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/amv/12.0.0/patch/115/sql/amviccn.sql &un_amv &pw_amv
Connected.
DECLARE
*
ERROR at line 1:
ORA-29855: error occurred in the execution of ODCIINDEXCREATE routine
ORA-20000: Oracle Text error:
DRG-00100: internal error, arguments : [51021],[drwaf.c],[1605],[],[]
ORA-06512: at “CTXSYS.DRUE”, line 160
ORA-06512: at “CTXSYS.TEXTINDEXMETHODS”, line 366
ORA-06512: at “SYS.DBMS_SQL”, line 1053
ORA-06512: at line 49

Solution:
1. Abort the AutoPatch session of applying the u6678700.drv driver. The restore files are in place. Later on you will restart the AutoPatch session and the executable would continue where it left off.
2. Shutdown the database and database listener.
3. Source the RDBMS ORACLE_HOME Environment.
4. Execute the script $ORACLE_HOME/appsutil/install/adlnkoh.sh (or $ORACLE_HOME/appsutil/clone/adlnkoh.sh) to relink the RDBMS executables. Reference is note 356878.1: “How to relink an Applications Installation of Release 11i and Release 12”.
5. Confirm that the relink completed without any errors.
6. Startup the database and database listener.
7. From the applications terminal window, rerun adpatch. AutoPatch will identify that another session was running and will prompt you to continue where it left off. You should proceed as prompted.
8. Launch adctrl and restart the failed workers.

Reference:
Several Scripts Failing With A ORA-29855 Executing ODCIINDEXCREATE Routine During Upgrade To 12.1.1 (u6678700.drv) (Doc ID 1286033.1)

Error message:
FAILED: file jtfDiagReportMig.sql on worker  3.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/fnd/12.0.0/patch/115/sql/jtfDiagReportMig.sql &un_jtf &pw_jtf &un_apps &pw_apps
Connected.
DECLARE
*
ERROR at line 1:
ORA-20101: Exception while trying to migrate data from jtf_diagnostic_log to
jtf_diagnostic_download_rpt:ORA-04063: package body “APPS.FND_GLOBAL” has errors
ORA-06512: at line 21

Solution:
1. Open a new shell and set the Applications environment using APPS<CONTEXT_NAME>.env
2. adjava -ms128m -mx256m -nojit oracle.apps.fnd.odf2.FndXdfCmp apps <PASS> apps <PASS> THIN “<machine.domain>:<PORT>:<SID>” view $FND_TOP/patch/115/xdf/fnd_user_resp_groups.xdf $FND_TOP/patch/115/xdf/xsl
3. adjava -ms128m -mx256m -nojit oracle.apps.fnd.odf2.FndXdfCmp apps <PASS> apps <PASS> THIN “<machine.domain>:<PORT>:<SID>” view $FND_TOP/patch/115/xdf/wf_user_role_assignments_v.xdf $FND_TOP/patch/115/xdf/xsl
4. adjava -ms128m -mx256m -nojit oracle.apps.fnd.odf2.FndXdfCmp apps <PASS> apps <PASS> THIN “<machine.domain>:<PORT>:<SID>” view $FND_TOP/patch/115/xdf/wf_user_roles.xdf $FND_TOP/patch/115/xdf/xsl
5. sqlplus apps/<PASS>
SQL>alter view fnd_user_resp_groups compile;
6. Launch adctrl and restart the failed worker.

Reference:
12.1.1 Upgrade fails On “Jtfdiagreportmig.Sql” [ID 841705.1]
 

Error message:
FAILED: file p002ncpls.sql on worker  2.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/admin/<ENV>/out/p002ncpls.sql &un_apps CTXSYS &un_csSession altered.Package body created.
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
BEGIN
*
ERROR at line 1:
ORA-04068: existing state of packages has been discarded
ORA-04065: not executed, altered or dropped stored procedure “PUBLIC.PLITBLM”
ORA-06508: PL/SQL: could not find program unit being called: “PUBLIC.PLITBLM”
ORA-06512: at “SYSTEM.AD_DDL”, line 165
ORA-06512: at line 2

Solution:
1. Run $ORACLE_HOME/rdbms/admin/utlirp.sql
2. Run the following query to check for objects with a timestamp mismatch:
select do.obj# d_obj,do.name d_name, po.obj# p_obj,po.name p_name, to_char(p_timestamp,’DD-MM-YYYY HH24:MI:SS’) p_timestamp, to_char(po.stime ,’DD-MM-YYYY HH24:MI:SS’) stime, decode(sign(po.stime-p_timestamp),0,’SAME’,’*DIFFER*’) X,do.type#,po.type#
from sys.obj$ do, sys.dependency$ d, sys.obj$ po
where P_OBJ#=po.obj#(+)
and D_OBJ#=do.obj#
and do.status=1 /*dependent is valid*/
and po.status=1 /*parent is valid*/
and po.stime!=p_timestamp /*parent timestamp not match*/
order by 2,1;
3. Compile any objects that are returned by the above query
SQL> alter package SYS.PLITBLM compile;
SQL> alter package SYS.DBMS_RESUMABLE compile;
SQL> alter package SYS.DBMS_REPCAT_AUTH compile;
SQL> alter package SYSTEM.DBMS_REPCAT_AUTH compile;
Check if the objects are valid now.
4. Launch adctrl and restart the failed worker

Reference:
DBMS_METADATA Fails With ORA-04068 And ORA-04065 [ID 1179501.1]
 

Error message:
FAILED: file p001ncpls.sql on worker 16.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/admin/<ENV>/out/p001ncpls.sql
Session altered.
DECLARE
*
ERROR at line 1:
ORA-00955: name is already used by an existing object
ORA-06512: at line 21
Time when worker failed: Sat Jun 16 2012 16:18:21

Solution:
1. Drop tables below:
drop table IGI.IGI_UPG_GL_BUDGET_ASSIGNMENT;
drop synonym APPS.IGI_UPG_GL_BUDGET_ASSIGNMENT;
select * from dba_objects where object_name=’IGI_UPG_GL_BUDGET_ASSIGNMENT’;

drop table IGI.IGI_UPG_GL_BUDORG_BC_OPTIONS;
drop synonym APPS.IGI_UPG_GL_BUDORG_BC_OPTIONS;
select * from dba_objects where object_name=’IGI_UPG_GL_BUDORG_BC_OPTIONS’;

drop table IGI.IGI_EFC_BUDGET_ASSIGNMENT_BCK;
drop synonym APPS.IGI_EFC_BUDGET_ASSIGNMENT_BCK;
select * from dba_objects where object_name=’IGI_EFC_BUDGET_ASSIGNMENT_BCK’;

drop table IGI.IGI_BUDORG_BC_OPTIONS_BCK;
drop synonym APPS.IGI_BUDORG_BC_OPTIONS_BCK;
select * from dba_objects where object_name=’IGI_BUDORG_BC_OPTIONS_BCK’;

drop table IGI.IGI_EFC_UPG_BACKUP_INFO;
drop synonym APPS.IGI_EFC_UPG_BACKUP_INFO;
select * from dba_objects where object_name=’IGI_EFC_UPG_BACKUP_INFO’;

drop table IGI.IGI_GL_BUDGET_ASSIGN_BCK;
drop synonym APPS.IGI_GL_BUDGET_ASSIGN_BCK;
select * from dba_objects where object_name=’IGI_GL_BUDGET_ASSIGN_BCK’;

2. Restart failed workers in adctrl.
 

Error message:
FAILED: file ibecitem.ldt on worker  1.

Uploading from the data file /<middletier>/apps/apps_st/appl/ibe/12.0.0/patch/115/import/US/ibecitem.ldt
Altering database NLS_LANGUAGE environment to AMERICAN
Dumping from LCT/LDT files (/<middle tier>/apps/apps_st/appl/ibc/12.0.0/patch/115/import/ibccitem.lct, /<middle tier>/apps/apps_st/appl/ibe/12.0.0/patch/115/import/US/ibecitem.ldt) to staging tables
Dumping LCT file /<middle tier>/apps/apps_st/appl/ibc/12.0.0/patch/115/import/ibccitem.lct into FND_SEED_STAGE_CONFIG
Dumping LDT file /<middle tier>/apps/apps_st/appl/ibe/12.0.0/patch/115/import/US/ibecitem.ldt into FND_SEED_STAGE_ENTITY
Dumped the batch (IBC_CONTENT_ITEMS 51 , IBC_CITEM_VERSIONS 52 ) into FND_SEED_STAGE_ENTITY
Uploading from staging tables
Error loading seed data for IBC_CITEM_VERSIONS:  CONTENT_ITEM_ID = 51, CITEM_VERSION_ID = 51,  ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE
Error loading seed data for IBC_CITEM_VERSIONS:  CONTENT_ITEM_ID = 52, CITEM_VERSION_ID = 52,  ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE
Concurrent request completed

Solution:
1. Check status of Domain index:
SQL> select index_name, owner, status, DOMIDX_status, domidx_opstatus from dba_indexes where index_type=’DOMAIN’ and domidx_opstatus=’FAILED’;

INDEX_NAME                            OWNER     STATUS DOMIDX_STATUS DOMIDX
————————–               ———–  ——–  ——————–  ——
IGP_US_PORTFOLIOS_CTX1       IGS          VALID    VALID                 FAILED
IGP_US_PORTFOLIOS_CTX2       IGS          VALID    VALID                 FAILED
IBC_ATTRIBUTE_BUNDLES_CTX IBC          VALID    VALID                 FAILED
IGP_US_PORTFOLIOS_CTX3       IGS          VALID    VALID                 FAILED

2. Fix Intermedia Tables:
SQL> exec ctx_ddl.sync_index(‘IBC.IBC_ATTRIBUTE_BUNDLES_CTX’);
3. Restart failed workers in adctrl.

Error message:
FAILED: file EGOSILDU.sql on worker  3.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/ego/12.0.0/patch/115/sql/EGOSILDU.sql &un_ego
DECLARE
*
ERROR at line 1:
ORA-02149: Specified partition does not exist
ORA-06512: at line 408

Solution:
1. Check if the EGO product is installed/used.  You can use “$AD_TOP/sql/adutconf.sql” to find out if you are using the product.
2. If you have not installed EGO product or if you are not using “Advanced Product Catalog” functionality, you can skip the failed job. Use adctrl hidden option #8, to skip the failed job.
3. If you are using this product, Please raise an SR in order to get the detailed procedure to fix the issue listed in bug 8485942

Reference:
Patch 7303030 Fails on Egosildu.sql : ORA-02149 Specified Partition Does Not Exist [ID 847687.1]
 

Error message:
FAILED: file zpbremcwm.sql on worker  1.

sqlplus -s APPS/***** @/<middle tier>/apps/apps_st/appl/zpb/12.0.0/patch/115/sql/zpbremcwm.sqlcwm2_olap_cube.drop_cube(G_SCHEMA, v_cube.name);
ERROR at line 235:
ORA-06550: line 235, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_CUBE
ORA-06550: line 235, column 6:
PL/SQL: Statement ignored
ORA-06550: line 239, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_DIMENSION
ORA-06550: line 239, column 6:
PL/SQL: Statement ignored
ORA-06550: line 243, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_CATALOG
ORA-06550: line 243, column 6:
PL/SQL: Statement ignored
ORA-06550: line 247, column 4:
PLS-00904: insufficient privilege to access object
OLAPSYS.CWM2_OLAP_METADATA_REFRESH
ORA-06550: line 247, column 4:
PL/SQL: Statement ignored

Solution:
1. Grant privileges:
SQL> grant execute on OLAPSYS.CWM2_OLAP_METADATA_REFRESH to apps;
SQL> grant execute on OLAPSYS.CWM2_OLAP_CATALOG to apps;
SQL> grant execute on OLAPSYS.CWM2_OLAP_DIMENSION to apps;
SQL> grant execute on OLAPSYS.CWM2_OLAP_CUBE to apps;
2. Use adctrl option #2 to restart the failed worker.

Reference:
ZPBREMCWM.SQL Returns Error: PLS-00904: Insufficient Privilege To Access Object (Doc ID 744973.1)
 

Error message:
FAILED: file csmidxdel.sql on worker  11.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/csm/12.0.0/patch/115/sql/csmidxdel.sql
CSM_UTIL_PKG.LOG(message =>’Dropping the index CSM_NEW_MESSAGES_ACC_U2′
*
ERROR at line 15:
ORA-06550: line 15, column 2:
PLS-00201: identifier ‘CSM_UTIL_PKG.LOG’ must be declared
…
PL/SQL: Statement ignored

Cause:
Package CSM_UTIL_PKG doesn’t exist.

Solution:
1. Check if the CSM product is installed/used.  You can use “$AD_TOP/sql/adutconf.sql” to find out if you are using the product.
2. If you have not installed CSM product, you can skip the failed job. Use adctrl hidden option #8, to skip the failed job.
3. If you are using this product, Please raise an SR in order to get the detailed procedure to fix the issue.
 

Error message:
FAILED: file pechktsk.sql on worker   3.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/per/12.0.0/patch/115/sql/pechktsk.sql
DECLARE
*
ERROR at line 1:
ORA-00979: not a GROUP BY expression
ORA-06512: at line 38

Solution:
1. Check current version of pechktsk.sql:
strings -a $PER_TOP/patch/115/sql/pechktsk.sql |grep ‘$Header’
Download patch 8855023 if the version is lower than 120.0.12010000.2.
2. Backup current version of pechktsk.sql:
mv $PER_TOP/patch/115/sql/pechktsk.sql $PER_TOP/patch/115/sql/pechktsk.sql.old
3. Copy new version of pechktsk.sql from patch 8855023 to $PER_TOP/patch/115/sql:
cp …/8855023/per/patch/115/sql/pechktsk.sql $PER_TOP/patch/115/sql/pechktsk.sql
4. Check version of new pechktsk.sql:
strings -a $PER_TOP/patch/115/sql/pechktsk.sql |grep ‘$Header’
rem $Header: pechktsk.sql 120.0.12010000.2 2009/09/15 06:31:36 sidsaxen ship $
5. Launch adctrl and restart the failed worker

Reference:
Applying Patch 6678700 or Patch 3500000 Pechktsk.Sql Errors with: ORA-00979: Not A Group By Expression (Doc ID 1083981.1)
 

Error message:
FAILED: file armquet.sql  on worker  1.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/ar/12.0.0/patch/115/sql/armquet.sql &un_ar &pw_ar &un_ar
Connected.
PL/SQL procedure successfully completed.
DECLARE
*
ERROR at line 1:
ORA-20000: Add subscriber Error ( AR_REV_REC_Q) : -4063 – ORA-04063: AR.AR_REV_REC_Q has errors
ORA-06512: at line 18

Solution:
1. Check if the status of the queue is invalid:
sqlplus apps
SQL> select owner, object_name, object_type, status from dba_objects where object_name=’AR_REV_REC_Q’;

OWNER  OBJECT_NAME     OBJECT_TYPE  STATUS
——— ——————-   —————-  ———————
AR         AR_REV_REC_Q   QUEUE            INVALID

2. Drop the queue and the queue table:
SQL> exec dbms_aqadm.drop_queue(queue_name =>’AR.AR_REV_REC_Q’,auto_commit=>TRUE);
PL/SQL procedure successfully completed.
SQL> exec dbms_aqadm.drop_queue_table(queue_table=>’AR.AR_REV_REC_QT’,force=>TRUE);
PL/SQL procedure successfully completed.
3. Create the queue and queue table again:
SQL> @$AR_TOP/patch/115/sql/arcquet.sql apps <apps pwd> AR APPS_TS_QUEUES
4. Grant privileges and create synonym:
# Reference: $AR_TOP/patch/115/sql/arcques.sql:
sqlplus apps
SQL> execute AD_INST.DO_APPS_DDL(‘AR’, ‘GRANT ALL ON AR_REV_REC_QT TO ‘||’APPS’||’ WITH GRANT OPTION’);
SQL> create or replace synonym apps.AR_REV_REC_QT for ar.AR_REV_REC_QT;
5. Start the queue.
sqlplus apps
SQL> @$AR_TOP/patch/115/sql/arstque.sql AR
6. Verify that the queue is valid:
sqlplus apps
SQL> select owner, object_name, object_type, status from dba_objects where object_name=’AR_REV_REC_Q’;

OWNER  OBJECT_NAME     OBJECT_TYPE   STATUS
———  ——————– —————– ———————
AR         AR_REV_REC_Q   QUEUE             VALID

7. Launch adctrl and restart the failed worker

Error message:
FAILED: file XNPQFIX.sql on worker 1.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/xnp/12.0.0/patch/115/sql/XNPQFIX.sql &un_xnp &pw_xnp &un_xnp
Connected.
DECLARE
*
ERROR at line 1:
ORA-20001: Error While modifying queue XNP.XNP_OUT_MSG_Q
ORA-06512: at line 77

Solution:
1. Check current queue status:
sqlplus “/ as sysdba”
SQL> select owner, object_name, object_type, status from dba_objects where object_type=’QUEUE’ and object_name=’XNP_OUT_MSG_Q’;

OWNER      OBJECT_NAME          OBJECT_TYPE        STATUS
———-    ———————— ——————-    ———————
XNP           XNP_OUT_MSG_Q     QUEUE                  INVALID

2. Stop the queues:
SQL> exec dbms_aqadm.stop_queue(queue_name => ‘XNP.XNP_OUT_MSG_Q’,wait => FALSE);
SQL> exec dbms_aqadm.stop_queue(queue_name => ‘XNP.XNP_OUT_MSG_EXCEPTION_Q’, wait => FALSE);
3. Drop the queue:
SQL> exec dbms_aqadm.drop_queue(queue_name=> ‘XNP.XNP_OUT_MSG_Q’);
4. Create the queue:
SQL> exec dbms_aqadm.create_queue (queue_name  => ‘XNP.XNP_OUT_MSG_Q’, queue_table => ‘XNP.XNP_OUT_MSG_QTAB’, max_retries => 5, retry_delay => 3600, retention_time => 0, auto_commit => FALSE );
5. Start the queues:
SQL> exec dbms_aqadm.start_queue(queue_name => ‘XNP.XNP_OUT_MSG_Q’);
SQL> exec dbms_aqadm.start_queue(queue_name => ‘XNP.XNP_OUT_MSG_EXCEPTION_Q’, enqueue => FALSE, dequeue => TRUE);
6. Check current queue status:
SQL> select owner, object_name, object_type, status from dba_objects where object_type=’QUEUE’ and object_name=’XNP_OUT_MSG_Q’;

OWNER   OBJECT_NAME                  OBJECT_TYPE   STATUS
———- —————————— —————–   —————-
XNP        XNP_OUT_MSG_Q             QUEUE              VALID
7. Launch adctrl and restart the failed worker

Reference:
SFM Event Manager Queue Service (SFM) Error ORA-00904: “TAB”.”USER_PROP”: INVALID IDENTIFIER (Doc ID 1061711.1). The note is specifically written for Install Base users experiencing a specific error with the 2 queue services required for Install Base functionality but the same steps given in the solution can be used to rebuild the XNP queues.
 

Error message:
FAILED: file PO_VENDORS_OBS_F1.xdf on worker  3.

AD Worker error:
The utility FndXdfCmp returned error for the above task.

Solution:
1. Drop index:
sqlplus apps/<apps pwd>
SQL> drop index PO.PO_VENDORS_F1;
2. Launch adctrl and restart the failed worker.

Reference:
Creation of Index PO_VENDORS_OBS_F1 Failed When Applying 12.1.1 Upgrade Patch 6678700 (Doc ID 859556.1)
 

Error Message:
FAILED: file cstfifo.odf  on worker  1.

ORA-00979: not a GROUP BY expression

Solution:
1. Download and unzip patch 9868229.
2. Backup current cstfifo.odf file.
cp $BOM_TOP/patch/115/odf/cstfifo.odf $BOM_TOP/patch/115/odf/cstfifo.odf.org
3. Copy cstfifo.odf file from patch 9868229 to the $BOM_TOP/patch/115/odf directory:
cp …/9868229/bom/patch/115/odf/cstfifo.odf $BOM_TOP/patch/115/odf/cstfifo.odf
4. Launch adctrl and restart the failed worker.

Reference:
This error can be avoided when applying patch 9868229 before upgrade patch 6678700.
12.1.1 Upgrade – cstfifo.odf fails with error ‘ORA-00979: not a group by expression’ (Doc ID 1274012.1)

Error message:
FAILED: file ircntf.wft   on worker  1.

Access level: 20, Mode: UPGRADE
ITEM_TYPE ‘WFSTD’ is protected, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/EVENTNAME’ is customized, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/N’ is protected, no changes were saved.
…

Solution:
1. Run WFLOAD with FORCE option as the applmgr user:
$FND_TOP/bin/WFLOAD apps/<apps pwd> 0 Y FORCE $IRC_TOP/patch/115/import/US/ircntf.wft
2. Use adctrl hidden option #8, to skip the failed job.

Reference:
IRCNTF.WFT Fails During R12.1.1 Upgrade (Doc ID 1329530.1).
Oracle Support advice is to skip this error and complete the upgrade.  This error should not impact any iRecruitment functionality apart from Generic Notifications in iRecruitment which is introduced in 12.1. Furthermore they suggest to apply Patch 9371492: IRCNTF.WFT FAILS DURING R12.1.1 UPGRADE. This isn’t possible during deployment of patch 6678700, so I run WFLOAD with the FORCE option.
 

Error message:
FAILED: file OKLIAADD.wft on worker  2.

Access level: 20, Mode: UPGRADE
ITEM_TYPE ‘WFSTD’ is protected, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/N’ is protected, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/REQUESTOR_USERNAME’ is protected, no changes were saved.
…

Solution:
1. Run WFLOAD with FORCE option as the applmgr user:
$FND_TOP/bin/WFLOAD apps/<apps pwd> 0 Y FORCE $OKL_TOP/patch/115/import/US/OKLIAADD.wft
2. Use adctrl hidden option #8, to skip the failed job.

Reference:
IRCNTF.WFT Fails During R12.1.1 Upgrade (Doc ID 1329530.1).
Oracle Support advice is to skip this error and complete the upgrade.  This error should not impact any iRecruitment functionality apart from Generic Notifications in iRecruitment which is introduced in 12.1. Furthermore they suggest to apply Patch 9371492: IRCNTF.WFT FAILS DURING R12.1.1 UPGRADE. This isn’t possible during deployment of patch 6678700, so I run WFLOAD with the FORCE option.

Error message:
FAILED: file PAPWPARN.wft on worker  3.

Access level: 20, Mode: UPGRADE
ITEM_TYPE ‘WFSTD’ is protected, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/N’ is protected, no changes were saved.
ITEM_ATTRIBUTE ‘WFSTD/REQUESTOR_USERNAME’ is protected, no changes were saved.
…

Solution:
1. Run WFLOAD with FORCE option as the applmgr user:
$FND_TOP/bin/WFLOAD apps/<apps pwd> 0 Y FORCE $PA_TOP/patch/115/import/US/PAPWPARN.wft
2. Use adctrl hidden option #8, to skip the failed job.

Reference:
This error is described in MOS 1061488.1. PAPWPARN.WFT ERRORS ON UPGRADE TO 12.1.1 (Doc ID 1061488.1). Oracle Support suggest to apply patch 12347791: 1OFF:11841086:12.1.1:PATCH 6678700 FAILS 1602 ORACLE WORKFLOW DEFINITION. This isn’t possible during deployment of patch 6678700, so I run WFLOAD with the FORCE option.
 

Error message:
FAILED: file IGIGITRA.wft on worker  8.

wferr:
– 1602: Could not save.
– 1400: Could not save to database. MODE=UPGRADE EFFDATE=<null>
– 1401: Could not upload MESSAGE entity ‘ITRAPPRV/NOTIFY_PREPARER_OF_REJECTION’.
– 1401: Could not upload ATTRIBUTE entity ‘<null>/COMMENT’.
– 210: Oracle Error: ORA-00001: unique constraint (APPLSYS.WF_MESSAGE_ATTRIBUTES_TL_U2) violated
ORA-06512: at “APPS.WF_LOAD”, line 1067
ORA-06512: at line 1.  SQL text: BEGIN WF_LOAD.UPLOAD_MESSAGE_ATTRIBUTE(:message_type, :message_name, :name, :display_name, :description, :sequence, :type, :subtype, :protect, :custom, :format, :default, :value_type, :attach, :level_error); END;
Oracle Workflow Definition Loader 2.6.4.0.

Solution: 
1. Update wf_message_attributes_tl:
sqlplus apps
SQL> UPDATE wf_message_attributes_tl
SET display_name =’Approver Comment’
WHERE message_type=’ITRAPPRV’
AND message_name in (‘NOTIFY_PREPARER_OF_APPROVAL’,
‘NOTIFY_PREPARER_OF_REJECTION’, ‘NOTIFY_PREPARER_NOT_SUBMITTED’)
AND name=’APPROVER_COMMENT’
AND display_name=’Comment’
AND language = ‘DK’;
2. Run WFLOAD with FORCE option as the applmgr user:
$FND_TOP/bin/WFLOAD apps/<apps pwd> 0 Y FORCE $IGI_TOP/patch/115/import/US/IGIGITRA.wft
3. Use adctrl hidden option #8, to skip the failed job.

Reference:
DOC ID 471851.1 which is visible for Oracle Support only.
 

Error message:
FAILED: file oklresp.ldt  on worker  5.

Error calling FNDLOAD function.
Error loading seed data for FND_RESP_FUNCTIONS:  APPLICATION_SHORT_NAME = OKL, RESP_KEY = LEASE_TERMINATIONS, RULE_TYPE = M, ACTION = OKL_AM_REMARKET,  ORA-01403: no data found
Error loading seed data for FND_RESP_FUNCTIONS:  APPLICATION_SHORT_NAME = OKL, RESP_KEY = LEASE_CONTRACT_ADMINISTRATOR, RULE_TYPE = M, ACTION = OKL_AM_REMARKET,  ORA-01403: no data found

Solution:
1. Check if the OKL product is installed/used.  You can use “$AD_TOP/sql/adutconf.sql” to find out if you are using the product.
2. If you have not installed OKL product, you can skip the failed job. Use adctrl hidden option #8, to skip the failed job.
3. If you are using this product, Please raise an SR in order to get the detailed procedure to fix.

Error message:
FAILED: file MSDODPCODE.sql on worker  1.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/msd/12.0.0/patch/115/sql/MSDODPCODE.sql
begin
*
ERROR at line 1:
ORA-33292: Insufficient permissions to access analytic workspace APPS.ODPCODE
using the specified access mode.
ORA-06512: at “SYS.DBMS_AW”, line 67
ORA-06512: at “SYS.DBMS_AW”, line 131
ORA-06512: at “SYS.DBMS_AW”, line 977
ORA-06512: at “SYS.DBMS_AW”, line 930
ORA-33262: Analytic workspace APPS.ODPCODE does not exist.
ORA-06512: at “SYS.DBMS_AW”, line 901
ORA-06512: at “APPS.MSD_AW_LOADER”, line 10
ORA-06512: at line 3

Solution:
1. Unlock the OLAPSYS account:
sqlplus “/ as sysdba”
SQL> select username ,account_status from dba_users where username like ‘%OLAP%’;

USERNAME                 ACCOUNT_STATUS
————————- ——————————–
OLAPSYS                   LOCKED

SQL> ALTER USER OLAPSYS ACCOUNT UNLOCK;
2. Run next OLAP DML command/procedure:
sqlplus apps
SQL> exec dbms_aw.execute(‘aw create odpcode’);
3. Launch adctrl and restart the failed worker.

Reference:
Patch 6678700 Worker Fails On Applying MSDODPCODE.sql With ORA-33292 Insufficient Permissions To Access Analytic Workspace APPS.ODPCODE; ORA-33262: Analytic workspace ODPCODE does not exist. (Doc ID 1054417.1)

Error message:
FAILED: file rgcolset.ldt on worker  3.

Error calling FNDLOAD function.
Uploading from staging tables
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_1, AXIS_SEQ = 40, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_2, AXIS_SEQ = 30, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_10, AXIS_SEQ = 130, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_11, AXIS_SEQ = 50, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_12, AXIS_SEQ = 50, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows
Error loading seed data for RG_REPORT_CALCULATIONS:  SEEDED_NAME = FSG_COLUMN_SET_14, AXIS_SEQ = 130, CALCULATION_SEQ = 10,  ORA-01422: exact fetch returns more than requested number of rows

Solution:
1. Verify that duplicates exist:
sqlplus apps
SQL> SELECT axis_set_id, axis_seq, calculation_seq, application_id, count(*)
FROM RG_REPORT_CALCULATIONS
group by axis_set_id, axis_seq, calculation_seq, application_id
having count(*) > 1;
SQL> SELECT axis_set_id, axis_seq, calculation_seq, count(*)
FROM RG_REPORT_CALCULATIONS
group by axis_set_id, axis_seq, calculation_seq
having count(*) > 1;
2. Create table backups.
SQL> CREATE TABLE xx_rg_calc_dup
AS
SELECT axis_set_id, axis_seq, calculation_seq, application_id
FROM rg_report_calculations
GROUP BY axis_set_id, axis_seq, calculation_seq, application_id
HAVING count(*) > 1;
SQL> create table XX_rg_report_calculations_BCK as select * from rg_report_calculations;
3. Delete duplicate rows.
SQL> delete from rg_report_calculations
where rowid not in (select min(rowid)
from rg_report_calculations
group by axis_set_id, axis_seq, calculation_seq);
SQL> commit;
4. Verify that no duplicates exist anymore:
SQL> SELECT rcd.*
FROM xx_rg_calc_dup rcd
WHERE NOT EXISTS (SELECT 1
FROM rg_report_calculations rrc
WHERE rcd.APPLICATION_ID = rrc.APPLICATION_ID
AND rcd.AXIS_SEQ = rrc.AXIS_SEQ
AND rcd.AXIS_SET_ID = rrc.AXIS_SET_ID
AND rcd.CALCULATION_SEQ = rrc.CALCULATION_SEQ) ;
no rows selected
5. Launch adctrl and restart the failed worker.
6. Cleanup/remove backup tables:
sqlplus apps
SQL> DROP TABLE xx_rg_calc_dup;
SQL> DROP TABLE XX_rg_report_calculations_BCK;

Error message:
FAILED: file wfstartq.sql on worker  3.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/fnd/12.0.0/patch/115/sql/wfstartq.sql &un_fnd
declare
*
ERROR at line 1:
ORA-04063: APPLSYS.WF_DEFERRED_QUEUE_M has errors
ORA-06512: at “SYS.DBMS_AQADM_SYS”, line 5356
ORA-06512: at “SYS.DBMS_AQADM”, line 289
ORA-06512: at line 21

Solution:
1. Drop the queue and the queue table:
sqlplus “/ as sysdba”
SQL> select owner, object_name, object_type, status from dba_objects where object_name like ‘WF_%’ and object_type=’QUEUE’ and status=’INVALID’;

OWNER      OBJECT_NAME                  OBJECT   STATUS
———- ——————————–  ———-  ———————
APPLSYS    WF_DEFERRED_QUEUE_M  QUEUE     INVALID

SQL> exec dbms_aqadm.drop_queue(queue_name=> ‘APPLSYS.WF_DEFERRED_QUEUE_M’);
SQL> exec dbms_aqadm.drop_queue_table(‘APPLSYS.WF_DEFERRED_TABLE_M’,TRUE);
2. Create the queue and the queue table again (applmgr user):
sqlplus apps/<apps pwd> @$FND_TOP/patch/115/sql/wfquec2.sql APPS APPLSYS <applsys pwd>
3. Launch adctrl and restart the failed worker.

Reference:
Workflow Queues Creation Scripts (Doc ID 398412.1)

Error message:
FAILED: file cskbcat.ldt  on worker  3.

Error calling FNDLOAD function.
Uploading from staging tables
Error loading seed data for CS_KB_SOLN_CATEGORIES_VL:  CATEGORY_ID = 1,  ORA-29861: domain index is marked LOADING/FAILED/UNUSABLE

Solution:
1. Create Packages and Install CS Text index preferences and indexes
sqlplus apps
SQL> @$CS_TOP/patch/115/sql/cskfctxs.pls APPS
SQL> @$CS_TOP/patch/115/sql/cskfctxb.pls APPS
SQL> @$CS_TOP/patch/115/sql/cskbctxp.sql APPS CS CTXSYS
SQL> @$CS_TOP/patch/115/sql/cskbsetx.sql CS
SQL> @$CS_TOP/patch/115/sql/cssrcrix.sql APPS CS
SQL> @$CS_TOP/patch/115/sql/cskbelex.sql CS
SQL> @$CS_TOP/patch/115/sql/cskbcatx.sql CS
SQL> @$CS_TOP/patch/115/sql/cskbforx.sql CS
SQL> @$CS_TOP/patch/115/sql/csksynib.pls CS APPS
2. Launch adctrl and restart the failed worker.
 

Error message:
FAILED: file jtfpfstart.sql on worker  1.

sqlplus -s APPS/***** @/opt/ebsr12/apps/apps_st/appl/jtf/12.0.0/patch/115/sql/jtfpfstart.sql &un_apps &un_jtf
dbms_aqadm.drop_queue_table(queue_table=>JTF_PF_CONV_PKG.QTABLENAME, force=>true);
*
ERROR at line 23:
ORA-06550: line 23, column 47:
PLS-00905: object APPS.JTF_PF_CONV_PKG is invalid
ORA-06550: line 23, column 6:
PL/SQL: Statement ignored
…

Solution:
1. Logon as the applmgr user and change directory:
cd $JTF_TOP/patch/115/xdf
2. Run next java command:
adjava -mx512m -nojit oracle.apps.fnd.odf2.FndXdfCmp apps <apps pwd> apps <apps pwd> thin <database tier>:<db port>:<ENV> type jtf_pf_tabletype.xdf $FND_TOP/patch/115/xdf/xsl
3. If there are no errors, recompile the JTF_PF_CONV_PKG package:
sqlplus apps/<apps pwd>
SQL> alter package APPS.JTF_PF_CONV_PKG compile;
SQL> show err
4. Launch adctrl and restart the failed worker.

Reference:
Upgrade 11i to R12.1.3 worker fails on jtfpfstart.sql PLS-00905: object APPS.JTF_PF_CONV_PKG is invalid (Doc ID 1463373.1)
 

Error message:
FAILED: file okladtxm.sql on worker  1.

ERROR at line 110:
ORA-06550: line 110, column 9:
PL/SQL: ORA-02289: sequence does not exist
ORA-06550: line 68, column 5:
PL/SQL: SQL Statement ignored

Solution:
1. Check if the OKL product is installed/used.  You can use “$AD_TOP/sql/adutconf.sql” to find out if you are using the product.
2. If you have not installed the OKL product, you can skip the failed job. Use adctrl hidden option #8, to skip the failed job.
3. If you are using this product, see MOS 1570669.1 or raise an SR in order to get the detailed procedure to fix.

Reference:
Patch 9239090 Fails While Running Okladtxm.sql “PL/SQL: ORA-02289: sequence does not exist” (Doc ID 1570669.1)
 

Error message:
FAILED: file zpbremcwm.sql on worker  1.

ERROR at line 235:
ORA-06550: line 235, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_CUBE
ORA-06550: line 235, column 6:
PL/SQL: Statement ignored
ORA-06550: line 239, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_DIMENSION
ORA-06550: line 239, column 6:
PL/SQL: Statement ignored
ORA-06550: line 243, column 6:
PLS-00904: insufficient privilege to access object OLAPSYS.CWM2_OLAP_CATALOG
ORA-06550: line 243, column 6:
PL/SQL: Statement ignored
ORA-06550: line 247, column 4:
PLS-00904: insufficient privilege to access object
OLAPSYS.CWM2_OLAP_METADATA_REFRESH
ORA-06550: line 247, column 4:
PL/SQL: Statement ignored

Solution:
1. Grant the following privileges to the APPS user:
sqlpus system
grant execute on OLAPSYS.CWM2_OLAP_METADATA_REFRESH to apps;
grant execute on OLAPSYS.CWM2_OLAP_CATALOG to apps;
grant execute on OLAPSYS.CWM2_OLAP_DIMENSION to apps;
grant execute on OLAPSYS.CWM2_OLAP_CUBE to apps;
2. Launch adctrl and restart the failed worker.

Reference:
ZPBREMCWM.SQL Returns Error: PLS-00904: Insufficient Privilege To Access Object (Doc ID 744973.1) 

Error message:
FAILED: file biv_b_age_h_sum_mv.xdf on worker  5.

Exception occured  ORA-54015: Duplicate column expression was specified
ORA-06512: at “APPS.AD_MV”, line 131
ORA-06512: at “APPS.AD_MV”, line 289
ORA-06512: at line 1

Solution:
1. Download and unzip patch 10163753.
2. Backup current biv_b_age_h_sum_mv.xdf file:
cp $BIV_TOP/patch/115/xdf/biv_b_age_h_sum_mv.xdf $BIV_TOP/patch/115/xdf/biv_b_age_h_sum_mv.xdf.org
3. Copy the biv_b_age_h_sum_mv.xdf file from patch 10163753 to the $BIV_TOP/patch/115/xdf directory:
cp …/10163753/biv/patch/115/xdf/biv_b_age_h_sum_mv.xdf $BIV_TOP/patch/115/xdf
4. Launch adctrl and restart the failed worker.

Reference:
This error is described in MOS 1322144.1. This error can be avoided when applying patch 10163753 before upgrade patch 6678700.
Error – ORA-54015 : biv_b_age_h_sum_mv.xdf Failes with Duplicate Column Expression was Specified (Doc ID 1322144.1)

Error message:
FAILED: file MSC_RESOURCE_HIERARCHY_MV.xdf on worker  3.

Exception occured  ORA-00955: name is already used by an existing object
ORA-06512: at “APPS.AD_MV”, line 131
ORA-06512: at “APPS.AD_MV”, line 289
ORA-06512: at line 1

Solution:
1. Check if object MSC_RESOURCE_HIERARCHY_MV already exist:
SQL> select owner, object_name, object_type, status, created
from dba_objects where object_name in (‘MSC_RESOURCE_HIERARCHY_MV’)
order by 2;

OWNER  OBJECT_NAME                             OBJECT_TYPE       STATUS   CREATED
——— —————————————- ——————– ———– —————
MSC      MSC_RESOURCE_HIERARCHY_MV  TABLE                  VALID     03-SEP-11
MSC      MSC_RESOURCE_HIERARCHY_MV  TABLE PARTITION VALID     03-SEP-11
MSC      MSC_RESOURCE_HIERARCHY_MV  TABLE PARTITION VALID     03-SEP-11
MSC      MSC_RESOURCE_HIERARCHY_MV  TABLE PARTITION VALID     03-SEP-11
APPS     MSC_RESOURCE_HIERARCHY_MV  SYNONYM             VALID     03-SEP-11

2. Check if  MSC_RESOURCE_HIERARCHY_MV contain any rows:
SQL> select count(*) from MSC.MSC_RESOURCE_HIERARCHY_MV;

COUNT(*)
———-
0

3. Drop the object if it contains no rows:
SQL> Drop synonym MSC_RESOURCE_HIERARCHY_MV;
SQL> drop table MSC.MSC_RESOURCE_HIERARCHY_MV;
4. Verify that no objects with the name MSC_RESOURCE_HIERARCHY_MV exist:
SQL> select owner, object_name, object_type, status, created
from dba_objects where object_name in (‘MSC_RESOURCE_HIERARCHY_MV’) order by 2;
no rows selected
5. Launch adctrl and restart the failed worker.
 

Error message:
FAILED: file MSC_ITEM_HIERARCHY_MV.xdf on worker  4.

Job failed with error below:
Exception occured  ORA-00955: name is already used by an existing object
ORA-06512: at “APPS.AD_MV”, line 131
ORA-06512: at “APPS.AD_MV”, line 289
ORA-06512: at line 1

Solution:
1. Check if object MSC_ITEM_HIERARCHY_MV already exist:
SQL> select owner, object_name, object_type, status, created
from dba_objects where object_name in (‘MSC_ITEM_HIERARCHY_MV’)
order by 2;

OWNER OBJECT_NAME                     OBJECT_TYPE       STATUS  CREATED
——– ——————————–   ——————–  ———- —————
MSC     MSC_ITEM_HIERARCHY_MV  TABLE PARTITION  VALID    03-SEP-11
MSC     MSC_ITEM_HIERARCHY_MV  TABLE                   VALID    03-SEP-11
MSC     MSC_ITEM_HIERARCHY_MV  TABLE PARTITION  VALID    03-SEP-11
MSC     MSC_ITEM_HIERARCHY_MV  TABLE PARTITION  VALID    03-SEP-11
APPS    MSC_ITEM_HIERARCHY_MV  SYNONYM              VALID    03-SEP-11

2. Check if MSC_ITEM_HIERARCHY_MV contain any rows:
SQL> select count(*) from MSC.MSC_RESOURCE_HIERARCHY_MV;

COUNT(*)
———-
0

3. Drop the object if it contains no rows:
SQL> drop synonym MSC_ITEM_HIERARCHY_MV;
SQL> drop table MSC.MSC_ITEM_HIERARCHY_MV;
4. Verify that no objects with the name MSC_ITEM_HIERARCHY_MV exist:
SQL> select owner, object_name, object_type, status, created
from dba_objects where object_name in (‘MSC_ITEM_HIERARCHY_MV’) order by 2;
no rows selected
5. Launch adctrl and restart the failed worker.
 

Error message:
FAILED: file fem_bal_nacc_hier_l2_mv.xdf on worker  1.

Exception occured  ORA-54015: Duplicate column expression was specified
ORA-06512: at “APPS.AD_MV”, line 131
ORA-06512: at “APPS.AD_MV”, line 289
ORA-06512: at line 1
from FEM_DIS_NAT_ACCTS_HIER_B,
*
ERROR at line 34:
ORA-54015: Duplicate column expression was specified

Solution:
1. Download and unzip patch 10406817
2. Backup the fem_bal_nacc_hier_l2_mv.xdf file:
mv $FEM_TOP/patch/115/xdf/fem_bal_nacc_hier_l2_mv.xdf $FEM_TOP/patch/115/xdf/fem_bal_nacc_hier_l2_mv.xdf.org
3. Copy the fem_bal_nacc_hier_l2_mv.xdf file from the patch to the $FEM_TOP/patch/115/xdf directory:
cp …/10406817/fem/patch/115/xdf/fem_bal_nacc_hier_l2_mv.xdf $FEM_TOP/patch/115/xdf/fem_bal_nacc_hier_l2_mv.xdf
4. Launch adctrl and restart the failed worker.

Reference:
Patch Fails on Fem_bal_nacc_hier_l2_mv.xdf After Upgrade to Oracle 11.2.0.2 Database (Doc ID 1284055.1)
Index Creation Failed on FEM_BAL_NACC_HIER_L2_MV With ORA-54015 During Import Of Oracle Apps (12.1.1) On Oracle DB (11.2.0.2) (Doc ID 1526255.1)

Error message:
FAILED: file czhist.sql   on worker  1.

BEGIN FND_STATS.GATHER_COLUMN_STATS(‘CZ’,’CZ_DEVL_PROJECTS’,’DELETED_FLAG’); END;
ERROR at line 1:
ORA-20002: Version of statistics table APPLSYS.FND_STATTAB is too old.  Please try upgrading it with dbms_stats.upgrade_stat_table
ORA-06512: at “APPS.FND_STATS”, line 664
ORA-06512: at “APPS.FND_STATS”, line 2497
ORA-06512: at line 1

Solution:
1. Run the upgrade_stat_table procedure on the fnd_stattab table:
sqlplus “/ as sysdba”
SQL> EXECUTE DBMS_STATS.UPGRADE_STAT_TABLE(‘APPLSYS’,’FND_STATTAB’);
2. Launch adctrl and restart the failed worker.

Reference:
Script Czhist.Sql Fails During Application Of Patch 6678700 (Doc ID 1281478.1)
 

Error message:
FAILED: file ademusr.sql  on worker  2.

GRANT select on WF_DEFERRED to em_oam_monitor_role
AD Worker error:
The following ORACLE error:
ORA-01031: insufficient privileges

Solution:
1. Grant privileges to APPS:
sqlplus “/ as sysdba”
SQL> GRANT select on applsys.WF_DEFERRED to APPS with grant option;
SQL> GRANT select on applsys.WF_DEFERRED_TABLE_M to APPS with grant option;
SQL> GRANT select on applsys.WF_ERROR to APPS with grant option;
2. Launch adctrl and restart the failed worker.

Reference:
Get ORA-01031: Insufficient Privileges Running Ademusr.Sql On WF_ERROR Table (Doc ID 437760.1)