After several HDDs were replaced in a FCO (Field Change Operation) by Oracle Support on two ODAs, we ran into an unpleasent ORA-600 [kfdJoin3], which resulted in ASM disk groups not mounting, thereby causing the CRS stack to not startup properly.
The “crsctl check crs” command returned the following:
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
Normally you’ll see this picture during startup of the GI stack – simply repeat the check command and have patience for a couple of minutes. In our case it never changed to ONLINE for all components. This issue was first discovered after the ODA-nodes were rebooted – the GI stack and databases had continued to run for a couple of weeks after the HDD replacement had been carried out (an online operation, which by the way, was done by the book according to Oracle Support).
An attempt to stop CRS (had to use “force” option) and then start CRS resulted in the following error:
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-00600: internal error code, arguments: [kfdJoin3], [7], [12], [500], [], [], [], [], [], [], [], []
ASM alert-log showed error messages like the following:
ERROR: diskgroup DATA was not mounted
ORA-00600: internal error code, arguments: [kfdJoin3], [7], [12], [500], [], [], [], [], [], [], [], []
ERROR: ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:0:2} */
NOTE: cache dismounting (clean) group 3/0xFC4A3E2F (REDO)
NOTE: messaging CKPT to quiesce pins Unix process pid: 1348, image: oracle@ODA01A
NOTE: dbwr not being msg'd to dismount
...
NOTE: cache dismounted group 2/0xFC4A3E2E (RECO)
NOTE: cache ending mount (fail) of group RECO number=2 incarn=0xfc4a3e2e
NOTE: cache deleting context for group RECO 2/0xfc4a3e2e
GMON dismounting group 2 at 3 for pid 23, osid 1348
ERROR: diskgroup RECO was not mounted
When trying to startup the ASM instance manually, I got the ORA-600:
SQL> startup
ASM instance started
Total System Global Area 1068937216 bytes
Fixed Size 2235208 bytes
Variable Size 1041536184 bytes
ASM Cache 25165824 bytes
ORA-00600: internal error code, arguments: [kfdJoin3], [7], [12], [500], [], [], [], [], [], [], [], []
SQL> shutdown immediate;
ORA-15100: invalid or missing diskgroup name
ORA-15100: invalid or missing diskgroup name
This particular ORA-600 argument is actually mentioned in the README for patch 2.6.0 under “Known Issues” (item 11). But first after the GI part of the patch had been replaced, this could occur (and we had not applied the patch yet to this ODA). Stopping/starting Clusterware did not do any changes (was not possible due to ASM disk groups not mounting).
An SR was opened with Oracle Support and after a thorough analysis, upload of “sundiag” information etc., this issue was filed under bug 16820438.
The resolution was implemented in cooperation with Oracle Support/Development; a hidden parameter (_disable_appliance_check=TRUE) was added to the ASM-instance (via PFILE as the SPFILE not available) and then ASM started and diskgroups could be mounted:
SQL> startup pfile='/u01/app/11.2.0.3/grid/dbs/init_+ASM1.ora';
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
ASM diskgroups volume enabled
SQL>
NOTICE: Remember to only use such hidden parameters under supervision by Oracle Support!
The problematic disk devices were dropped from the ASM disk groups and then devices were added again. After this all DGs could mount, no dataloss was experienced and the whole GI stack started up.
According to Oracle Support bug 16820438 was caused by a unique set of circumstances related to the FCO and will not require any fix – we will not run into it ever again, they state.
The lessons learned from this experience
- Always implement ODAs in pairs (PRIMARY + STANDBY/TEST)
- Always perform changes on STANDBY/TEST first
- Perform OS reboot after any change – then evaluate and implement change on PRIMARY ODA
Conclusion: The power of the ODA is the high degree of standardisation through the whole stack compared to custom-built systems, throughout your own data center, but importantly also accross Oracle customers worldwide. In our example the two ODAs behaved 100% identically, the only problem was that we did not catch the error right after the FCO, because no reboot was performed.
If you would like more information on the Oracle Database Appliance X3-2 feel free to contact our support team at +45 70 272 247 or fill out the contact form.