What Happens When a Patch Breaks?

---

Patches will break. A worker will not be able to finish. A copy driver will fail trying to replace a program unit. A generate driver will be unable to generate forms and reports that the patch provided. Innumerable things can break. If you install enough patches (or not enough patches) at some point you will have a patch that will not install correctly. When a patch driver runs, it creates a table in the database called fnd_install_process. This table is one of the components that allow a patch to be restarted at the point where it failed. The existence of this table tells ADPATCH that it has already started a patch and it might need to finish it. If for any reason you need to move a patch out of the way to install a fix for a problem that it encountered, you will need to rename this table so that the patch you are installing to fix the broken patch that you want to restart later will be able to run successfully. Create table fnd_install_processes_<initials>_<date> or create table fnd_install_processes_<driver> as select * from fnd_install_processes followed by drop table fnd_install_processes when logged in as Apps user (apps/apps) will effectively move the table out of the way and allow you to run another patch to fix a problem.

But how do you figure out what broke, let alone how to fix it?

If you look at the ADWORKER logs, relink logs, rebase logs, or the main patch driver logs, they will tell you a considerable amount of information about what is going on within the patch and where to look for problems that might have happened. Frequently, it is as simple as a database error, a tablespace out of space, a table or index hitting the maximum number of extents, or a patch trying to drop or add a column to a table that already has been dropped or added by another step or another patch. If this is the case, restarting the driver will not get you anywhere; if the DDL statement had already been run by something else, it will not run again successfully. None of the visible options in ADCTRL will get you past the failed step in the driver. There does not appear to be any way past the error or any way to stop the ADPATCH run short of shutting the window or aborting the patching session (but this will not get the patch to finish, either).

There are, however, three invisible options when running ADCTRL. Option 8 will allow you to skip a worker. Option 9 will tell a worker, unconditionally, to wait. Option 10 exits (as does Option 7) ADCTRL. None of these options are supposed to be used without expressed direction of Oracle Support. I do not suggest that anyone use any of these options without being sure of what the option is going to do or what is likely to be the result of the action. I have made the decision to use Option 8 independent of Oracle Support when I have looked at the failing step of the patch and determined that not forcing the broken step would not cause any problems later. For example, if a sql statement is trying to drop a column and the column to be dropped had already been dropped by a previous run of the same patch or by another patch in the series, trying to make it drop the same column again would be an exercise in waiting for an extended period of time for Oracle Support to tell you that you cannot drop a column that had already been dropped. Knowing when you can safely skip a step comes with understanding what the worker is doing and taking the leap. Having a good backup to fall back on is also a good thing to have, so you can know that if you make a decision like this, you can always fall back and restore from the backup and make a different choice the next time. Option 8 takes, as a requested parameter, the worker number you want to skip. Make sure that if you do this, you are skipping the correct one.

Option 9, I have never been instructed to use by Oracle Support, and I have not ever determined when this option would be useful, but it is a hidden option in the ADCTRL utility. It takes, as its requested parameters, a worker number, a series of worker numbers, or "all" for telling all workers to wait.

I also do not recommend trying to fix broken patches on your own, until you have developed some confidence in what you are doing. As soon as your patch gets to what you perceive to be a broken point (you might think it is broken because it does not appear to be doing anything for several hours, you may have workers that fail, or you may find errors or warnings in the log files), log an iTAR, make it as severe as it needs to be, but do not tell Oracle Support that it is Production if it is not. The analysts will be much more likely to go out of their way to help you if you are honest with them all along the way. The more you work through the problems that you encounter with Oracle Support's help, the more confidence you will build in knowing where to look when something breaks, the more you will find yourself being able to rely on yourself and on your abilities, and to rely on your own judgment. While you wait for Oracle Support to respond, you can start looking at worker logs, the main log, the req files in the log directory to see if you can reconstruct what was going on at the time of the failure. This will help you gain confidence in what you are doing and will give you a head start on being able to provide Oracle Support with requested information quickly to help with the resolution.

---